佳佳的博客
Menu
首页
在 Windows 上运行 Apache Storm 2.0.0
Posted by
佳佳
on 2019-09-06
IT
Zookeeper
Storm
<!-- # 在 Windows 上运行 Apache Storm 2.0.0 --> <!-- running-apache-storm-2-0-0-on-windows --> 本文主要参考 [Running Apache Storm on Windows][6](其使用的是 *0.9.1-incubating* 版)。 前置需要安装 Java 和 Python,[Storm 2.0.0][9] 需要安装 Java 8。 安装后不要忘记配置环境变量。 | 变量名 | 变量值 | | --- | --- | | `JAVA_HOME` | *C:\Progra~1\Java\jdk1.8.0_111* | | `Path` (追加) | *C:\Python37\Scripts\;C:\Python37\;%JAVA_HOME%\bin;* | | `PATHEXT` (追加) | *.PY;* | 安装后可使用命令 `java -version` 和 `py -V` 确认安装的版本。 ```bash C:\Users\liujiajia>java -version java version "1.8.0_111" Java(TM) SE Runtime Environment (build 1.8.0_111-b14) Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode) ``` ```bash C:\Users\liujiajia>py -V Python 3.7.3 ``` ## 安装 Zookeeper 下载 [zookeeper-3.4.14.tar.gz][8] 解压到 *D:\zookeeper\zookeeper-3.4.14* 目录后执行如下命令。 ```bash D: cd D:\zookeeper\zookeeper-3.4.14 copy conf\zoo_sample.cfg conf\zoo.cfg bin\zkServer.cmd ``` 执行成功的窗口打印日志: ```java D:\zookeeper\zookeeper-3.4.14>call "C:\Progra~1\Java\jdk1.8.0_111"\bin\java "-Dz ookeeper.log.dir=D:\zookeeper\zookeeper-3.4.14\bin\.." "-Dzookeeper.root.logger= INFO,CONSOLE" -cp "D:\zookeeper\zookeeper-3.4.14\bin\..\build\classes;D:\zookeep er\zookeeper-3.4.14\bin\..\build\lib\*;D:\zookeeper\zookeeper-3.4.14\bin\..\*;D: \zookeeper\zookeeper-3.4.14\bin\..\lib\*;D:\zookeeper\zookeeper-3.4.14\bin\..\co nf" org.apache.zookeeper.server.quorum.QuorumPeerMain "D:\zookeeper\zookeeper-3. 4.14\bin\..\conf\zoo.cfg" 2019-09-05 14:13:04,838 [myid:] - INFO [main:QuorumPeerConfig@136] - Reading co nfiguration from: D:\zookeeper\zookeeper-3.4.14\bin\..\conf\zoo.cfg 2019-09-05 14:13:04,856 [myid:] - INFO [main:DatadirCleanupManager@78] - autopu rge.snapRetainCount set to 3 2019-09-05 14:13:04,857 [myid:] - INFO [main:DatadirCleanupManager@79] - autopu rge.purgeInterval set to 0 2019-09-05 14:13:04,857 [myid:] - INFO [main:DatadirCleanupManager@101] - Purge task is not scheduled. 2019-09-05 14:13:04,860 [myid:] - WARN [main:QuorumPeerMain@116] - Either no co nfig or no quorum defined in config, running in standalone mode 2019-09-05 14:13:04,935 [myid:] - INFO [main:QuorumPeerConfig@136] - Reading co nfiguration from: D:\zookeeper\zookeeper-3.4.14\bin\..\conf\zoo.cfg 2019-09-05 14:13:04,936 [myid:] - INFO [main:ZooKeeperServerMain@98] - Starting server 2019-09-05 14:13:04,951 [myid:] - INFO [main:Environment@100] - Server environm ent:zookeeper.version=3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT 2019-09-05 14:13:04,951 [myid:] - INFO [main:Environment@100] - Server environm ent:host.name=EDG2GYKVF8G5P6Z.octopus.888ly.cn 2019-09-05 14:13:04,952 [myid:] - INFO [main:Environment@100] - Server environm ent:java.version=1.8.0_111 2019-09-05 14:13:04,952 [myid:] - INFO [main:Environment@100] - Server environm ent:java.vendor=Oracle Corporation 2019-09-05 14:13:04,953 [myid:] - INFO [main:Environment@100] - Server environm ent:java.home=C:\Progra~1\Java\jdk1.8.0_111\jre 2019-09-05 14:13:04,953 [myid:] - INFO [main:Environment@100] - Server environm ent:java.class.path=D:\zookeeper\zookeeper-3.4.14\bin\..\build\classes;D:\zookee per\zookeeper-3.4.14\bin\..\build\lib\*;D:\zookeeper\zookeeper-3.4.14\bin\..\zoo keeper-3.4.14.jar;D:\zookeeper\zookeeper-3.4.14\bin\..\lib\audience-annotations- 0.5.0.jar;D:\zookeeper\zookeeper-3.4.14\bin\..\lib\jline-0.9.94.jar;D:\zookeeper \zookeeper-3.4.14\bin\..\lib\log4j-1.2.17.jar;D:\zookeeper\zookeeper-3.4.14\bin\ ..\lib\netty-3.10.6.Final.jar;D:\zookeeper\zookeeper-3.4.14\bin\..\lib\slf4j-api -1.7.25.jar;D:\zookeeper\zookeeper-3.4.14\bin\..\lib\slf4j-log4j12-1.7.25.jar;D: \zookeeper\zookeeper-3.4.14\bin\..\conf 2019-09-05 14:13:04,953 [myid:] - INFO [main:Environment@100] - Server environm ent:java.library.path=C:\Progra~1\Java\jdk1.8.0_111\bin;C:\Windows\Sun\Java\bin; C:\Windows\system32;C:\Windows;C:\ProgramData\Oracle\Java\javapath;C:\Windows\sy stem32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell \v1.0\;C:\Windows\system32\config\systemprofile\.dnx\bin;C:\Program Files\Micros oft DNX\Dnvm\;C:\Program Files\Microsoft SQL Server\120\Tools\Binn\;C:\Program F iles\Microsoft SQL Server\130\Tools\Binn\;C:\Program Files\Git\cmd;C:\Program Fi les\dotnet\;C:\Program Files\TortoiseSVN\bin;C:\Program Files\Microsoft SQL Serv er\120\DTS\Binn\;C:\Program Files\Microsoft SQL Server\Client SDK\ODBC\110\Tools \Binn\;C:\Program Files (x86)\Microsoft SQL Server\120\Tools\Binn\;C:\Program Fi les (x86)\Microsoft SQL Server\120\Tools\Binn\ManagementStudio\;C:\Program Files (x86)\Microsoft SQL Server\120\DTS\Binn\;C:\Program Files\Microsoft SQL Server\ Client SDK\ODBC\170\Tools\Binn\;C:\Program Files\PuTTY\;C:\PROGRA~2\Groovy\GROOV Y~1.0\bin;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\Window sPowerShell\v1.0\;C:\Program Files\Redis\;D:\hadoop-3.2.0\bin;D:\hadoop-3.2.0\sb in;C:\Users\liujiajia\AppData\Local\Programs\Python\Python37\Scripts\;C:\Users\l iujiajia\AppData\Local\Programs\Python\Python37\;C:\Users\liujiajia\AppData\Loca l\Programs\Microsoft VS Code\bin;C:\Users\liujiajia\.dotnet\tools;;C:\Program Fi les\Docker Toolbox;. 2019-09-05 14:13:04,954 [myid:] - INFO [main:Environment@100] - Server environm ent:java.io.tmpdir=C:\Users\LIUJIA~1\AppData\Local\Temp\ 2019-09-05 14:13:04,955 [myid:] - INFO [main:Environment@100] - Server environm ent:java.compiler=<NA> 2019-09-05 14:13:04,956 [myid:] - INFO [main:Environment@100] - Server environm ent:os.name=Windows 7 2019-09-05 14:13:04,956 [myid:] - INFO [main:Environment@100] - Server environm ent:os.arch=amd64 2019-09-05 14:13:04,957 [myid:] - INFO [main:Environment@100] - Server environm ent:os.version=6.1 2019-09-05 14:13:04,957 [myid:] - INFO [main:Environment@100] - Server environm ent:user.name=liujiajia 2019-09-05 14:13:04,957 [myid:] - INFO [main:Environment@100] - Server environm ent:user.home=C:\Users\liujiajia 2019-09-05 14:13:04,958 [myid:] - INFO [main:Environment@100] - Server environm ent:user.dir=D:\zookeeper\zookeeper-3.4.14 2019-09-05 14:13:04,988 [myid:] - INFO [main:ZooKeeperServer@836] - tickTime se t to 2000 2019-09-05 14:13:04,988 [myid:] - INFO [main:ZooKeeperServer@845] - minSessionT imeout set to -1 2019-09-05 14:13:04,989 [myid:] - INFO [main:ZooKeeperServer@854] - maxSessionT imeout set to -1 2019-09-05 14:13:05,200 [myid:] - INFO [main:ServerCnxnFactory@117] - Using org .apache.zookeeper.server.NIOServerCnxnFactory as server connection factory 2019-09-05 14:13:05,202 [myid:] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:2181 ``` ## 安装 Storm 下载 [apache-storm-2.0.0.zip][5] 解压到 *D:\storm\apache-storm-2.0.0* 目录 ## 配置环境变量 | 变量名 | 变量值 | | --- | --- | | `STORM_HOME` | *D:\storm\apache-storm-2.0.0* | | `Path` (追加) | *%STORM_HOME%\bin;* | ## 启动 Nimbus、Supervisor 和 Storm UI - Nimbus ```bash D: cd %STORM_HOME% storm.py nimbus ``` - Supervisor ```bash D: cd %STORM_HOME% storm.py supervisor ``` - Storm UI ```bash D: cd %STORM_HOME% storm.py ui ``` 若启动失败,可在 *logs* 目录下查看对应的日志文件。 ## 访问 StormUI 默认地址:[http://localhost:8080/](http://localhost:8080/) 若 *8080* 端口被占用,可在 *conf/storm.yaml* 指定端口号。 ```yaml ui.port: 8071 ``` 这里指定为 *8071* 端口,然后访问 [http://localhost:8071/](http://localhost:8071/) 。  ## 部署 Storm 自带的 examples 中的 "WordCount" Topology 代码在 *D:\storm\apache-storm-2.0.0\examples\storm-starter* 目录,需要自己打包。 打包后生成 *storm-starter-2.0.0.jar* 文件(172MB - 这么大难怪下载包里不提供)。 执行如下命令部署: ```bash storm.py jar D:\storm\apache-storm-2.0.0\examples\storm-starter\target\storm-starter-2.0.0.jar org.apache.storm.starter.WordCountTopology WordCount -c nimbus.host=localhost ``` 控制台输出: ```bash D:\storm\apache-storm-2.0.0>storm.py jar D:\storm\apache-storm-2.0.0\examples\storm-starter\target\storm-starter-2.0.0.jar org.apache.storm.starter.WordCountTopology WordCount -c nimbus.host=localhost Running: C:\Progra~1\Java\jdk1.8.0_111\bin\java.exe -client -Ddaemon.name= -Dstorm.options=nimbus.host%3Dlocalhost -Dstorm.home=D:\storm\apache-storm-2.0.0 -Dstorm.log.dir=D:\storm\apache-storm-2.0.0\logs -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib:/usr/lib64 -Dstorm.conf.file= -cp D:\storm\apache-storm-2.0.0\*;D:\storm\apache-storm-2.0.0\lib\*;D:\storm\apache-storm-2.0.0\extlib\*;D:\storm\apache-storm-2.0.0\examples\storm-starter\target\storm-starter-2.0.0.jar;D:\storm\apache-storm-2.0.0\conf;D:\storm\apache-storm-2.0.0\bin; -Dstorm.jar=D:\storm\apache-storm-2.0.0\examples\storm-starter\target\storm-starter-2.0.0.jar -Dstorm.dependency.jars= -Dstorm.dependency.artifacts={} org.apache.storm.starter.WordCountTopology WordCount b'10:17:19.107 [main] INFO o.a.s.StormSubmitter - Generated ZooKeeper secret payload for MD5-digest: -8881801371475247195:-5380596067515307374\r\n10:17:19.153 [main] WARN o.a.s.v.ConfigValidation - task.heartbeat.frequency.secs is a deprecated config please see class org.apache.storm.Config.TASK_HEARTBEAT_FREQUENCY_SECS for more information.\r\n10:17:19.227 [main] INFO o.a.s.u.NimbusClient - Found leader nimbus : EDG2GYKVF8G5P6Z.octopus.888ly.cn:6627\r\n10:17:19.228 [main] INFO o.a.s.s.a.ClientAuthUtils - Got AutoCreds []\r\n10:17:19.256 [main] INFO o.a.s.StormSubmitter - Uploading dependencies - jars...\r\n10:17:19.257 [main] INFO o.a.s.StormSubmitter - Uploading dependencies - artifacts...\r\n10:17:19.257 [main] INFO o.a.s.StormSubmitter - Dependency Blob keys - jars : [] / artifacts : []\r\n10:17:19.267 [main] INFO o.a.s.StormSubmitter - Uploading topology jar D:\\storm\\apache-storm-2.0.0\\examples\\storm-starter\\target\\storm-starter-2.0.0.jar to assigned location: D:\\storm\\apache-storm-2.0.0\\storm-local\\nimbus\\inbox/stormjar-ebe35cf8-cbc5-4821-a95f-f604bcebe442.jar\r\n10:17:26.576 [main] INFO o.a.s.StormSubmitter - Successfully uploaded topology jar to assigned location: D:\\storm\\apache-storm-2.0.0\\storm-local\\nimbus\\inbox/stormjar-ebe35cf8-cbc5-4821-a95f-f604bcebe442.jar\r\n10:17:26.576 [main] INFO o.a.s.StormSubmitter - Submitting topology WordCount in distributed mode with conf {"storm.zookeeper.topology.auth.scheme":"digest","storm.zookeeper.topology.auth.payload":"-8881801371475247195:-5380596067515307374","nimbus.host":"localhost","topology.workers":3,"topology.debug":true}\r\n10:17:27.604 [main] INFO o.a.s.StormSubmitter - Finished submitting topology: WordCount\r\n' ``` 在 Storm UI 上查看 Topology summary  <!-- 参考 --> [1]: http://storm.apache.org/ (Apache Storm) [2]: http://storm.apache.org/releases/2.0.0/index.html (Documentation Version: 2.0.0) [3]: https://github.com/ppat/storm-rabbitmq (storm-rabbitmq) [4]: http://storm.apache.org/downloads.html (Apache Storm downloads) [5]: https://www.apache.org/dyn/closer.lua/storm/apache-storm-2.0.0/apache-storm-2.0.0.zip (apache-storm-2.0.0.zip) [6]: http://ptgoetz.github.io/blog/2013/12/18/running-apache-storm-on-windows/ (Running Apache Storm on Windows) [7]: http://zookeeper.apache.org/ (Apache ZooKeeper) [8]: http://mirror.bit.edu.cn/apache/zookeeper/zookeeper-3.4.14/zookeeper-3.4.14.tar.gz (zookeeper-3.4.14.tar.gz) [9]: http://storm.apache.org/2019/05/30/storm200-released.html (Apache Storm 2.0.0 Released) [10]: https://stackoverflow.com/questions/28473309/storm-ui-not-working (Storm UI not working)
版权声明:原创文章,未经允许不得转载。
https://www.liujiajia.me/2019/9/6/running-apache-storm-2-0-0-on-windows
“Buy me a nongfu spring”
« 通过 IntelliJ IDEA 打包 Flink Scala 项目
在 Windows 上安装和运行 Flink »
昵称
*
电子邮箱
*
回复内容
*
(回复审核后才会显示)
提交
目录
AUTHOR
刘佳佳
江苏 - 苏州
软件工程师