zookeeper + Kafka 集群部署

10分钟完成 zookeeper + Kafka 集群部署

环境： Centos，内存 4G+ 以上，配置jdk （zookeeper、Kafka 运行都需要 Java 环境）

一、准备工作：

# 安装在此目录下
mkdir /usr/java

# 1、下载 zookeeper、Kafka 包
wget https://mirror.bit.edu.cn/apache/zookeeper/zookeeper-3.6.2/apache-zookeeper-3.6.2-bin.tar.gz
wget https://www.apache.org/dyn/closer.cgi?path=/kafka/2.7.0/kafka_2.13-2.7.0.tgz

# 2、解压安装包
tar -zxf apache-zookeeper-3.6.2-bin.tar.gz
tar -zxf kafka_2.13-2.7.0.tgz
mv apache-zookeeper-3.6.2-bin zookeeper-3.6.2 #修改文件名

# 3、配置zookeeper 集群服务文件
cd zookeeper-3.6.2/conf
# 拷贝三份配置文件（三实例集群）
cp zoo_sample.cfg zoo-1.cfg
cp zoo_sample.cfg zoo-2.cfg
cp zoo_sample.cfg zoo-3.cfg

# 修改 配置文件中的（直接执行脚本即可）
sed -i 's/^dataDir=.*/dataDir=/tmp/zookeeper-1/' zoo-1.cfg
sed -i 's/^dataDir=.*/dataDir=/tmp/zookeeper-2/' zoo-2.cfg
sed -i 's/^dataDir=.*/dataDir=/tmp/zookeeper-3/' zoo-3.cfg

sed -i 's/^clientPort=.*/clientPort=2181/' zoo-1.cfg
sed -i 's/^clientPort=.*/clientPort=2182/' zoo-2.cfg
sed -i 's/^clientPort=.*/clientPort=2183/' zoo-3.cfg

# 每个配置文件的末尾追加内容
echo '
server.1=127.0.0.1:2888:3888
server.2=127.0.0.1:2889:3889
server.3=127.0.0.1:2890:3890
' | tee -a zoo-1.cfg zoo-2.cfg zoo-3.cfg

# 创建数据目录及配置实例ID
mkdir /tmp/zookeeper-1 /tmp/zookeeper-2 /tmp/zookeeper-3
echo '0' > /tmp/zookeeper-1/myid
echo '1' > /tmp/zookeeper-2/myid
echo '2' > /tmp/zookeeper-3/myid

# 4、配置 Kafka 集群文件  （执行下面的脚本前修改  ${yourhost} 参数）
cd /usr/java/kafka_2.13-2.7.0/config
# 拷贝三分配置文件(三个集群实例)
cp service.properties service-1.properties
cp service.properties service-2.properties
cp service.properties service-3.properties

# 修改节点ID
sed -i 's/^broker.id=.*/broker.id=0/' service-1.properties
sed -i 's/^broker.id=.*/broker.id=1/' service-2.properties
sed -i 's/^broker.id=.*/broker.id=2/' service-3.properties
# 修改监听 端口、IP地址
sed -i 's/^listeners=.*/listeners=PLAINTEXT://:9092/' service-1.properties
sed -i 's/^listeners=.*/listeners=PLAINTEXT://:9093/' service-2.properties
sed -i 's/^listeners=.*/listeners=PLAINTEXT://:9094/' service-3.properties

sed -i 's/^advertised.listeners=.*/advertised.listeners=PLAINTEXT://${yourhost}:9092/' service-1.properties # yourhost 使用 Kafka对外提供服务的IP
sed -i 's/^advertised.listeners=.*/advertised.listeners=PLAINTEXT://${yourhost}:9093/' service-1.properties
sed -i 's/^advertised.listeners=.*/advertised.listeners=PLAINTEXT://${yourhost}:9094/' service-1.properties

# 修改数据存放地址
sed -i 's/^log.dirs=.*/log.dirs=/tmp/kafka-logs-1/' service-1.properties
sed -i 's/^log.dirs=.*/log.dirs=/tmp/kafka-logs-2/' service-2.properties
sed -i 's/^log.dirs=.*/log.dirs=/tmp/kafka-logs-3/' service-3.properties

# 修改分片、副本数
sed -i 's/^num.partitions=.*/num.partitions=3/' service-*.properties
sed -i 's/^offsets.topic.replication.factor=.*/offsets.topic.replication.factor=3/' service-*.properties
sed -i 's/^transaction.state.log.replication.factor=.*/transaction.state.log.replication.factor=3/' service-*.properties
sed -i 's/^transaction.state.log.min.isr=.*/transaction.state.log.min.isr=3/' service-*.properties

# 修改 zookeeper 地址
sed -i 's/^zookeeper.connect=.*/zookeeper.connect=localhost:2181,localhost:2182,localhost:2183/' service-*.properties

# 5、启动 zookeeper 集群、Kafka 集群 
# 分别启动 zookeeper 的三个实例
cd /usr/java/zookeeper-3.6.2
nohup bin/zkServer.sh start conf/zoo-1.cfg &
nohup bin/zkServer.sh start conf/zoo-2.cfg &
nohup bin/zkServer.sh start conf/zoo-3.cfg &

# 分别启动 Kafka 的三个实例
cd /usr/java/kafka_2.13-2.7.0
nohup  bin/kafka-server-start.sh config/server-1.properties &
nohup  bin/kafka-server-start.sh config/server-2.properties &
nohup  bin/kafka-server-start.sh config/server-3.properties &

# 6、测试集群工作情况
...

博客地址：https://1216.top 码云/GitHub:https://gitee.com/tc608