Kafka

다운로드 및 설치

공식홈 Apache Kafka에서 다운받아 압축 풀기

 

Zookeeper

zookeeper.properties

### 로컬환경이라 디렉토리 변경 ###
dataDir=./tmp/zookeeper/
admin.enableServer=false
### 로컬환경이라 포트를 변경하여 추가 ###
clientPort=2181

tickTime=1000
initLimit=5
syncLimit=2

server.1=127.0.0.1:2801:3801
server.2=127.0.0.1:2802:3802
server.3=127.0.0.1:2803:3803
#운영의 경우 일반적으로 아이피가 다름. 같은 서버 또는 POD에 설치할 경우 클러스터링 의미가 퇴색됨
#server.1=xxx.x.x.11:2888:3888
#server.2=xxx.x.x.12:2888:3888
#server.3=xxx.x.x.13:2888:3888

zookeeper1.properties

### 로컬환경이라 디렉토리 변경 ###
dataDir=./tmp/zookeeper1/
admin.enableServer=false
### 로컬환경이라 포트를 변경하여 추가 ###
clientPort=2182

tickTime=1000
initLimit=5
syncLimit=2

server.1=127.0.0.1:2801:3801
server.2=127.0.0.1:2802:3802
server.3=127.0.0.1:2803:3803

zookeeper2.properties

### 로컬환경이라 디렉토리 변경 ###
dataDir=./tmp/zookeeper2/
admin.enableServer=false
### 로컬환경이라 포트를 변경하여 추가 ###
clientPort=2183

tickTime=1000
initLimit=5
syncLimit=2

server.1=127.0.0.1:2801:3801
server.2=127.0.0.1:2802:3802
server.3=127.0.0.1:2803:3803

 

실행

원래 윈도우에서는 C:\kafka>bin\windows\zookeeper-server-start.bat config\zookeeper.properties 같은 방식으로 실행을 하면 되는데, 3 node 실행 시, 각각의 dataDir 폴더 내부에 myid 라는 파일을 만들어 1,2,3 이라는 아이디를 부여해야 함.

 

아래는 git-scm을 설치하여 git bash를 통하여 실행한 예제

echo "starting zookeeper 0"
./bin/zookeeper-server-start.sh config/zookeeper.properties

echo "starting zookeeper 1"
./bin/zookeeper-server-start.sh config/zookeeper1.properties

echo "starting zookeeper 2"
./bin/zookeeper-server-start.sh config/zookeeper2.properties

 

---

Kafka

server.properties

############################# Server Basics #############################
broker.id=3

############################# Socket Server Settings #############################
listeners=PLAINTEXT://0.0.0.0:9093
advertised.listeners=PLAINTEXT://127.0.0.1:9093
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600

############################# Partitions #############################
num.partitions=1
num.recovery.threads.per.data.dir=1

############################# Internal Topic Settings  #############################
### 각각의 숫자는 개발의 경우 1 검증/운영은 3을 권장 ###
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1

############################# Log Basics #############################
### 개발 환경이라 경로가 겹치지 않도록 변경 ###
log.dirs=./tmp/kafka1-logs

############################# Log Flush Policy #############################
#log.flush.interval.messages=10000
#log.flush.interval.ms=1000

############################# Log Retention Policy #############################
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000

############################# Zookeeper #############################
zookeeper.connect=127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183
zookeeper.connection.timeout.ms=18000

############################# Group Coordinator Settings #############################
group.initial.rebalance.delay.ms=0

# 운영의 경우 아래 옵션 활성화
#default.replication.factor = 3
#min.insync.replicas = 2
#auto.create.topics.enable=false
#queued.max.requests=500
#delete.topic.enable=false
#unclean.leader.election.enable=false

server1.properties

############################# Server Basics #############################
broker.id=1

############################# Socket Server Settings #############################
listeners=PLAINTEXT://0.0.0.0:9091
advertised.listeners=PLAINTEXT://127.0.0.1:9091
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600

############################# Partitions #############################
num.partitions=1
num.recovery.threads.per.data.dir=1

############################# Internal Topic Settings  #############################
### 각각의 숫자는 개발의 경우 1 검증/운영은 3을 권장 ###
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1

############################# Log Basics #############################
### 개발 환경이라 경로가 겹치지 않도록 변경 ###
log.dirs=./tmp/kafka2-logs

############################# Log Flush Policy #############################
#log.flush.interval.messages=10000
#log.flush.interval.ms=1000

############################# Log Retention Policy #############################
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000

############################# Zookeeper #############################
zookeeper.connect=127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183
zookeeper.connection.timeout.ms=18000

############################# Group Coordinator Settings #############################
group.initial.rebalance.delay.ms=0

server2.properties

############################# Server Basics #############################
broker.id=2

############################# Socket Server Settings #############################
listeners=PLAINTEXT://0.0.0.0:9092
advertised.listeners=PLAINTEXT://127.0.0.1:9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600

############################# Partitions #############################
num.partitions=1
num.recovery.threads.per.data.dir=1

############################# Internal Topic Settings  #############################
### 각각의 숫자는 개발의 경우 1 검증/운영은 3을 권장 ###
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1

############################# Log Basics #############################
### 개발 환경이라 경로가 겹치지 않도록 변경 ###
log.dirs=./tmp/kafka3-logs

############################# Log Flush Policy #############################
#log.flush.interval.messages=10000
#log.flush.interval.ms=1000

############################# Log Retention Policy #############################
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000

############################# Zookeeper #############################
zookeeper.connect=127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183
zookeeper.connection.timeout.ms=18000

############################# Group Coordinator Settings #############################
group.initial.rebalance.delay.ms=0

 

실행

zookeeper와 마찬가지로 git bash에서 실행을 추천

echo "starting kafka 0"
./bin/kafka-server-start.sh config/server.properties

echo "starting kafka 1"
./bin/kafka-server-start.sh config/server1.properties

echo "starting kafka 2"
./bin/kafka-server-start.sh config/server2.properties