docker compose를 활용한 Kafka 클러스터 구성

1. docker-compose.yml 생성

version: '2'

services:
  zookeeper1:
    image: confluentinc/cp-zookeeper:latest
    environment:
      ZOOKEEPER_SERVER_ID: 1
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
      ZOOKEEPER_INIT_LIMIT: 5
      ZOOKEEPER_SYNC_LIMIT: 2
    ports:
      - "22181:2181"

  kafka1:
    image: confluentinc/cp-kafka:latest
    depends_on:
      - zookeeper1
    ports:
      - "29092:29092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper1:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka1:9092,PLAINTEXT_HOST://localhost:29092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1

  kafka2:
    image: confluentinc/cp-kafka:latest
    depends_on:
      - zookeeper1
    ports:
      - "39092:39092"
    environment:
      KAFKA_BROKER_ID: 2
      KAFKA_ZOOKEEPER_CONNECT: zookeeper1:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka2:9092,PLAINTEXT_HOST://localhost:39092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1

  kafka3:
    image: confluentinc/cp-kafka:latest
    depends_on:
      - zookeeper1
    ports:
      - "49092:49092"
    environment:
      KAFKA_BROKER_ID: 3
      KAFKA_ZOOKEEPER_CONNECT: zookeeper1:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka3:9092,PLAINTEXT_HOST://localhost:49092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1

 

 

- zookeeper1이 kafka1,2,3을 관리

 

2. 클러스터 생성 및 확인

docker-compose -f docker-compose.yml up -d

docker desktop 성공 화면

 

# kafka docker cli 환경에서

1) kafka-topics --create --topic my-topic --bootstrap-server kafka1:9092, kafka2:9092 ,kafka3:9092  --replication-factor 3 

 

-클러스터링에 성공했다면 replication 3개가 생성 가능해야됨

 

2) kafka-console-producer --topic my-topic --broker-list kafka1:9092,kafka2:9092,kafka3:9092

 

- 브로커 1,2,3 중 응답하는 브로커에서 프로듀서 생성 시작

 

3) kafka-console-consumer --topic my-topic --bootstrap-server kafka1:9092,kafka2:9092,kafka3:9092 --group grp1

위에 명령어를 kafka 1,2,3 브로커에서 띄우고

프로듀서를 통해서 메세지를 전달하면

컨슈머가 클러스터 상태에서 3개의 브로커중에서 1개씩 오는지 확인

1개씩 끄면서 메세지가 정상적으로 cosume되는지 확인