카프카 도커라이징 관련 자료조사 #6

Kdreamtomaster · 2024-09-02T11:46:44Z

Kdreamtomaster
Sep 2, 2024
Maintainer

dnlpys · 2024-09-03T00:47:17Z

dnlpys
Sep 3, 2024
Maintainer

kafka kraft 모드 confluentinc github example
https://github.com/confluentinc/kafka-images/blob/master/examples/confluent-server-kraft/docker-compose.yml

0 replies

Kdreamtomaster · 2024-09-03T02:35:09Z

Kdreamtomaster
Sep 3, 2024
Maintainer Author

ver0.1

server01 .env

KAFKA_LISTENER_NAME=PLAINTEXT
KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092,CONTROLLER://server01:9093
KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://server01:9092,CONTROLLER://server01:9093
KAFKA_PROCESS_ROLES=broker,controller
KAFKA_NODE_ID=1
KAFKA_CONTROLLER_QUORUM_VOTERS=1@server01:9093,2@server02:9093,3@server03:9093
KAFKA_METASTORE_BOOTSTRAP_SERVERS=server01:9092
KAFKA_LOG_DIRS=/var/lib/kafka/data
KAFKA_METADATA_LOG_DIR=/var/lib/kafka/metadata
KAFKA_LOG_RETENTION_HOURS=168
KAFKA_CONFLUENT_SUPPORT_METRICS_ENABLE=false

server02 .env

KAFKA_LISTENER_NAME=PLAINTEXT
KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092,CONTROLLER://server01:9093
KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://server02:9092
KAFKA_PROCESS_ROLES=broker
KAFKA_NODE_ID=2
KAFKA_CONTROLLER_QUORUM_VOTERS=1@server01:9093,2@server02:9093,3@server03:9093
KAFKA_METASTORE_BOOTSTRAP_SERVERS=server01:9092
KAFKA_LOG_DIRS=/var/lib/kafka/data
KAFKA_METADATA_LOG_DIR=/var/lib/kafka/metadata
KAFKA_LOG_RETENTION_HOURS=168
KAFKA_CONFLUENT_SUPPORT_METRICS_ENABLE=false

server03 .env

KAFKA_LISTENER_NAME=PLAINTEXT
KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092,CONTROLLER://server01:9093
KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://server03:9092
KAFKA_PROCESS_ROLES=broker
KAFKA_NODE_ID=3
KAFKA_CONTROLLER_QUORUM_VOTERS=1@server01:9093,2@server02:9093,3@server03:9093
KAFKA_METASTORE_BOOTSTRAP_SERVERS=server01:9092
KAFKA_LOG_DIRS=/var/lib/kafka/data
KAFKA_METADATA_LOG_DIR=/var/lib/kafka/metadata
KAFKA_LOG_RETENTION_HOURS=168
KAFKA_CONFLUENT_SUPPORT_METRICS_ENABLE=false

docker-compose


services:
  kafka:
    image: confluentinc/cp-kafka:7.7.0
    container_name: kafka
    ports:
      - "9092:9092"
      - "9093:9093"
    extra_hosts:
      - "server01:172.31.3.183"
      - "server02:172.31.2.37"
      - "server03:172.31.0.17"
    env_file:
      - .env
    volumes:
      - kafka-data:/var/lib/kafka/data
      - kafka-metadata:/var/lib/kafka/metadata

volumes:
  kafka-data:
    driver: local
  kafka-metadata:
    driver: local

2 replies

Kdreamtomaster Sep 3, 2024
Maintainer Author

ubuntu@ip-172-31-0-17:~/app/kafka-docker$ docker compose up
[+] Running 14/14
 ✔ kafka Pulled                                                                                                                             16.6s
   ✔ 4798a7e93601 Pull complete                                                                                                              2.9s
   ✔ a453f30e82bf Pull complete                                                                                                             11.8s
   ✔ 016e383f3f47 Pull complete                                                                                                             11.9s
   ✔ f7d27dafad0a Pull complete                                                                                                             12.0s
   ✔ 56ccc8be1ca0 Pull complete                                                                                                             12.0s
   ✔ f77f01ac624c Pull complete                                                                                                             12.4s
   ✔ 1c6e35a73ed7 Pull complete                                                                                                             12.5s
   ✔ aa5e151b62ff Pull complete                                                                                                             12.5s
   ✔ 262d375318c3 Pull complete                                                                                                             12.5s
   ✔ 28a7d18ebda4 Pull complete                                                                                                             12.6s
   ✔ bdc615dfc787 Pull complete                                                                                                             12.6s
   ✔ 33966fd36306 Pull complete                                                                                                             13.7s
   ✔ 8b4455fb60b9 Pull complete                                                                                                             13.8s
[+] Running 4/4
 ✔ Network kafka-docker_default          Created                                                                                             0.1s
 ✔ Volume "kafka-docker_kafka-data"      Created                                                                                             0.0s
 ✔ Volume "kafka-docker_kafka-metadata"  Created                                                                                             0.0s
 ✔ Container kafka                       Created                                                                                             5.2s
Attaching to kafka
kafka  | ===> User
kafka  | uid=1000(appuser) gid=1000(appuser) groups=1000(appuser)
kafka  | ===> Configuring ...
kafka  | Running in KRaft mode...
kafka  | CLUSTER_ID is required.
kafka  | Command [/usr/local/bin/dub ensure CLUSTER_ID] FAILED !

CLUSTER_ID 사전 설정 방법 찾아야함

Kdreamtomaster Sep 3, 2024
Maintainer Author

KAFKA_KRAFT_MODE=true
CLUSTER_ID=8ea13e0c-5bb1-4d75-bb03-c61034a1388a

이런식으로 해결

Kdreamtomaster · 2024-09-03T02:36:28Z

Kdreamtomaster
Sep 3, 2024
Maintainer Author

  environment:
      KAFKA_LISTENER_NAME=PLAINTEXT
      KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093
      KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://server2:9092,CONTROLLER://server2:9093

 이부분 조사 필요

2 replies

Kdreamtomaster Sep 3, 2024
Maintainer Author

INTERNAL, EXTERNAL 관련내용 조사및 수정 필요

Kdreamtomaster Sep 3, 2024
Maintainer Author

기존에 쓰는 포트 9092, 9093 을 다른포트로 변경하여 설정하고자 할때 쓰는 것으로 파악됨

Kdreamtomaster · 2024-09-03T05:15:57Z

Kdreamtomaster
Sep 3, 2024
Maintainer Author

Exception in thread "main" java.lang.IllegalArgumentException: Error creating broker listeners from 'PLAINTEXT://0.0.0.0:9092,CONTROLLER://server01:9093': No security protocol defined for listener CONTROLLER

CONTROLLER 에 대해서 보안 프로토콜 설정 방법 찾아야함

1 reply

dnlpys Sep 3, 2024
Maintainer

.env 파일에 추가해서 테스트

KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT

Kdreamtomaster · 2024-09-03T05:40:21Z

Kdreamtomaster
Sep 3, 2024
Maintainer Author

ubuntu@ip-172-31-2-37:~/app/kafka-docker$ docker compose up
[+] Running 2/2
 ✔ Network kafka-docker_default  Created                                                                                                     0.1s
 ✔ Container kafka               Created                                                                                                     0.0s
Attaching to kafka
kafka  | ===> User
kafka  | uid=1000(appuser) gid=1000(appuser) groups=1000(appuser)
kafka  | ===> Configuring ...
kafka  | Running in KRaft mode...
kafka  | ===> Running preflight checks ...
kafka  | ===> Check if /var/lib/kafka/data is writable ...
kafka  | ===> Running in KRaft mode, skipping Zookeeper health check...
kafka  | ===> Using provided cluster id 8ea13e0c-5bb1-4d75-bb03-c61034a1388a ...
kafka  | Exception in thread "main" java.lang.IllegalArgumentException: requirement failed: 
The advertised.listeners config must not contain KRaft controller listeners from controller.listener.names 
when process.roles contains the broker role because Kafka clients that send requests 
via advertised listeners do not send requests to KRaft controllers -- they only send requests to KRaft brokers. 
at scala.Predef$.require(Predef.scala:337) 
at kafka.server.KafkaConfig.validateAdvertisedListenersDoesNotContainControllerListenersForKRaftBroker$1(KafkaConfig.scala:2352) 
at kafka.server.KafkaConfig.validateValues(KafkaConfig.scala:2420) 
at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:2290) 
at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1638) 
at kafka.tools.StorageTool$.$anonfun$execute$1(StorageTool.scala:71) 
at scala.Option.flatMap(Option.scala:283) at kafka.tools.StorageTool$.execute(StorageTool.scala:71) 
at kafka.tools.StorageTool$.main(StorageTool.scala:52) 
at kafka.tools.StorageTool.main(StorageTool.scala)

0 replies

Kdreamtomaster · 2024-09-03T07:51:29Z

Kdreamtomaster
Sep 3, 2024
Maintainer Author

ver 0.2 env 및 docker-compose 내용 모음 및 현재 에러

server01 .env

KAFKA_KRAFT_MODE=true
CLUSTER_ID=8ea13e0c-5bb1-4d75-bb03-c61034a1388a
KAFKA_LISTENER_NAME=PLAINTEXT
KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092,CONTROLLER://server01:9093
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT
KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://server01:9092
KAFKA_PROCESS_ROLES=broker,controller
KAFKA_NODE_ID=1
KAFKA_CONTROLLER_QUORUM_VOTERS=1@server01:9093,2@server02:9093,3@server03:9093
KAFKA_METASTORE_BOOTSTRAP_SERVERS=server01:9092,server02:9092,server03:9092
KAFKA_LOG_DIRS=/var/lib/kafka/data
KAFKA_METADATA_LOG_DIR=/var/lib/kafka/metadata
KAFKA_LOG_RETENTION_HOURS=168
KAFKA_CONFLUENT_SUPPORT_METRICS_ENABLE=false
KAFKA_CONTROLLER_LISTENER_NAMES=CONTROLLER

server02 .env

KAFKA_KRAFT_MODE=true
CLUSTER_ID=8ea13e0c-5bb1-4d75-bb03-c61034a1388a
KAFKA_LISTENER_NAME=PLAINTEXT
KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092,CONTROLLER://server01:9093
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT
KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://server02:9092
KAFKA_PROCESS_ROLES=broker,controller
KAFKA_NODE_ID=2
KAFKA_CONTROLLER_QUORUM_VOTERS=1@server01:9093,2@server02:9093,3@server03:9093
KAFKA_METASTORE_BOOTSTRAP_SERVERS=server01:9092,server02:9092,server03:9092
KAFKA_LOG_DIRS=/var/lib/kafka/data
KAFKA_METADATA_LOG_DIR=/var/lib/kafka/metadata
KAFKA_LOG_RETENTION_HOURS=168
KAFKA_CONFLUENT_SUPPORT_METRICS_ENABLE=false
KAFKA_CONTROLLER_LISTENER_NAMES=CONTROLLER

server03 .env

KAFKA_KRAFT_MODE=true
CLUSTER_ID=8ea13e0c-5bb1-4d75-bb03-c61034a1388a
KAFKA_LISTENER_NAME=PLAINTEXT
KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092,CONTROLLER://server01:9093
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT
KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://server03:9092
KAFKA_PROCESS_ROLES=broker,controller
KAFKA_NODE_ID=3
KAFKA_CONTROLLER_QUORUM_VOTERS=1@server01:9093,2@server02:9093,3@server03:9093
KAFKA_METASTORE_BOOTSTRAP_SERVERS=server01:9092,server02:9092,server03:9092
KAFKA_LOG_DIRS=/var/lib/kafka/data
KAFKA_METADATA_LOG_DIR=/var/lib/kafka/metadata
KAFKA_LOG_RETENTION_HOURS=168
KAFKA_CONFLUENT_SUPPORT_METRICS_ENABLE=false
KAFKA_CONTROLLER_LISTENER_NAMES=CONTROLLER

docker-compose.yml

ubuntu@ip-172-31-3-183:~/app/kafka-docker$ cat docker-compose.yml
services:
  kafka:
    image: confluentinc/cp-kafka:7.7.0
    container_name: kafka
    ports:
      - "9092:9092"
      - "9093:9093"
    extra_hosts:
      - "server01:172.31.3.183"
      - "server02:172.31.2.37"
      - "server03:172.31.0.17"
    env_file:
      - .env
    volumes:
      - kafka-data:/var/lib/kafka/data
      - kafka-metadata:/var/lib/kafka/metadata
    user: "1000:1000"
volumes:
  kafka-data:
    driver: local
  kafka-metadata:
    driver: local

ERROR

메타데이터 관련 권한 문제라고는 하는데 로컬 설치 과정 및 Doc 살펴보고 다시 보겠습니다.

ubuntu@ip-172-31-3-183:~/app/kafka-docker$ docker compose up
[+] Running 4/4
 ✔ Network kafka-docker_default          Created                                                                                             0.1s
 ✔ Volume "kafka-docker_kafka-data"      Created                                                                                             0.0s
 ✔ Volume "kafka-docker_kafka-metadata"  Created                                                                                             0.0s
 ✔ Container kafka                       Created                                                                                             0.1s
Attaching to kafka
kafka  | ===> User
kafka  | uid=1000(appuser) gid=1000(appuser) groups=1000(appuser)
kafka  | ===> Configuring ...
kafka  | Running in KRaft mode...
kafka  | ===> Running preflight checks ...
kafka  | ===> Check if /var/lib/kafka/data is writable ...
kafka  | ===> Running in KRaft mode, skipping Zookeeper health check...
kafka  | ===> Using provided cluster id 8ea13e0c-5bb1-4d75-bb03-c61034a1388a ...
kafka  | metaPropertiesEnsemble=MetaPropertiesEnsemble(metadataLogDir=Optional.empty, dirs={/var/lib/kafka/data: EMPTY, /var/lib/kafka/metadata: EMPTY}) Formatting /var/lib/kafka/data with metadata.version 3.7-IV4. Formatting /var/lib/kafka/metadata with metadata.version 3.7-IV4. Error while writing meta.properties file /var/lib/kafka/metadata: /var/lib/kafka/metadata/bootstrap.checkpoint.tmp

0 replies

Kdreamtomaster · 2024-09-03T10:50:07Z

Kdreamtomaster
Sep 3, 2024
Maintainer Author

version 0.21
단일 컨테이너로는 띄워짐 , 이미지 버전도 낮춰봄

services:
  kafka01:
    image: confluentinc/cp-kafka:7.5.3
    container_name: kafka
    extra_hosts:
      - "server01:172.31.3.183"
      - "server02:172.31.2.37"
      - "server03:172.31.0.17"

    ports:
      - 29092:9092
    env_file:
      - .env
    user: "1000:1000"

컨테이너 이름을 QUORUM_VOTERS로 줘봄

KAFKA_KRAFT_MODE=true
CLUSTER_ID=${KAFKA_CLUSTER_ID:-MkU3OEVBNTcwNTJENDM2Qk}
KAFKA_LISTENER_NAME=PLAINTEXT
KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:29092,CONTROLLER://0.0.0.0:9093,PLAINTEXT_HOST://0.0.0.0:9092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://host.docker.internal:29092,PLAINTEXT_HOST://host.docker.internal:9092
KAFKA_PROCESS_ROLES=broker,controller
KAFKA_NODE_ID=1
KAFKA_CONTROLLER_QUORUM_VOTERS=1@kafka01:29092
KAFKA_LOG_DIRS=/var/lib/kafka/data
KAFKA_LOG_RETENTION_HOURS=168
KAFKA_CONFLUENT_SUPPORT_METRICS_ENABLE=false
KAFKA_CONTROLLER_LISTENER_NAMES=CONTROLLER

1 reply

Kdreamtomaster Sep 4, 2024
Maintainer Author

버전을 낮추니 CLUSTER_ID를 UUID가 아닌 아래 글 안내처럼 작성해야 했음

Cluster Id
The cluster id is a unique and immutable identifier assigned to a Kafka cluster. The cluster id can have a maximum of 22 characters and the allowed characters are defined by the regular expression [a-zA-Z0-9_-]+, which corresponds to the characters used by the URL-safe Base64 variant with no padding. Conceptually, it is auto-generated when a cluster is started for the first time.

Implementation-wise, it is generated when a broker with version 0.10.1 or later is successfully started for the first time. The broker tries to get the cluster id from the /cluster/id znode during startup. If the znode does not exist, the broker generates a new cluster id and creates the znode with this cluster id.

기존 환경설정 설명

CLUSTER_ID=${KAFKA_CLUSTER_ID:-MkU3OEVBNTcwNTJENDM2Qk}

환경 변수 설정

${KAFKA_CLUSTER_ID}: 환경 변수 KAFKA_CLUSTER_ID의 값을 참조합니다.
:-MkU3OEVBNTcwNTJENDM2Qk: KAFKA_CLUSTER_ID 환경 변수가 설정되어 있지 않거나 빈 값일 때 사용할 기본값입니다.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

카프카 도커라이징 관련 자료조사 #6

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 7 comments 6 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

카프카 도커라이징 관련 자료조사 #6

Kdreamtomaster Sep 2, 2024 Maintainer

Replies: 7 comments · 6 replies

dnlpys Sep 3, 2024 Maintainer

Kdreamtomaster Sep 3, 2024 Maintainer Author

server01 .env

server02 .env

server03 .env

docker-compose

Kdreamtomaster Sep 3, 2024 Maintainer Author

Kdreamtomaster Sep 3, 2024 Maintainer Author

Kdreamtomaster Sep 3, 2024 Maintainer Author

Kdreamtomaster Sep 3, 2024 Maintainer Author

Kdreamtomaster Sep 3, 2024 Maintainer Author

Kdreamtomaster Sep 3, 2024 Maintainer Author

dnlpys Sep 3, 2024 Maintainer

Kdreamtomaster Sep 3, 2024 Maintainer Author

Kdreamtomaster Sep 3, 2024 Maintainer Author

server01 .env

server02 .env

server03 .env

docker-compose.yml

ERROR

Kdreamtomaster Sep 3, 2024 Maintainer Author

Kdreamtomaster Sep 4, 2024 Maintainer Author

버전을 낮추니 CLUSTER_ID를 UUID가 아닌 아래 글 안내처럼 작성해야 했음

기존 환경설정 설명

환경 변수 설정

Kdreamtomaster
Sep 2, 2024
Maintainer

Replies: 7 comments 6 replies

dnlpys
Sep 3, 2024
Maintainer

Kdreamtomaster
Sep 3, 2024
Maintainer Author

Kdreamtomaster Sep 3, 2024
Maintainer Author

Kdreamtomaster Sep 3, 2024
Maintainer Author

Kdreamtomaster
Sep 3, 2024
Maintainer Author

Kdreamtomaster Sep 3, 2024
Maintainer Author

Kdreamtomaster Sep 3, 2024
Maintainer Author

Kdreamtomaster
Sep 3, 2024
Maintainer Author

dnlpys Sep 3, 2024
Maintainer

Kdreamtomaster
Sep 3, 2024
Maintainer Author

Kdreamtomaster
Sep 3, 2024
Maintainer Author

Kdreamtomaster
Sep 3, 2024
Maintainer Author

Kdreamtomaster Sep 4, 2024
Maintainer Author