1.集群规划

一般模式下，元数据在 zookeeper 中，运行时动态选举 controller，由controller 进行 Kafka 集群管理。kraft 模式架构（实验性）下，不再依赖 zookeeper 集群，而是用三台 controller 节点代替 zookeeper，元数据保存在 controller 中，由 controller 直接进行 Kafka 集群管理。

好处有以下几个：

Kafka 不再依赖外部框架，而是能够独立运行controller 管理集群时，不再需要从 zookeeper 中先读取数据，集群性能上升由于不依赖 zookeeper，集群扩展时不再受到 zookeeper 读写能力限制controller 不再动态选举，而是由配置文件规定。可以有针对性的加强controller 节点的配置，而不是像以前一样对随机 controller 节点的高负载束手无策。

2.集群部署

1.下载kafka二进制包

https://kafka.apache.org/downloads

2.解压

 tar -zxvf /data/kafka_2.13-3.7.0.tgz

3.修改配置文件(kfka1 192.172.21.120上节点的配置为例)

cd /usr/kafka/kafka_2.13-3.7.0/config/kraft

vi server.properties

<code>注：Kraft模式的配置文件在config目录的kraft子目录下

# Licensed to the Apache Software Foundation (ASF) under one or more

# contributor license agreements.  See the NOTICE file distributed with

# this work for additional information regarding copyright ownership.

# The ASF licenses this file to You under the Apache License, Version 2.0

# (the "License"); you may not use this file except in compliance with

# the License.  You may obtain a copy of the License at

#

#    http://www.apache.org/licenses/LICENSE-2.0

#

# Unless required by applicable law or agreed to in writing, software

# distributed under the License is distributed on an "AS IS" BASIS,

# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

# See the License for the specific language governing permissions and

# limitations under the License.

#

# This configuration file is intended for use in KRaft mode, where

# Apache ZooKeeper is not present.

#

############################# Server Basics #############################

# The role of this server. Setting this puts us in KRaft mode

#角色

process.roles=broker,controller

# The node id associated with this instance's roles

#id

node.id=1

# The connect string for the controller quorum

controller.quorum.voters=1@192.172.21.120:19093,2@192.172.21.121:19093,3@192.172.21.122:19093

############################# Socket Server Settings #############################

# The address the socket server listens on.

# Combined nodes (i.e. those with `process.roles=broker,controller`) must list the controller listener here at a minimum.

# If the broker listener is not defined, the default listener will use a host name that is equal to the value of java.net.InetAddress.getCanonicalHostName(),

# with PLAINTEXT listener name, and port 19092.

#   FORMAT:

#     listeners = listener_name://host_name:port

#   EXAMPLE:

#     listeners = PLAINTEXT://your.host.name:19092

listeners=SASL_PLAINTEXT://192.172.21.120:19092,CONTROLLER://192.172.21.120:19093

# Name of listener used for communication between brokers.

inter.broker.listener.name=SASL_PLAINTEXT

# Listener name, hostname and port the broker will advertise to clients.

# If not set, it uses the value for "listeners".

advertised.listeners=SASL_PLAINTEXT://192.172.21.120:19092

# A comma-separated list of the names of the listeners used by the controller.

# If no explicit mapping set in `listener.security.protocol.map`, default will be using PLAINTEXT protocol

# This is required if running in KRaft mode.

controller.listener.names=CONTROLLER

# Maps listener names to security protocols, the default is for them to be the same. See the config documentation for more details

#CONTROLLER:SASL_PLAINTEXT需要修改

listener.security.protocol.map=CONTROLLER:SASL_PLAINTEXT,PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL

# 设置必须授权才能用

allow.everyone.if.no.acl.found=false

# The number of threads that the server uses for receiving requests from the network and sending responses to the network

num.network.threads=3

# The number of threads that the server uses for processing requests, which may include disk I/O

num.io.threads=8

# The send buffer (SO_SNDBUF) used by the socket server

socket.send.buffer.bytes=102400

# The receive buffer (SO_RCVBUF) used by the socket server

socket.receive.buffer.bytes=102400

# The maximum size of a request that the socket server will accept (protection against OOM)

socket.request.max.bytes=104857600

############################# Log Basics #############################

# A comma separated list of directories under which to store log files

log.dirs=/data/kafka/datas

# The default number of log partitions per topic. More partitions allow greater

# parallelism for consumption, but this will also result in more files across

# the brokers.

num.partitions=1

# The number of threads per data directory to be used for log recovery at startup and flushing at shutdown.

# This value is recommended to be increased for installations with data dirs located in RAID array.

num.recovery.threads.per.data.dir=1

############################# Internal Topic Settings  #############################

# The replication factor for the group metadata internal topics "__consumer_offsets" and "__transaction_state"

# For anything other than development testing, a value greater than 1 is recommended to ensure availability such as 3.

offsets.topic.replication.factor=1

transaction.state.log.replication.factor=1

transaction.state.log.min.isr=1

############################# Log Flush Policy #############################

# Messages are immediately written to the filesystem but by default we only fsync() to sync

# the OS cache lazily. The following configurations control the flush of data to disk.

# There are a few important trade-offs here:

#    1. Durability: Unflushed data may be lost if you are not using replication.

#    2. Latency: Very large flush intervals may lead to latency spikes when the flush does occur as there will be a lot of data to flush.

#    3. Throughput: The flush is generally the most expensive operation, and a small flush interval may lead to excessive seeks.

# The settings below allow one to configure the flush policy to flush data after a period of time or

# every N messages (or both). This can be done globally and overridden on a per-topic basis.

# The number of messages to accept before forcing a flush of data to disk

#log.flush.interval.messages=10000

# The maximum amount of time a message can sit in a log before we force a flush

#log.flush.interval.ms=1000

############################# Log Retention Policy #############################

# The following configurations control the disposal of log segments. The policy can

# be set to delete segments after a period of time, or after a given size has accumulated.

# A segment will be deleted whenever *either* of these criteria are met. Deletion always happens

# from the end of the log.

# The minimum age of a log file to be eligible for deletion due to age

log.retention.hours=168

# A size-based retention policy for logs. Segments are pruned from the log unless the remaining

# segments drop below log.retention.bytes. Functions independently of log.retention.hours.

#log.retention.bytes=1073741824

# The maximum size of a log segment file. When this size is reached a new log segment will be created.

log.segment.bytes=1073741824

# The interval at which log segments are checked to see if they can be deleted according

# to the retention policies

log.retention.check.interval.ms=300000

# 认证方式，用了最简单的PLAIN，缺点是不能动态添加用户

sasl.mechanism.inter.broker.protocol=PLAIN

sasl.enabled.mechanisms=PLAIN

sasl.mechanism=PLAIN

# 禁用了自动创建topic

auto.create.topics.enable = false

# 设置必须授权才能用

allow.everyone.if.no.acl.found=false

# 设置超级管理员

super.users=User:admin

# 这个是3.2.0版本新引入的认证方式，可以参考 https://cwiki.apache.org/confluence/display/KAFKA/KIP-801%3A+Implement+an+Authorizer+that+stores+metadata+in+__cluster_metadata

authorizer.class.name=org.apache.kafka.metadata.authorizer.StandardAuthorizer

# 集群间认证时用的认证方式

sasl.mechanism.controller.protocol=PLAIN

5.在其他节点上修改配置文件

在 192.172.21.121 和 192.172.21.122 上修改配置文件server.properties中

1.node.id

注：node.id 不得重复，整个集群中唯一，且值需要和controller.quorum.voters 对应。

2.dvertised.Listeners地址

根据各自的主机名称，修改相应的 dvertised.Listeners 地址

3.listeners地址

根据各自的主机IP修改

# 节点 ID

node.id=2

#不同服务器绑定的端口

listeners=SASL_PLAINTEXT://192.172.21.121:19092,CONTROLLER://192.172.21.121:19093

# 侦听器名称、主机名和代理将向客户端公布的端口.(broker 对外暴露的地址)

# 如果未设置，则使用"listeners"的值.

advertised.listeners=SASL_PLAINTEXT://192.172.21.121:19092

# 节点 ID

node.id=3

#不同服务器绑定的端口

listeners=SASL_PLAINTEXT://192.172.21.122:19092,CONTROLLER://192.172.21.122:19093

# 侦听器名称、主机名和代理将向客户端公布的端口.(broker 对外暴露的地址)

# 如果未设置，则使用"listeners"的值.

advertised.listeners=SASL_PLAINTEXT://192.172.21.122:19092

6.创建Kraft账号密码认证文件

KafkaServer {

   org.apache.kafka.common.security.plain.PlainLoginModule required

   username="admin"code>

   password="password"code>

   user_admin="password"code>

   user_test="test";code>

};

username/password 表示了认证时用的用户。suer_admin="password",这个表示一个用户名为admin用户，密码是password，这个必须要有一个，且要这一个跟上面的username和password保持一致。user_test="test" 是第二个用户，表示的是用户名为test的账户，密码为test。