Why Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandaries Explored)

Why Apache Kafka
Clusters Are Like
Galaxies
And Other Cosmic Kafka Quandaries
Explored
Paul Brebner
Instaclustr Technology Evangelist
June 5 2024
© 2024 NetApp, Inc. All rights reserved.

Performance Engineering Track on again at
Community over Code 7-10 October Denver 2024

3
Instaclustr Managed Platform
• Cloud Platform for Big
Data Open Source
Technologies
• Free 30 day trial
• Focus of this talk is on
Apache Kafka®

Centenary of Franz Kafka’s death - June 3 2024
4
Head of Kafka, Prague (Paul Brebner)

Overview
1 Kafka Scalability
2 Kafka Clusters and Zipf’s Law
3 Kafka Clusters and Storage
4 Top 10 Kafka Clusters and Performance
Thanks to Instaclustr colleagues for Kafka cluster data:
Kafka Clusters & Storage - Alastair Daivis & Kafka Team
Top 10 Clusters - Joseph Clay & Ramana Selvaratnam (Technical Operations Team)

A note on Kafka cluster metrics
Easy Performance Metrics Harder
Broker Cluster All Clusters
Size Metrics available
Focus of our metrics
collection is
Per broker
Not per cluster or all clusters
DALL·E 3

Part 1 Kafka Scalability
(Source: Shutterstock)

Kafka is a distributed streams processing
system—it allows distributed producers to send
messages to distributed consumers via a Kafka
cluster.
What is
Kafka?

Cluster = Brokers + Partitions
Enabling Write & Read Concurrency

Partition n
Topic
Partition 1
Producer
Partition 2
Consumer Group
Consumer
Consumer
Consumers share
work within groups
Consumer
Partitions enable Consumers to share work
(c.f. Amish Barn raising) within a consumer group

Multiple groups enable message broadcasting.
Messages are duplicated (c.f. clones) across groups, as each
consumer group receives a copy of each message.
Multiple Groups Enable Message Broadcasting
Consumer
Consumer
Consumer
Consumer
Topic
Partition 1
Partition 2
Partition n
Producer
Consumer Group
Consumer Group
Messages are
duplicated across
Consumer groups
Messages are duplicated (c.f. clones) across groups,
as each consumer group receives a copy of each message

Partitions – concurrency mechanism –
more is better – until it’s not
You need sufficient partitions to benefit from the cluster concurrency
And not too many that the replication overhead impacts overall throughput
0
0.5
1
1.5
2
2.5
1 10 100 1000 10000
Partitions vs. Throughput (M TPS)
ZK TPS (M) KRAFT TPS (M) 2020 TPS (M)
2022 - Better
2020 - Worse
2022 results better due to improvements to Kafka and h/w

17
• Horizontal Scalability (Brokers/Nodes)
• Vertical Scalability (more/faster cores per Broker)
• Hardware (cores, CPU speed/type, RAM, disk, network, etc)
• Partitions + Consumers
• Optimise number of Partitions
• Consumer speed optimization (slow consumers are bad – high latency and too many partitions)
• Kafka cluster and client configurations (many and complex)
• Goals are typically
• High Throughput
• Fast Latency (low 10s ms)
Kafka Scalability and Performance Summary
(Slow consumers are a problem: Getty)

Part 2 Kafka Clusters and Zipf’s Law – size
distribution
Visual size comparison of the six largest Local Group galaxies, with details (Wikipedia)

19
• Distribution function
• Most frequent observation is twice as common
• as next and so on (i.e. 1/rank)
• Long-tailed distribution
• 80/20 rule (20% of people own 80% of $)
• C.f. Pareto (discrete vs. continuous)
• Log-log rank vs frequency/size gives approx. straight line
• Common examples
• Frequency of words
• Wealth distribution
• Animal species size
• Earthquakes
• City sizes
• Computer systems (e.g. workload modelling, subsystem capacity)
• Galaxy sizes
Scaling/power law
Zipf’s Law

20
• Question: How large are the largest
structures in the universe?
• Answer: Bigger!
• Zipf’s law predicted that
• bigger galaxies would be detected in older parts of
the universe
• beyond the reach of the Hubble at the time
• confirmed with the James Webb telescope
observations
• But what’s this got to do with Kafka?
Size and Scale Predictions
Apache Kafka + Galaxies?
Image from NASA’s James Webb Space Telescope showing older and bigger galaxy clusters

21
Raw Kafka Cluster Size Data - Summary Statistics
3 3 3
4.520702635
7.023373433
96
797
3603
1
10
100
1000
10000
Nodes/Cluster
Summary Statistics (log nodes/cluster)
min median mode average stdev max count sum

22
Histogram (size vs count) – skewed distribution
Raw Kafka Cluster Size Data
0
100
200
300
400
500
600
700
800
Total
3 4 6 8 9 12 15 18 21 24 27 30 33 36 39 48 60 72 78 96

23
What is the distribution? Definitely a long-tailed power law
Kafka Clusters and Zipf’s Law
0
20
40
60
80
100
120
0 100 200 300 400 500 600 700 800 900
Size
Cluster
Cluster Size Distribution (largest to smallest)

24
Approximately Zipfian
Kafka Clusters – log size vs log rank
1
10
100
1000
1 10 100
Log
rank
Log size
Kafka Clusters - Log size vs log rank

25
Can expect larger clusters (animals, galaxies etc)
So What? Kafka and Zipf’s Law (1)
African Elephant, 7 t
Maraapunisaurus, extinct dinosaur, 150 t

26
Extrapolation of size from Zipf’s law + largest observed cluster
Predicted larger clusters
0.1
1
10
100
1000
1 10 100 1000
Log
rank
Log size
Kafka Clusters - Log size vs log rank
Rank Predicted larger clusters
Predicted larger clusters
Larger

27
Estimate total nodes for more clusters
Animal transportation problem
So What? Kafka and Zipf’s Law (2)
How many animals can fit in a boat? Public Domain

28
Total weight of animals on Ark (assuming Elephant is the largest) tends to 90 tonnes
If you know the size of the biggest thing you can predict the total size

29
Only increases total nodes by 25%
Doubling number of Kafka clusters
0
500
1000
1500
2000
2500
3000
3500
4000
4500
5000
0 200 400 600 800 1000 1200 1400 1600 1800
Cumulative total nodes
100% more clusters
25%
more
nodes

Part 3 Kafka cluster storage
DALL·E 3
Storage for all Kafka clusters
Available from a recent project

31
Correlation coefficient between size and disk = 0.9
5.6 PB Total Disk across all Kafka clusters
Raw Data – total disk per cluster
0
50
100
150
200
250
300
350
400
450
500
0 20 40 60 80 100 120
Disk
(TB)
Nodes/cluster
Disk (TB) per cluster

32
• Disk space used is a function of average write rate x average message size x retention period x RF (Little’s
Law)
• Our metrics our total disk available, not used
• Some clusters are DEV not PROD – not real workloads, and RF may be < 3
• Approximation - number of nodes as a proxy for cluster size – actual instance sizes impact capacity
• Kafka log retention policy and time impact how many messages are retained
• Kafka clusters are sized for peak load not average load
• Some clusters may be older than others (disk can be increased)
• Write vs. Read workload imbalance
• Some clusters may have higher write workload rate (requiring more disk) vs.
• Higher read workload rates (requiring less disk)
What’s going on?
0
100
200
300
400
500
0 20 40 60 80 100 120
Disk
(TB)
Nodes/cluster
Disk (TB) per cluster

Part 4
Performance Metrics for
Top Ten Kafka Clusters
Top 10 tallest buildings (Wikipedia)

But in reality more people are killed by horses, cows, dogs,
and bees than kangaroos, sharks, snakes, crocodiles,
emus, jellyfish, etc!
Most Dangerous
Australian Critters?
Ranking can be tricky
Most “dangerous” = most teeth? Most venomous?
(Paul Brebner) (Wikimedia)

36
• For all clusters
• Size (number of nodes) and type
• Disk (from extra project)
• Performance Metrics are collected for all clusters
• But not easily available as the focus is per-cluster operations
• Requested Performance Metrics for Top Ten Clusters
• What did I get?
• Static (per cluster):
• Nodes, Topics, Partitions
• For 24 hours (per broker):
• Resource Utilisation: CPU (avg, max)
• Throughput: Bytes in (avg, max), Bytes out (avg, max), Messages in (avg, max) [Have to scale by number of nodes to get cluster metrics]
• Performance: Producer and consumer latency (avg, p99)
What metrics are available for Kafka clusters?
Broker metrics need scaling to cluster metrics
Variation in broker metric values
24 hour sampling loses accuracy
24 hour sample size is limited/biased
Real workloads not benchmarking
Ten biggest clusters by node count only
Speculative Results!
Warning!

37
Min, Avg, Max
Summary Statistics: Nodes, Topics, Partitions
27
7
2598
56.4
429.7
92145.3
96
1755
508800
1
10
100
1000
10000
100000
1000000
Nodes Topics Partitions
Nodes, Topics, Partitions (Log)
Min Avg Max

38
Summary Statistics: CPU, GB/s in/out, Message/s (in)
2
0.396
0.12
24.5
3.14175
1.419
67.5
14.4
8.4
0.1
1
10
100
CPU Bytes in/out (GB/s) Messages in (M/s)
CPU, GB/s (in+out), Messages/s (in, M/s) (Log)
Min Avg Max

39
Producers faster than Consumers
Note that some clusters use EBS, others use SSDs (faster!)
Summary Statistics: Latency (ms)
0.075
6.5
3.2925
106.65
90
700
0.01
0.1
1
10
100
1000
Producer latency (ms) Consumer latency (ms)
Latency (Log)
Min Avg Max

40
50% of clusters have sub 50ms average latency
Consumer latency distribution
0
50
100
150
200
250
300
350
1 2 3 4 5 6 7 8 9 10
Latency distribution (ms) – increasing

41
150-3k Bytes
Summary Statistics: Message size (Bytes)
150
1163.950072
3000
0
500
1000
1500
2000
2500
3000
3500
Message size (avg, Bytes)
Message size (avg, Bytes)
min avg max

42
0.4 to 25 Million/s
Using Average message size, compute messages out à total messages in+out
0
5
10
15
20
25
30
Msgs in+out (M/s)
Msgs in+out (M/s)
min avg max

43
1.4 to 28 – i.e. 28 consumer groups potentially
Fan out (ratio of consumer to producer messages)
0
5
10
15
20
25
30
Fan out
Fan out
min avg max

44
Knowing metrics for top 10 clusters we can estimate total values for ALL CLUSTERS
27K topics (probably underestimate), 5.8 M partitions; 321-564 Million messages/s
Assuming Zipf distribution…
27.45051596
5.886516239
321.3248554
564.9712845
1
10
100
1000
1
Grand Totals for All Kafka Clusters
Topics (k) Partitions (M) Msgs in+out (avg, M/s) Msgs in+out (max, M/s)

45
Nodes – 27 to 96 (1% of clusters, 564 nodes total, 16% of total nodes overall)
Static data – top 10 clusters (largest on right)
27
36 36
48
51
60 60
72
78
96
0
20
40
60
80
100
120
1 2 3 4 5 6 7 8 9 10
Nodes/Cluster

46
Ranges, odd ones out
Biggest (10) cluster has most partitions; cluster 6 has “hottest” topics (max partitions/topic)
Topics/Partitions/Nodes
7
631
13
1337
57 7 27 101
362
1755
0
500
1000
1500
2000
1 2 3 4 5 6 7 8 9 10
Topics/Cluster
6675
57672
2598
200490
11940 11394 13038 23046
85800
508800
0
100000
200000
300000
400000
500000
600000
1 2 3 4 5 6 7 8 9 10
Partitions/Cluster
0
200
400
600
800
1000
1200
1400
1600
1800
1 2 3 4 5 6 7 8 9 10
Partitions/Topic
0
200
400
600
800
1000
1200
1400
1600
1800
2000
1 2 3 4 5 6 7 8 9 10
Partitions/Node
Most topics Most partitions
Hottest topics

47
Cluster 4 has highest max = highest topics/partitions per cluster/node
Cluster 6 has highest average = highest partitions/topic (“hot” topics)
These are both ”hot” clusters
CPU
0%
10%
20%
30%
40%
50%
60%
70%
80%
1 2 3 4 5 6 7 8 9 10
CPU (Avg, max)
CPU (Avg) CPU (max)
Hottest
Hot

48
Topics? Theory and our Technical operations people say probably not
as topics are not correlated to throughput (or size)
Correlation = 0.4, some known smaller clusters with way more topics (e.g. 10,000!)
Any obvious correlations to cluster size?
0
200
400
600
800
1000
1200
1400
1600
1800
2000
0 20 40 60 80 100 120
Total topics in cluster

49
Partitions are related to throughput and size in theory
Correlation = 0.63, and the largest cluster has most and above average partitions/nodes
Size/Partition correlation?
0
100000
200000
300000
400000
500000
600000
0 20 40 60 80 100 120
Total partitions

50
Average – poor correlation
Size/Throughput?
0
5000000
10000000
15000000
20000000
25000000
0 20 40 60 80 100 120
Msgs in+out (avg/s)

51
Max – poor correlation
But avg & peak TP correlates with “hot” cluster
Real workloads in 24 hour sample period don’t necessarily correlate with cluster capacities
Size/Throughput?
0
5000000
10000000
15000000
20000000
25000000
30000000
0 20 40 60 80 100 120
Msgs in+out (max/s)

52
• AWS ARM Graviton2 R6g high price performance for memory-intensive workloads
• R6g.4xlarge 16 core (EBS) (4 clusters)
• R6g.2xlarge 8 cores (EBS) (2 clusters)
• AWS ARM Graviton2 Im4gn Nitro SSD for I/O intensive workloads
• Im4gn.4xlarge 16 core SSD (2 clusters, including “hot” cluster)
• AWS ARM Graviton2 M6g for balanced workloads
• M6gd.4xlarge 16 cores SSD (1 cluster)
• AWS x86 I3en for data-intensive workloads
• I3en.3xlarge 12 cores SSD (1 cluster)
A mix of EC2 instance types/sizes (4/5) and storage - EBS (6)/SSD (4)
Top 10 clusters have heterogeneous h/w

53
Good correlation (0.8) – definite increase in total cores for bigger clusters
Cores per Cluster
0
200
400
600
800
1000
1200
1400
1600
1800
0 20 40 60 80 100 120
Cores
per
cluster
Nodes per cluster
Cores per cluster

54
• Insights from our Techops team – thanks!
• Biggest cluster (#10)
• Over provisioned, 96 nodes, 1536 cores
• EBS (slow)
• Peak in messages/s = 1M/s
• Consumer latency 200 - 400ms
• Runs “cool” (18-45%)
• Most partitions (0.5088 Million)
• Hottest cluster (#6)
• 60 nodes, 960 cores
• Runs “hot” (45-55%)
• But lowest consumer latency
• Faster SSDs
• Few topics, most partitions/topic (hot “topics”)
Drill down
Biggest cluster vs “hottest” cluster

55
Average for cluster = 290 ms but actually a large variation across brokers
Also illustrates that metrics are per broker – and have wide variability

56
For target throughput how many cores and partitions are needed (in practice need both)?
Can only predict a range from this data (avg=conservative; max=optimistic)
Capacity Planning
6288.039891
431.386635
25583.88158
2155.566642
0
5000
10000
15000
20000
25000
30000
Msgs/s per core Msgs/s per partition
Msgs/s per core and partition
Avg Max

57
Range: Avg (conservative), Max (optimistic)
Cores for target throughput (x2 max current cluster)
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
0 10 20 30 40 50 60
TPS (Million/s) vs Cores
Cores (avg) Cores (max)

58
Range: Avg (conservative) max (optimistic)
Note: This is probably skewed due to large cluster with most partitions having low throughput
and “hot” cluster with highest throughput having few partitions!
Partitions for target throughput (x2 max current cluster)
0
20000
40000
60000
80000
100000
120000
0 10 20 30 40 50 60
TPS (Million/s) vs Partitions
Partitions (avg) Partitions (max)

59
• Lots of small clusters
• Few big clusters
• Even bigger clusters are likely
• A wide distribution of sizes is observed
• Kafka is horizontally scalable
• Fits many different customer workloads
• Some customers have many smaller clusters
• Some clusters grow in size over time
Conclusions?
Kafka cluster size distribution is Zipfian
DALL·E 3

61
• Wide range of workloads, throughputs, hot vs cold CPU, fan-outs, latency,
message size and hardware
• Some interesting “odd ones out”
• Biggest
• Hottest
• Performance metrics were
• biased & coarse grain
• due to broker level collection and 24 hour sample & average & summary
• and from real workloads not benchmarks
• Hard to find correlations and make accurate predictions
• Some broad correlations and range predictions possible
Conclusions?
Top 10 clusters are “diverse”
(Paul Brebner)
Adolf Hoffmeister & Franz Kafka (Wikimedia)

63
• Is normal for our managed Kafka clusters
• Usage/workload varies widely for customers
• Including topics, partitions, throughput, message sizes, client
settings (e.g. batching), fan-out, latency SLAs etc
• Many bigger clusters are dedicated to very specific customer
workloads
• Higher throughput clusters are not representative of lower
throughput clusters
• Hardware varies and is optimized/customized to take into
account specific customer workloads, cost and SLA
requirements
Conclusions?
Custom Cluster Optimization and Sizing
DALL·E 3

64
• Performance prediction from coarse-grained
metrics feels like Déjà vu
• 2007-2017 I developed an automated approach
to Performance Modelling from distributed
application traces
• This could work for Kafka
• Instrument Apache Kafka source code with
OpenTelemetry to provide
• Kafka specific resource (CPU, IO, network) + time spans
• Run Kafka benchmarks on representative hardware
• Transform OT traces into a performance model
• Make more accurate predictions
Conclusions?
Performance Prediction
DALL·E 3

65
What next?
• Try us out!
• Free 30 day trial
• Developer size clusters
• www.instaclustr.com
• All my blogs (100+):
• https://instaclustr.com/paul-brebner

Thank you

Why Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandaries Explored)

Related slideshows

More Related Content

Why Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandaries Explored)