Upgrade Kafka Plugin to Use Kafka Client 4.0 #195
Description
Summary
Upgrade to Kafka Client 4.0, enabling KIP‑848 new consumer rebalance protocol.
This change removes “stop-the-world” pauses and supports true dynamic scaling of consumer groups—critical for resilient, low-latency ingestion pipelines.
Context & Problem
Our pipelines rely heavily on Logstash + Kafka. With the 3.x client, any consumer restart or failure triggers a full-group rebalance, pausing processing for all consumers. Despite tuning and cooperative protocols, outages persist and dynamic scaling is effectively impossible without interruption.
Kafka 4.0 introduces KIP‑848, a broker-driven, incremental rebalance protocol that:
- Eliminates global sync barriers—pausing only the partitions that are reassigned
- Enables incremental adds/removals without disrupting other consumers
- Simplifies client logic by migrating coordination to brokers
Benefits:
- No full-group stalls during consumer restarts
- Seamless, true elastic scaling
- Reduced operational burden and better SLA alignment
- Alignment with Kafka ecosystem (KRaft mode, ZooKeeper removal, modern Java usage)