Raise Kafka Streams transaction timeout above the commit interval

Under exactly_once_v2, Streams defaults the producer transaction timeout
to 10s. With commit.interval.ms raised to 10s (665cd1a), transactions on
the heavy map-aggregation topology timed out, got fenced
(ProducerFencedException / TaskMigratedException ~130 times during one
large chunk import), and their work was redone. Default the transaction
timeout to 60s, overridable via KAFKA_STREAMS_TRANSACTION_TIMEOUT_MS.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Lazzexus 2026-10-13 07:00:06 +02:00
parent 9067694172
commit e183ccd1fc

View file

@ -23,6 +23,9 @@ kafka-streams-config:
# repeated updates to the same (large) map chunk be merged before being forwarded.
commit.interval.ms: ${KAFKA_STREAMS_COMMIT_INTERVAL_MS:-100}
cache.max.bytes.buffering: ${KAFKA_STREAMS_CACHE_MAX_BYTES:-10485760}
# Under exactly_once_v2 Streams defaults the transaction timeout to 10s. It must be well above
# commit.interval.ms, or transactions time out, get fenced, and the work is redone.
producer.transaction.timeout.ms: ${KAFKA_STREAMS_TRANSACTION_TIMEOUT_MS:-60000}
processing.guarantee: exactly_once_v2
auto.offset.reset: earliest
default.key.serde: org.apache.kafka.common.serialization.Serdes$StringSerde