Under exactly_once_v2, Streams defaults the producer transaction timeout
to 10s. With commit.interval.ms raised to 10s (665cd1a), transactions on
the heavy map-aggregation topology timed out, got fenced
(ProducerFencedException / TaskMigratedException ~130 times during one
large chunk import), and their work was redone. Default the transaction
timeout to 60s, overridable via KAFKA_STREAMS_TRANSACTION_TIMEOUT_MS.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The zoom-out map layers aggregate 32x32 chunks into 256x256 and 512x512
chunks, holding up to 65k/262k tiles. Every incoming chunk rebuilt the
whole aggregate, and with exactly_once_v2's default 100ms commit
interval nearly every intermediate result was forwarded downstream,
cascading into the next level. A small map also has few 512-level keys,
so this work landed on one partition and one thread.
Make commit.interval.ms and cache.max.bytes.buffering configurable
(defaults unchanged), and in the Unraid example use a 10s commit
interval with a 128MB cache, so repeated updates to the same chunk are
merged in the record cache first. Measured on a live backlog: 41 -> 575
chunks/minute. Map updates now appear within ~10s instead of instantly.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- config.yml: num.stream.threads can be set with
KAFKA_STREAMS_NUM_THREADS (default unchanged: 1). Map processing was
bottlenecked on a single stream thread while the host was mostly idle.
- Unraid example: run 3 stream threads (topics have 3 partitions), and
cap each JVM's heap via JAVA_OPTS. Without a cap each JVM defaults to
1/4 of the host's RAM, and the stack grew to ~14 GB.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>