Speed up map processing 14x by merging aggregate updates before forwarding

The zoom-out map layers aggregate 32x32 chunks into 256x256 and 512x512
chunks, holding up to 65k/262k tiles. Every incoming chunk rebuilt the
whole aggregate, and with exactly_once_v2's default 100ms commit
interval nearly every intermediate result was forwarded downstream,
cascading into the next level. A small map also has few 512-level keys,
so this work landed on one partition and one thread.

Make commit.interval.ms and cache.max.bytes.buffering configurable
(defaults unchanged), and in the Unraid example use a 10s commit
interval with a 128MB cache, so repeated updates to the same chunk are
merged in the record cache first. Measured on a live backlog: 41 -> 575
chunks/minute. Map updates now appear within ~10s instead of instantly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Lazzexus 2026-10-12 19:43:17 +02:00
parent f02e1a783e
commit 665cd1a7d6
2 changed files with 8 additions and 0 deletions

View file

@ -26,6 +26,9 @@ services:
KAFKA_BOOTSTRAP_SERVERS: "kafka:29092"
# one stream thread per partition (topics have 3) - parallelises map processing
KAFKA_STREAMS_NUM_THREADS: "3"
# merge repeated map-chunk updates for 10s before forwarding them (128MB cache per topology)
KAFKA_STREAMS_COMMIT_INTERVAL_MS: "10000"
KAFKA_STREAMS_CACHE_MAX_BYTES: "134217728"
JAVA_OPTS: "-Xmx4g"
networks: [ kafkatorio-nw ]
depends_on: [ infra-kafka ]

View file

@ -18,6 +18,11 @@ kafka-streams-config:
bootstrap.servers: ${KAFKA_BOOTSTRAP_SERVERS:-localhost:9092}
state.dir: ${KAFKA_STREAMS_STATE_DIR:-.state/kafka}
num.stream.threads: ${KAFKA_STREAMS_NUM_THREADS:-1}
# With exactly_once_v2 the default commit interval is 100ms, so almost every intermediate
# aggregate update is forwarded downstream. A longer interval and a larger record cache let
# repeated updates to the same (large) map chunk be merged before being forwarded.
commit.interval.ms: ${KAFKA_STREAMS_COMMIT_INTERVAL_MS:-100}
cache.max.bytes.buffering: ${KAFKA_STREAMS_CACHE_MAX_BYTES:-10485760}
processing.guarantee: exactly_once_v2
auto.offset.reset: earliest
default.key.serde: org.apache.kafka.common.serialization.Serdes$StringSerde