Kafkatorio/docs
Lazzexus 665cd1a7d6 Speed up map processing 14x by merging aggregate updates before forwarding
The zoom-out map layers aggregate 32x32 chunks into 256x256 and 512x512
chunks, holding up to 65k/262k tiles. Every incoming chunk rebuilt the
whole aggregate, and with exactly_once_v2's default 100ms commit
interval nearly every intermediate result was forwarded downstream,
cascading into the next level. A small map also has few 512-level keys,
so this work landed on one partition and one thread.

Make commit.interval.ms and cache.max.bytes.buffering configurable
(defaults unchanged), and in the Unraid example use a 10s commit
interval with a 128MB cache, so repeated updates to the same chunk are
merged in the record cache first. Measured on a live backlog: 41 -> 575
chunks/minute. Map updates now appear within ~10s instead of instantly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 19:43:17 +02:00
..
kafkatorio-platform Speed up map processing 14x by merging aggregate updates before forwarding 2026-10-12 19:43:17 +02:00
media/img move logo, add function to auto-init the project logo 2022-10-30 18:44:47 +01:00
guide.md Add Unraid deployment example to docs 2026-10-12 18:09:03 +02:00