Kafkatorio/modules
Lazzexus 665cd1a7d6 Speed up map processing 14x by merging aggregate updates before forwarding
The zoom-out map layers aggregate 32x32 chunks into 256x256 and 512x512
chunks, holding up to 65k/262k tiles. Every incoming chunk rebuilt the
whole aggregate, and with exactly_once_v2's default 100ms commit
interval nearly every intermediate result was forwarded downstream,
cascading into the next level. A small map also has few 512-level keys,
so this work landed on one partition and one thread.

Make commit.interval.ms and cache.max.bytes.buffering configurable
(defaults unchanged), and in the Unraid example use a 10s commit
interval with a 128MB cache, so repeated updates to the same chunk are
merged in the record cache first. Measured on a live backlog: 41 -> 575
chunks/minute. Map updates now appear within ~10s instead of instantly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 19:43:17 +02:00
..
events-library Fix kxs-ts-gen-core dependency resolution and unblocked compile bugs 2026-10-12 15:40:33 +02:00
events-mod Fix '/kafkatorio chunks' not sending most chunks 2026-10-12 18:16:20 +02:00
events-processor-core Speed up map processing 14x by merging aggregate updates before forwarding 2026-10-12 19:43:17 +02:00
events-processors big ol' refactor of the Gradle Factorio Mod 2022-12-18 01:41:14 +01:00
events-server [docker-files] remove user config 2022-10-22 23:36:00 +02:00
events-server-syslog Fix syslog server silently dropping packets under load 2026-10-12 18:58:21 +02:00
events-server-web Fix web map blanking and websocket dropping behind a reverse proxy 2026-10-12 18:09:03 +02:00
infra-factorio-client tidy up after refactoring Gradle Factorio plugin 2022-12-19 16:53:18 +01:00
infra-factorio-server Port events-mod to Factorio 2.0.77, declare SE/Krastorio 2 compatibility 2026-10-12 15:06:30 +02:00
infra-kafka-cluster Fix Docker image builds and local platform setup 2026-10-12 17:27:11 +02:00
infra-kafka-pipe update dockerfiles 2022-10-22 19:55:02 +02:00
kt-rcon migrate buildSrc plugins to includedBuild plugins 2022-09-18 20:49:32 +02:00
versions-platform adjust geedeecee config, so images can be built 2022-10-05 00:06:46 +02:00
web-map Fix web map blanking and websocket dropping behind a reverse proxy 2026-10-12 18:09:03 +02:00
modules.md big ol' refactor to split up the events-server and events-processors 2022-09-13 17:56:57 +02:00