Commit graph

15 commits

Author SHA1 Message Date
Lazzexus
d5f722b998 Raise Kafka Streams transaction timeout above the commit interval
Under exactly_once_v2, Streams defaults the producer transaction timeout
to 10s. With commit.interval.ms raised to 10s (665cd1a), transactions on
the heavy map-aggregation topology timed out, got fenced
(ProducerFencedException / TaskMigratedException ~130 times during one
large chunk import), and their work was redone. Default the transaction
timeout to 60s, overridable via KAFKA_STREAMS_TRANSACTION_TIMEOUT_MS.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 07:00:06 +02:00
Lazzexus
424e12309d Speed up map processing 14x by merging aggregate updates before forwarding
The zoom-out map layers aggregate 32x32 chunks into 256x256 and 512x512
chunks, holding up to 65k/262k tiles. Every incoming chunk rebuilt the
whole aggregate, and with exactly_once_v2's default 100ms commit
interval nearly every intermediate result was forwarded downstream,
cascading into the next level. A small map also has few 512-level keys,
so this work landed on one partition and one thread.

Make commit.interval.ms and cache.max.bytes.buffering configurable
(defaults unchanged), and in the Unraid example use a 10s commit
interval with a 128MB cache, so repeated updates to the same chunk are
merged in the record cache first. Measured on a live backlog: 41 -> 575
chunks/minute. Map updates now appear within ~10s instead of instantly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 19:43:17 +02:00
Lazzexus
5ad3e6f27b Make Kafka Streams thread count configurable, cap JVM heaps on Unraid
- config.yml: num.stream.threads can be set with
  KAFKA_STREAMS_NUM_THREADS (default unchanged: 1). Map processing was
  bottlenecked on a single stream thread while the host was mostly idle.
- Unraid example: run 3 stream threads (topics have 3 partitions), and
  cap each JVM's heap via JAVA_OPTS. Without a cap each JVM defaults to
  1/4 of the host's RAM, and the stack grew to ~14 GB.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 19:21:22 +02:00
Adam
3f771f9838 fix bootstrap server, remove http:// 2022-10-23 21:54:38 +02:00
Adam
ce0d4b7378 [events-processors] change where topo description is written 2022-10-22 19:52:02 +02:00
Adam
f99208b5b2 more fixing of Kafkatorio packets... 2022-10-18 23:22:26 +02:00
Adam
dda48bac78 refactor the KafkatorioPacket tag that the mod logs, and the syslog-server parses 2022-10-18 21:35:49 +02:00
Adam
1cc5afa6e2 set default socket-server host to 0.0.0.0 2022-10-17 00:12:03 +02:00
Adam
b0722487a0 fix appProps in events-server-web 2022-10-08 14:20:01 +02:00
Adam
6035c2eaa0 move kafka topics creator into events-processors 2022-10-08 14:16:43 +02:00
Adam
46db151f6e rename SERVER_PORT -> WEB_SERVER_PORT 2022-10-02 14:03:51 +02:00
Adam
f12a505d22 migrate buildSrc plugins to includedBuild plugins 2022-09-18 20:49:32 +02:00
Adam
bb973c7b9b fix tests 2022-09-16 22:31:32 +02:00
Adam
873983f9a3 big ol' refactor to split up the events-server and events-processors 2022-09-13 17:56:57 +02:00
Adam
c8815726b1 start splitting up events-server 2022-09-11 00:48:16 +02:00