Commit graph

9 commits

Author SHA1 Message Date
665cd1a7d6 Speed up map processing 14x by merging aggregate updates before forwarding
The zoom-out map layers aggregate 32x32 chunks into 256x256 and 512x512
chunks, holding up to 65k/262k tiles. Every incoming chunk rebuilt the
whole aggregate, and with exactly_once_v2's default 100ms commit
interval nearly every intermediate result was forwarded downstream,
cascading into the next level. A small map also has few 512-level keys,
so this work landed on one partition and one thread.

Make commit.interval.ms and cache.max.bytes.buffering configurable
(defaults unchanged), and in the Unraid example use a 10s commit
interval with a 128MB cache, so repeated updates to the same chunk are
merged in the record cache first. Measured on a live backlog: 41 -> 575
chunks/minute. Map updates now appear within ~10s instead of instantly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 19:43:17 +02:00
f02e1a783e Make Kafka Streams thread count configurable, cap JVM heaps on Unraid
- config.yml: num.stream.threads can be set with
  KAFKA_STREAMS_NUM_THREADS (default unchanged: 1). Map processing was
  bottlenecked on a single stream thread while the host was mostly idle.
- Unraid example: run 3 stream threads (topics have 3 partitions), and
  cap each JVM's heap via JAVA_OPTS. Without a cap each JVM defaults to
  1/4 of the host's RAM, and the stack grew to ~14 GB.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 19:21:22 +02:00
5f7a631936 Unraid example: keep Kafka data off Unraid's FUSE layer
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 18:46:13 +02:00
6cb070fa63 Add Unraid deployment example to docs
A compose file for running the Kafkatorio services on the same Unraid
server as the Factorio server, via the Compose Manager plugin:

- syslog ingest bound to 127.0.0.1 only (the Docker syslog log driver
  connects from the host), Kafka not exposed
- persistent Kafka data, with a fixed cluster ID so the data survives
  image rebuilds
- the web map is the single public entry point

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 18:09:03 +02:00
cae2eb09c6 Fix Docker image builds and local platform setup
Getting the backend running locally in Docker surfaced several issues:

- geedeecee: REGISTRY_HOST came from the unset Gradle property
  'dockerContainerRegistryHost'. A missing value in a MapProperty makes
  the whole map absent, so every generated .env was empty and all
  docker-compose builds failed. Default it to 'localhost'.
- kafka image: dlcdn.apache.org only hosts current releases, so Kafka
  3.3.1 404s. Use archive.apache.org. Replace the deprecated openjdk:11
  base with eclipse-temurin:11-jre, bump the download stage from EOL
  Debian 11 to 12, and drop 'COPY --chmod' so the image builds without
  BuildKit/buildx.
- web-map: the map calls the API and websocket on its own origin
  (/kafkatorio/...), which previously relied on an external reverse
  proxy. Proxy /kafkatorio/ to events-server-web in the map's nginx so
  one port serves everything, and publish that port in the example
  platform compose file.
- docs: the example secret config mapped server IDs to descriptions,
  but the schema is token (JWT 'sub') -> server ID, and both must match
  [a-zA-Z0-9-_]. Descriptions with spaces crashed every service.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 17:27:11 +02:00
Adam
76a661e444 move logo, add function to auto-init the project logo 2022-10-30 18:44:47 +01:00
Adam
5bb8cc789c add/update docs 2022-10-30 17:21:32 +01:00
Adam
4cb00febc2 rm docs/notes/question.md - not needed 2022-07-17 23:50:33 +02:00
Adam
08ab1c582c tidying up 2021-12-28 17:05:52 +01:00