The zoom-out map layers aggregate 32x32 chunks into 256x256 and 512x512
chunks, holding up to 65k/262k tiles. Every incoming chunk rebuilt the
whole aggregate, and with exactly_once_v2's default 100ms commit
interval nearly every intermediate result was forwarded downstream,
cascading into the next level. A small map also has few 512-level keys,
so this work landed on one partition and one thread.
Make commit.interval.ms and cache.max.bytes.buffering configurable
(defaults unchanged), and in the Unraid example use a 10s commit
interval with a 128MB cache, so repeated updates to the same chunk are
merged in the record cache first. Measured on a live backlog: 41 -> 575
chunks/minute. Map updates now appear within ~10s instead of instantly.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- config.yml: num.stream.threads can be set with
KAFKA_STREAMS_NUM_THREADS (default unchanged: 1). Map processing was
bottlenecked on a single stream thread while the host was mostly idle.
- Unraid example: run 3 stream threads (topics have 3 partitions), and
cap each JVM's heap via JAVA_OPTS. Without a cap each JVM defaults to
1/4 of the host's RAM, and the stack grew to ~14 GB.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A compose file for running the Kafkatorio services on the same Unraid
server as the Factorio server, via the Compose Manager plugin:
- syslog ingest bound to 127.0.0.1 only (the Docker syslog log driver
connects from the host), Kafka not exposed
- persistent Kafka data, with a fixed cluster ID so the data survives
image rebuilds
- the web map is the single public entry point
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Getting the backend running locally in Docker surfaced several issues:
- geedeecee: REGISTRY_HOST came from the unset Gradle property
'dockerContainerRegistryHost'. A missing value in a MapProperty makes
the whole map absent, so every generated .env was empty and all
docker-compose builds failed. Default it to 'localhost'.
- kafka image: dlcdn.apache.org only hosts current releases, so Kafka
3.3.1 404s. Use archive.apache.org. Replace the deprecated openjdk:11
base with eclipse-temurin:11-jre, bump the download stage from EOL
Debian 11 to 12, and drop 'COPY --chmod' so the image builds without
BuildKit/buildx.
- web-map: the map calls the API and websocket on its own origin
(/kafkatorio/...), which previously relied on an external reverse
proxy. Proxy /kafkatorio/ to events-server-web in the map's nginx so
one port serves everything, and publish that port in the example
platform compose file.
- docs: the example secret config mapped server IDs to descriptions,
but the schema is token (JWT 'sub') -> server ID, and both must match
[a-zA-Z0-9-_]. Descriptions with spaces crashed every service.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>