Commit graph

12 commits

Author SHA1 Message Date
Lazzexus
ba9cbe5afe Reduce memory use of the JVM services
The services now run on Java 17, with a periodic G1 GC that returns unused
heap to the OS when idle. Previously Java 11 kept the committed heap, so
events-processors held 5.5 GB while its live data was ~250 MB.

All RocksDB state stores now share one bounded block cache and write buffer
(ROCKSDB_TOTAL_MEMORY_MB, default 256 MB), instead of each of the ~45 stores
having its own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 10:31:04 +01:00
Lazzexus
9d1f9b7623 Show building rotation, recipes and other surfaces on the web map
The mod now sends each building's direction and recipe (furnaces: the last
recipe, since they only have one while working).
- Rotated buildings are drawn at their real footprint.
- Crafting machines show their recipe icon, like Factorio's alt mode, and
  the hover tooltip includes the recipe.

The web map has a surface drop-down (e.g. Space Exploration planets and
orbits), backed by a new surfaces endpoint. Every layer, the resource labels
and the player markers follow the chosen surface.

Resource labels use the in-game names (e.g. "Iridite" instead of
"Se iridium ore"). export_building_icons.py also exports recipe icons, plus
recipe and resource names.

The three in-memory views in events-server-web now share one helper for
following a Kafka topic.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 10:14:39 +02:00
Lazzexus
c773b05922 Add a building icon layer to the web map
events-server-web keeps the latest state of every building in memory (read
from the map-chunk-entity topic, like the resource patches) and serves them
per area. The web map draws the icons at their real size onto canvas tiles,
so large bases stay fast, and shows each building's name, status and health
on hover. Ghosts are ignored, and buildings are removed when they're mined.

The icons belong to Wube and the mod authors, so they aren't in the repo.
export_building_icons.py builds them from a Factorio client's
--dump-data/--dump-prototype-locale/--dump-icon-sprites output, and the
web-map nginx serves them from a mounted volume.

Also label single-well oil fields "1 well".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 09:52:07 +02:00
Lazzexus
424e12309d Speed up map processing 14x by merging aggregate updates before forwarding
The zoom-out map layers aggregate 32x32 chunks into 256x256 and 512x512
chunks, holding up to 65k/262k tiles. Every incoming chunk rebuilt the
whole aggregate, and with exactly_once_v2's default 100ms commit
interval nearly every intermediate result was forwarded downstream,
cascading into the next level. A small map also has few 512-level keys,
so this work landed on one partition and one thread.

Make commit.interval.ms and cache.max.bytes.buffering configurable
(defaults unchanged), and in the Unraid example use a 10s commit
interval with a 128MB cache, so repeated updates to the same chunk are
merged in the record cache first. Measured on a live backlog: 41 -> 575
chunks/minute. Map updates now appear within ~10s instead of instantly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 19:43:17 +02:00
Lazzexus
5ad3e6f27b Make Kafka Streams thread count configurable, cap JVM heaps on Unraid
- config.yml: num.stream.threads can be set with
  KAFKA_STREAMS_NUM_THREADS (default unchanged: 1). Map processing was
  bottlenecked on a single stream thread while the host was mostly idle.
- Unraid example: run 3 stream threads (topics have 3 partitions), and
  cap each JVM's heap via JAVA_OPTS. Without a cap each JVM defaults to
  1/4 of the host's RAM, and the stack grew to ~14 GB.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 19:21:22 +02:00
Lazzexus
52fb9e5773 Unraid example: keep Kafka data off Unraid's FUSE layer
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 18:46:13 +02:00
Lazzexus
47ee084eab Add Unraid deployment example to docs
A compose file for running the Kafkatorio services on the same Unraid
server as the Factorio server, via the Compose Manager plugin:

- syslog ingest bound to 127.0.0.1 only (the Docker syslog log driver
  connects from the host), Kafka not exposed
- persistent Kafka data, with a fixed cluster ID so the data survives
  image rebuilds
- the web map is the single public entry point

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 18:09:03 +02:00
Lazzexus
e52ea9d9cf Fix Docker image builds and local platform setup
Getting the backend running locally in Docker surfaced several issues:

- geedeecee: REGISTRY_HOST came from the unset Gradle property
  'dockerContainerRegistryHost'. A missing value in a MapProperty makes
  the whole map absent, so every generated .env was empty and all
  docker-compose builds failed. Default it to 'localhost'.
- kafka image: dlcdn.apache.org only hosts current releases, so Kafka
  3.3.1 404s. Use archive.apache.org. Replace the deprecated openjdk:11
  base with eclipse-temurin:11-jre, bump the download stage from EOL
  Debian 11 to 12, and drop 'COPY --chmod' so the image builds without
  BuildKit/buildx.
- web-map: the map calls the API and websocket on its own origin
  (/kafkatorio/...), which previously relied on an external reverse
  proxy. Proxy /kafkatorio/ to events-server-web in the map's nginx so
  one port serves everything, and publish that port in the example
  platform compose file.
- docs: the example secret config mapped server IDs to descriptions,
  but the schema is token (JWT 'sub') -> server ID, and both must match
  [a-zA-Z0-9-_]. Descriptions with spaces crashed every service.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 17:27:11 +02:00
Adam
76a661e444 move logo, add function to auto-init the project logo 2022-10-30 18:44:47 +01:00
Adam
5bb8cc789c add/update docs 2022-10-30 17:21:32 +01:00
Adam
4cb00febc2 rm docs/notes/question.md - not needed 2022-07-17 23:50:33 +02:00
Adam
08ab1c582c tidying up 2021-12-28 17:05:52 +01:00