Commit graph

16 commits

Author SHA1 Message Date
Lazzexus
47ab9f30fc Add a forwarder for Factorio servers that don't run in Docker
The mod prints its packets to the server's standard output, and Docker's
syslog log driver sends them to Kafkatorio. kafkatorio-forward.py does
the same without Docker: it runs the server (or reads its output from a
pipe), prints the output as usual, and sends the mod's packets to the
syslog port in the same format as Docker, tagged with the token.

- the server keeps running when Kafkatorio is unreachable: packets wait in
  a bounded buffer, and are sent after reconnecting
- before each send it checks Kafkatorio hasn't closed the connection, so
  lines aren't silently lost in a dead socket
- Ctrl+C and SIGTERM are passed to the server once, so it saves and stops
- the token is read from a file or the environment, not the command line
- only the Python standard library

util-linux's logger can't be used instead: it rejects the token, which is
longer than syslog's 48 character APP-NAME limit.

Tested on Linux with a Factorio 2.0.77 server: packets arrive, the
forwarder reconnects when the syslog server restarts (no packets lost),
and the server saves when stopped.

The guide has a section for it, with a systemd service example, and the
README no longer says the Factorio server must run in Docker.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-22 11:00:24 +01:00
Lazzexus
1f4786c1bd Rewrite the README and setup guide, and fix the example Compose setup
The README describes what Kafkatorio does now (the live map, production
and research pages), that this is a Factorio 2.0 fork, and how the
services fit together.

The guide is rewritten as numbered steps, from building the images to a
public website: configuration, creating the server's token locally
(instead of typing the secret into jwt.io), connecting the Factorio
server's log, sending the existing map, exporting icons and names, and
the reverse proxy. Also: the Unraid setup, updating, and troubleshooting.

It warns that the mod portal's kafkatorio-events is the original, for
Factorio 1.1, with overlapping version numbers, so players must install
the mod zip by hand.

The example docker-compose.yml lost all data when containers were
recreated (e.g. on every update), and exposed Kafka, which has no
authentication, on all interfaces. It's now based on the Unraid setup:
Kafka data, map tiles, state and icons are kept next to it, only the
website is exposed, and the optional Factorio server (--profile factorio)
starts after the syslog server.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-22 10:36:01 +01:00
Lazzexus
0e1b315a6d Add a production dashboard
Every minute, the mod diffs the force's production statistics for each
surface against the totals it sent last time (kept in
storage.productionTotals), and sends what was produced and consumed as a
ProductionUpdate.

events-server-web keeps the statistics in ring buffers (1440 minutes and
168 hours), and serves them at
/kafkatorio/data/servers/{id}/production, for a range of game time and
optionally one surface.

The web map has a production page (/servers/{id}/production, linked from
the header) with science packs used per minute (in total and per pack),
production per minute as a line chart, and a searchable table of
everything produced and consumed.

export_building_icons.py now also exports every item and fluid icon and
name (items.json, fluids.json), for the production page.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-22 10:26:25 +01:00
Lazzexus
8770a0617b Add a research page, and keep the topics that views are rebuilt from
The mod sends the research state of a force (researched technologies,
levels, current research, progress and queue) whenever research changes,
and every minute for the progress. events-server-web keeps the latest state
and serves it at /kafkatorio/data/servers/{id}/research.

The web map has a research page (/servers/{id}/research, linked from the
header), with:
- progress, the current research and the queue
- the science packs needed for everything that's left
- every technology, filterable by status, science pack and name, with its
  cost (including levelled technologies' count formulas)
- details: the total cost to reach a technology, its prerequisites, what it
  leads to and what it unlocks

The technology tree comes from export_building_icons.py, which now also
exports technologies.json, items.json and technology/science pack icons.

Kafka deleted data older than 7 days, but the building, resource and
research views are rebuilt by reading their topics from the beginning, so
buildings disappeared after a restart. These topics are now kept forever.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 10:46:09 +01:00
Lazzexus
ba9cbe5afe Reduce memory use of the JVM services
The services now run on Java 17, with a periodic G1 GC that returns unused
heap to the OS when idle. Previously Java 11 kept the committed heap, so
events-processors held 5.5 GB while its live data was ~250 MB.

All RocksDB state stores now share one bounded block cache and write buffer
(ROCKSDB_TOTAL_MEMORY_MB, default 256 MB), instead of each of the ~45 stores
having its own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 10:31:04 +01:00
Lazzexus
9d1f9b7623 Show building rotation, recipes and other surfaces on the web map
The mod now sends each building's direction and recipe (furnaces: the last
recipe, since they only have one while working).
- Rotated buildings are drawn at their real footprint.
- Crafting machines show their recipe icon, like Factorio's alt mode, and
  the hover tooltip includes the recipe.

The web map has a surface drop-down (e.g. Space Exploration planets and
orbits), backed by a new surfaces endpoint. Every layer, the resource labels
and the player markers follow the chosen surface.

Resource labels use the in-game names (e.g. "Iridite" instead of
"Se iridium ore"). export_building_icons.py also exports recipe icons, plus
recipe and resource names.

The three in-memory views in events-server-web now share one helper for
following a Kafka topic.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 10:14:39 +02:00
Lazzexus
c773b05922 Add a building icon layer to the web map
events-server-web keeps the latest state of every building in memory (read
from the map-chunk-entity topic, like the resource patches) and serves them
per area. The web map draws the icons at their real size onto canvas tiles,
so large bases stay fast, and shows each building's name, status and health
on hover. Ghosts are ignored, and buildings are removed when they're mined.

The icons belong to Wube and the mod authors, so they aren't in the repo.
export_building_icons.py builds them from a Factorio client's
--dump-data/--dump-prototype-locale/--dump-icon-sprites output, and the
web-map nginx serves them from a mounted volume.

Also label single-well oil fields "1 well".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 09:52:07 +02:00
Lazzexus
424e12309d Speed up map processing 14x by merging aggregate updates before forwarding
The zoom-out map layers aggregate 32x32 chunks into 256x256 and 512x512
chunks, holding up to 65k/262k tiles. Every incoming chunk rebuilt the
whole aggregate, and with exactly_once_v2's default 100ms commit
interval nearly every intermediate result was forwarded downstream,
cascading into the next level. A small map also has few 512-level keys,
so this work landed on one partition and one thread.

Make commit.interval.ms and cache.max.bytes.buffering configurable
(defaults unchanged), and in the Unraid example use a 10s commit
interval with a 128MB cache, so repeated updates to the same chunk are
merged in the record cache first. Measured on a live backlog: 41 -> 575
chunks/minute. Map updates now appear within ~10s instead of instantly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 19:43:17 +02:00
Lazzexus
5ad3e6f27b Make Kafka Streams thread count configurable, cap JVM heaps on Unraid
- config.yml: num.stream.threads can be set with
  KAFKA_STREAMS_NUM_THREADS (default unchanged: 1). Map processing was
  bottlenecked on a single stream thread while the host was mostly idle.
- Unraid example: run 3 stream threads (topics have 3 partitions), and
  cap each JVM's heap via JAVA_OPTS. Without a cap each JVM defaults to
  1/4 of the host's RAM, and the stack grew to ~14 GB.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 19:21:22 +02:00
Lazzexus
52fb9e5773 Unraid example: keep Kafka data off Unraid's FUSE layer
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 18:46:13 +02:00
Lazzexus
47ee084eab Add Unraid deployment example to docs
A compose file for running the Kafkatorio services on the same Unraid
server as the Factorio server, via the Compose Manager plugin:

- syslog ingest bound to 127.0.0.1 only (the Docker syslog log driver
  connects from the host), Kafka not exposed
- persistent Kafka data, with a fixed cluster ID so the data survives
  image rebuilds
- the web map is the single public entry point

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 18:09:03 +02:00
Lazzexus
e52ea9d9cf Fix Docker image builds and local platform setup
Getting the backend running locally in Docker surfaced several issues:

- geedeecee: REGISTRY_HOST came from the unset Gradle property
  'dockerContainerRegistryHost'. A missing value in a MapProperty makes
  the whole map absent, so every generated .env was empty and all
  docker-compose builds failed. Default it to 'localhost'.
- kafka image: dlcdn.apache.org only hosts current releases, so Kafka
  3.3.1 404s. Use archive.apache.org. Replace the deprecated openjdk:11
  base with eclipse-temurin:11-jre, bump the download stage from EOL
  Debian 11 to 12, and drop 'COPY --chmod' so the image builds without
  BuildKit/buildx.
- web-map: the map calls the API and websocket on its own origin
  (/kafkatorio/...), which previously relied on an external reverse
  proxy. Proxy /kafkatorio/ to events-server-web in the map's nginx so
  one port serves everything, and publish that port in the example
  platform compose file.
- docs: the example secret config mapped server IDs to descriptions,
  but the schema is token (JWT 'sub') -> server ID, and both must match
  [a-zA-Z0-9-_]. Descriptions with spaces crashed every service.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 17:27:11 +02:00
Adam
76a661e444 move logo, add function to auto-init the project logo 2022-10-30 18:44:47 +01:00
Adam
5bb8cc789c add/update docs 2022-10-30 17:21:32 +01:00
Adam
4cb00febc2 rm docs/notes/question.md - not needed 2022-07-17 23:50:33 +02:00
Adam
08ab1c582c tidying up 2021-12-28 17:05:52 +01:00