The syslog parser reuses its result map for each line it parses, and SyslogMsg kept a reference to that map instead of a copy. The producer consumes messages concurrently with the reader, so by the time a message was checked, its map had sometimes already been overwritten by the next line; the message then failed the 'KafkatorioPacket:::' filter and was dropped without any log. Under real load (bursts of map chunk packets) most packets were lost. Copy the parsed map, and log Kafka producer send failures instead of ignoring them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|---|---|---|
| buildSrc | ||
| docs | ||
| gradle | ||
| gradle-plugins | ||
| modules | ||
| .gitignore | ||
| build.gradle.kts | ||
| gradle.properties | ||
| gradlew | ||
| gradlew.bat | ||
| kt52647.gradle.kts | ||
| LICENSE | ||
| README.md | ||
| settings.gradle.kts | ||
Kafkatorio
Kafkatorio is a platform used for creating Factorio mods that require communication with an external server.
Development is ongoing.
At present Kafkatorio is used to create a live-view web map, that shows the current status of a Factorio server, and any connected players.
However, Kafkatorio can be used for more than this. It has the potential to export metrics, and allow for inter-server communication.
Kafkatorio was created to explore the possibilities of using Apache Kafka to process updates from a Factorio server.
Running
Instructions for running a Kafkatorio instance are available in the docs.
Overview
This is a brief overview of how Kafkatorio gathers data, processes it, and uses it to create the live web-map.
Kafkatorio receives updates from a Factorio server, processes them, and exposes them over a REST and WebSocket APIs.
-
The Kafkatorio game mod,
events-mod, collects event data, de-bouncing and grouping events, and converts them to JSON packets -
events-modprints the packets to the Factorio game server logs -
The Factorio logs are forwarded to a Kafka cluster, either using
- syslog (using Docker logging plugin)
- or Kafka Pipe (a script that reads the server logs and forwards them to a Kafka topic)
-
The packets are consumed by
events-server, which processes the packets using Kafka Streams, creating and hosting map tiles.events-serverserves the map tiles via a REST API, and produces WebSocket messages regarding updates (for example, when a player's location has changed). -
The
web-mapuses Leaflet/JS to display the map, using the tiles and WebSocket messages fromevents-server
sequenceDiagram
Factorio multiplayer server->>events-mod: in-game events
Note over events-mod: converts events to packets
events-mod->>logs processor: packets logged to console
logs processor->>Kafka: publishes packets<br/>(Requires API key!)
Kafka->>events-server:
Note over events-server: Packets are processed
events-server->>web-map: Web map tiles (REST API)
events-server-)web-map: WebSocket updates<br/>(e.g. player movement)
License
Because I care about sharing improvements, Kafkatorio is licensed under the
GNU General Public License v3.0.