No description
Find a file
Lazzexus fa28cd9713 Fix syslog server silently dropping packets under load
The syslog parser reuses its result map for each line it parses, and
SyslogMsg kept a reference to that map instead of a copy. The producer
consumes messages concurrently with the reader, so by the time a message
was checked, its map had sometimes already been overwritten by the next
line; the message then failed the 'KafkatorioPacket:::' filter and was
dropped without any log. Under real load (bursts of map chunk packets)
most packets were lost.

Copy the parsed map, and log Kafka producer send failures instead of
ignoring them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 18:58:21 +02:00
buildSrc Gradle 7.6 2022-12-05 23:18:00 +01:00
docs Unraid example: keep Kafka data off Unraid's FUSE layer 2026-10-12 18:46:13 +02:00
gradle Remove dead kotlinx-nodejs dependency from web-map 2026-10-12 15:46:04 +02:00
gradle-plugins Fix Docker image builds and local platform setup 2026-10-12 17:27:11 +02:00
modules Fix syslog server silently dropping packets under load 2026-10-12 18:58:21 +02:00
.gitignore migrate buildSrc plugins to includedBuild plugins 2022-09-18 20:49:32 +02:00
build.gradle.kts Gradle 7.6 2022-12-05 23:18:00 +01:00
gradle.properties set org.gradle.jvmargs 2022-10-15 23:53:33 +02:00
gradlew Gradle 7.6 2022-12-05 23:18:00 +01:00
gradlew.bat Gradle 7.6 2022-12-05 23:18:00 +01:00
kt52647.gradle.kts add 'skip worktree' example command 2022-12-05 23:21:29 +01:00
LICENSE update license to be gpl-3.0 2022-10-30 18:58:50 +01:00
README.md add some nice shields 2022-10-30 20:32:21 +01:00
settings.gradle.kts big ol' refactor of the Gradle Factorio Mod 2022-12-18 01:41:14 +01:00

Mod Portal Factorio Version

Kafkatorio

Kafkatorio is a platform used for creating Factorio mods that require communication with an external server.

Development is ongoing.

At present Kafkatorio is used to create a live-view web map, that shows the current status of a Factorio server, and any connected players.

However, Kafkatorio can be used for more than this. It has the potential to export metrics, and allow for inter-server communication.

Kafkatorio was created to explore the possibilities of using Apache Kafka to process updates from a Factorio server.

Running

Instructions for running a Kafkatorio instance are available in the docs.

Overview

This is a brief overview of how Kafkatorio gathers data, processes it, and uses it to create the live web-map.

Kafkatorio receives updates from a Factorio server, processes them, and exposes them over a REST and WebSocket APIs.

  1. The Kafkatorio game mod, events-mod, collects event data, de-bouncing and grouping events, and converts them to JSON packets

  2. events-mod prints the packets to the Factorio game server logs

  3. The Factorio logs are forwarded to a Kafka cluster, either using

    • syslog (using Docker logging plugin)
    • or Kafka Pipe (a script that reads the server logs and forwards them to a Kafka topic)
  4. The packets are consumed by events-server, which processes the packets using Kafka Streams, creating and hosting map tiles.

    events-server serves the map tiles via a REST API, and produces WebSocket messages regarding updates (for example, when a player's location has changed).

  5. The web-map uses Leaflet/JS to display the map, using the tiles and WebSocket messages from events-server

sequenceDiagram

Factorio multiplayer server->>events-mod: in-game events
Note over events-mod: converts events to packets
events-mod->>logs processor: packets logged to console
logs processor->>Kafka: publishes packets<br/>(Requires API key!)

Kafka->>events-server: 

Note over events-server: Packets are processed

events-server->>web-map: Web map tiles (REST API)
events-server-)web-map: WebSocket updates<br/>(e.g. player movement)

License

Because I care about sharing improvements, Kafkatorio is licensed under the
GNU General Public License v3.0.