No description
Find a file
Lazzexus 665cd1a7d6 Speed up map processing 14x by merging aggregate updates before forwarding
The zoom-out map layers aggregate 32x32 chunks into 256x256 and 512x512
chunks, holding up to 65k/262k tiles. Every incoming chunk rebuilt the
whole aggregate, and with exactly_once_v2's default 100ms commit
interval nearly every intermediate result was forwarded downstream,
cascading into the next level. A small map also has few 512-level keys,
so this work landed on one partition and one thread.

Make commit.interval.ms and cache.max.bytes.buffering configurable
(defaults unchanged), and in the Unraid example use a 10s commit
interval with a 128MB cache, so repeated updates to the same chunk are
merged in the record cache first. Measured on a live backlog: 41 -> 575
chunks/minute. Map updates now appear within ~10s instead of instantly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 19:43:17 +02:00
buildSrc Gradle 7.6 2022-12-05 23:18:00 +01:00
docs Speed up map processing 14x by merging aggregate updates before forwarding 2026-10-12 19:43:17 +02:00
gradle Remove dead kotlinx-nodejs dependency from web-map 2026-10-12 15:46:04 +02:00
gradle-plugins Fix Docker image builds and local platform setup 2026-10-12 17:27:11 +02:00
modules Speed up map processing 14x by merging aggregate updates before forwarding 2026-10-12 19:43:17 +02:00
.gitignore migrate buildSrc plugins to includedBuild plugins 2022-09-18 20:49:32 +02:00
build.gradle.kts Gradle 7.6 2022-12-05 23:18:00 +01:00
gradle.properties set org.gradle.jvmargs 2022-10-15 23:53:33 +02:00
gradlew Gradle 7.6 2022-12-05 23:18:00 +01:00
gradlew.bat Gradle 7.6 2022-12-05 23:18:00 +01:00
kt52647.gradle.kts add 'skip worktree' example command 2022-12-05 23:21:29 +01:00
LICENSE update license to be gpl-3.0 2022-10-30 18:58:50 +01:00
README.md add some nice shields 2022-10-30 20:32:21 +01:00
settings.gradle.kts big ol' refactor of the Gradle Factorio Mod 2022-12-18 01:41:14 +01:00

Mod Portal Factorio Version

Kafkatorio

Kafkatorio is a platform used for creating Factorio mods that require communication with an external server.

Development is ongoing.

At present Kafkatorio is used to create a live-view web map, that shows the current status of a Factorio server, and any connected players.

However, Kafkatorio can be used for more than this. It has the potential to export metrics, and allow for inter-server communication.

Kafkatorio was created to explore the possibilities of using Apache Kafka to process updates from a Factorio server.

Running

Instructions for running a Kafkatorio instance are available in the docs.

Overview

This is a brief overview of how Kafkatorio gathers data, processes it, and uses it to create the live web-map.

Kafkatorio receives updates from a Factorio server, processes them, and exposes them over a REST and WebSocket APIs.

  1. The Kafkatorio game mod, events-mod, collects event data, de-bouncing and grouping events, and converts them to JSON packets

  2. events-mod prints the packets to the Factorio game server logs

  3. The Factorio logs are forwarded to a Kafka cluster, either using

    • syslog (using Docker logging plugin)
    • or Kafka Pipe (a script that reads the server logs and forwards them to a Kafka topic)
  4. The packets are consumed by events-server, which processes the packets using Kafka Streams, creating and hosting map tiles.

    events-server serves the map tiles via a REST API, and produces WebSocket messages regarding updates (for example, when a player's location has changed).

  5. The web-map uses Leaflet/JS to display the map, using the tiles and WebSocket messages from events-server

sequenceDiagram

Factorio multiplayer server->>events-mod: in-game events
Note over events-mod: converts events to packets
events-mod->>logs processor: packets logged to console
logs processor->>Kafka: publishes packets<br/>(Requires API key!)

Kafka->>events-server: 

Note over events-server: Packets are processed

events-server->>web-map: Web map tiles (REST API)
events-server-)web-map: WebSocket updates<br/>(e.g. player movement)

License

Because I care about sharing improvements, Kafkatorio is licensed under the
GNU General Public License v3.0.