Kafkatorio/docs/guide.md
Lazzexus 47ab9f30fc Add a forwarder for Factorio servers that don't run in Docker
The mod prints its packets to the server's standard output, and Docker's
syslog log driver sends them to Kafkatorio. kafkatorio-forward.py does
the same without Docker: it runs the server (or reads its output from a
pipe), prints the output as usual, and sends the mod's packets to the
syslog port in the same format as Docker, tagged with the token.

- the server keeps running when Kafkatorio is unreachable: packets wait in
  a bounded buffer, and are sent after reconnecting
- before each send it checks Kafkatorio hasn't closed the connection, so
  lines aren't silently lost in a dead socket
- Ctrl+C and SIGTERM are passed to the server once, so it saves and stops
- the token is read from a file or the environment, not the command line
- only the Python standard library

util-linux's logger can't be used instead: it rejects the token, which is
longer than syslog's 48 character APP-NAME limit.

Tested on Linux with a Factorio 2.0.77 server: packets arrive, the
forwarder reconnects when the syslog server restarts (no packets lost),
and the server saves when stopped.

The guide has a section for it, with a systemd service example, and the
README no longer says the Factorio server must run in Docker.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-22 11:00:24 +01:00

16 KiB

Running Kafkatorio

This guide sets up Kafkatorio for a Factorio server, from building it to a public website.

What you need

  • A Factorio 2.0 server. Kafkatorio reads what the server prints. In Docker (for example factoriotools/factorio), Docker's syslog log driver sends it; the example setup can run the Factorio server for you. Without Docker, a small Python script does it (see below).
  • A Docker host for Kafkatorio, ideally the same one as the Factorio server. It needs about 5 GB of memory while the existing map is sent (step 5), and much less after that. (The memory limits are the -Xmx options in docker-compose.yml.)
  • A machine to build on, with Docker, Git and a JDK (11 or newer). There are no published images, so you build them yourself. Gradle downloads everything else (including Node.js, for the mod).
Port Service Expose?
12081 the website (web-map) yes, through a reverse proxy (step 7)
12035 receives the Factorio server's log (syslog) only to the Factorio server's host
18090 Kafka UI, for debugging no - Kafka has no authentication
34197/udp the Factorio game (if the example runs it) yes, for players

1. Build the Docker images and the mod

git clone <this repository> kafkatorio
cd kafkatorio

./gradlew \
  :modules:infra-kafka-cluster:dockerComposeBuild \
  :modules:events-processors:dockerComposeBuild \
  :modules:events-server-syslog:dockerComposeBuild \
  :modules:events-server-web:dockerComposeBuild \
  :modules:web-map:dockerComposeBuild \
  :modules:events-mod:distZip

This creates five images, tagged localhost/dev.adamko.kafkatorio/<name>:<version> (kafka-kraft, events-processors, events-server-syslog, events-server-web, web-map), and the mod, modules/events-mod/build/distributions/kafkatorio-events_<version>.zip. The version is the version in build.gradle.kts.

If you build on a different machine from the Docker host, copy the images over (no registry needed):

docker save $(docker images --format '{{.Repository}}:{{.Tag}}' | grep dev.adamko.kafkatorio) \
  | gzip | ssh <docker host> 'gunzip | docker load'

2. Configure Kafkatorio

Copy the kafkatorio-platform directory to the Docker host. All Kafkatorio's data is kept in it. In that directory:

  1. Copy example.env to .env, and check KAFKATORIO_VERSION matches the images you built.

  2. Copy example.secret.kafkatorio-config.yml to .secret.kafkatorio-config.yml, and set

    • jwt-secret: a long random password, used to sign the Factorio server's token. You won't need to type it anywhere. For example, openssl rand -hex 32.
    • kafkatorio-servers: your Factorio server. The key is the server's token name (put in the token, below), and the value is its ID, which is used in the website's URLs (/servers/<ID>/). They can be the same, e.g. my-server: "my-server".
  3. Create a Kafka cluster ID, so Kafka's data survives image rebuilds:

    docker run --rm --entrypoint ./bin/kafka-storage.sh \
      localhost/dev.adamko.kafkatorio/kafka-kraft:<version> random-uuid > config/cluster_id
    

Create a token for the Factorio server

The Factorio server's log is tagged with a token (a JWT), so Kafkatorio knows which server it's from, and ignores logs from anyone else. Create it with the jwt-secret and the server's token name (sub):

python3 -c 'import base64,hashlib,hmac,json,sys,time
b=lambda d: base64.urlsafe_b64encode(d).rstrip(b"=").decode()
j=lambda o: b(json.dumps(o,separators=(",",":")).encode())
m=j({"alg":"HS256","typ":"JWT"})+"."+j({"sub":sys.argv[2],"iat":int(time.time())})
print(m+"."+b(hmac.new(sys.argv[1].encode(),m.encode(),hashlib.sha256).digest()))' \
  '<jwt-secret>' '<token name>'

(Or use jwt.io, with algorithm HS256, the payload {"sub": "<token name>", "iat": <current unix time>}, and the jwt-secret as the secret. But then you're typing the secret into a website.)

The token starts with ey. Put it in .env as KAFKATORIO_TOKEN.

3. Start Kafkatorio

docker compose up -d
docker compose logs -f

The website is on http://<docker host>:12081. It's empty until the Factorio server sends something.

4. Connect the Factorio server

Install the mod

Put kafkatorio-events_<version>.zip in the Factorio server's mods directory (for the example server, factorio-server/mods/), and restart it.

Players need the same mod, with the same version, to join, so give them the zip too (it goes in their Factorio mods directory). Don't let them download it from the mod portal: the portal's kafkatorio-events is the original project, for Factorio 1.1, and its version numbers overlap with this one.

Forward the server's log to Kafkatorio

If the example runs your Factorio server, this is already set up. Start it with:

docker compose --profile factorio up -d

If you already have a Factorio container, add these options to it, then recreate it:

--log-driver=syslog \
--log-opt syslog-address=tcp://127.0.0.1:12035 \
--log-opt syslog-format=rfc5424 \
--log-opt mode=non-blocking \
--log-opt tag=<KAFKATORIO_TOKEN>

or in a Compose file:

    logging:
      driver: syslog
      options:
        syslog-address: "tcp://127.0.0.1:12035"
        syslog-format: "rfc5424"
        mode: "non-blocking"
        tag: "<KAFKATORIO_TOKEN>"

The log driver runs in the Docker daemon, not in the container, so 127.0.0.1 is the Docker host. If the Factorio server runs on another host, use this host's address, and in docker-compose.yml change 127.0.0.1:12035:9002 to 12035:9002 (and firewall it so only the Factorio host can connect).

Docker won't start a container if its syslog server can't be reached, so Kafkatorio must be running before the Factorio server starts. After a reboot, delay the Factorio server's start by about 30 seconds.

To check it's working, docker compose logs -f events-server-syslog while someone plays. If the token wasn't signed with the jwt-secret, you'll see Could not decode JWT there.

Factorio server without Docker

The mod prints its packets to the server's standard output (not to factorio-current.log). kafkatorio-forward.py runs the server, prints its output as usual, and sends the mod's packets to Kafkatorio. It needs Python 3.8 or newer, and nothing else.

Copy it to the Factorio machine, put the token in a file only the server's user can read, and start the server through it:

echo '<KAFKATORIO_TOKEN>' > kafkatorio-token.txt
chmod 600 kafkatorio-token.txt

python3 kafkatorio-forward.py --host <kafkatorio host> --token-file kafkatorio-token.txt -- \
  ./bin/x64/factorio --start-server my-save.zip --server-settings server-settings.json
  • Everything after -- is the server's usual command.
  • --host is Kafkatorio's Docker host (default 127.0.0.1). If that's another machine, publish port 12035 to it: in docker-compose.yml, change 127.0.0.1:12035:9002 to 12035:9002, and firewall it so only the Factorio machine can connect.
  • Ctrl+C (or SIGTERM) is passed to the server, so it saves and stops as usual.
  • If Kafkatorio can't be reached, the server keeps running. The packets are kept in memory (up to 100,000, see --buffer) and sent when it's back.
  • You can also pipe into it, factorio --start-server ... | python3 kafkatorio-forward.py ..., but then stopping the script stops the server without saving. Prefer the first way.
  • KAFKATORIO_TOKEN=<token> in the environment works instead of --token-file. Don't put the token on the command line, where other users can see it in ps.

As a systemd service, for example /etc/systemd/system/factorio.service:

[Unit]
Description=Factorio server, forwarding to Kafkatorio
After=network-online.target
Wants=network-online.target

[Service]
User=factorio
WorkingDirectory=/opt/factorio
ExecStart=/usr/bin/python3 /opt/factorio/kafkatorio-forward.py \
  --host <kafkatorio host> --token-file /opt/factorio/kafkatorio-token.txt -- \
  /opt/factorio/bin/x64/factorio --start-server /opt/factorio/saves/my-save.zip \
  --server-settings /opt/factorio/data/server-settings.json
# stop the script, which passes the signal to the server once, so it saves
KillMode=mixed
TimeoutStopSec=60
Restart=on-failure

[Install]
WantedBy=multi-user.target

Messages like can't connect to ... retrying from [kafkatorio-forward] mean Kafkatorio isn't reachable yet; it keeps trying. It's tested on Linux.

5. Send the existing map

New chunks, buildings and research are sent as they happen. To send what's already there, join the game and run, in the chat console:

/kafkatorio prototypes
/kafkatorio chunks 10
  • prototypes sends the tile and entity types. Run it first, and again after changing mods.
  • chunks <radius> sends the terrain, resources and buildings of every chunk within <radius> chunks of your position (on every surface). Each chunk is 32 tiles. Start small, and increase it until the whole map is on the website. Large radiuses (e.g. chunks 30) take minutes to process, and need the most memory.

Research is sent every minute. Production statistics are sent every minute, but only while the game is running: if the server pauses when nobody's online (auto_pause), there are no statistics then.

6. Icons and names

The building icons on the map, the production page and the research page need icons, building sizes, in-game names and the technology tree. They're exported from a Factorio client with the same mods as the server (they're not in this repository, since they belong to Wube and the mod authors). Without them, the map has no building icons, and those pages show internal names, or nothing.

On a computer with Factorio, with the game closed:

factorio --dump-data
factorio --dump-prototype-locale
factorio --dump-icon-sprites
python3 docs/kafkatorio-platform/building-icons/export_building_icons.py ~/.factorio/script-output ./icons
scp -r ./icons <docker host>:<kafkatorio-platform directory>/
  • With Steam, run SteamAppId=427520 factorio --dump-icon-sprites, or Steam restarts the game normally instead.
  • factorio is the game's executable, e.g. ~/.steam/steam/steamapps/common/Factorio/bin/x64/factorio. ~/.factorio/script-output is the Linux location; on Windows it's %APPDATA%\Factorio\script-output.
  • If Pillow is installed, the (large) technology icons are shrunk.

The website uses them straight away, but browsers keep icons for up to a day, so reload with Ctrl+Shift+R. Export them again when the mods change.

7. Put it on the internet

Point a reverse proxy (Nginx Proxy Manager, Caddy, Traefik, ...) at port 12081, and enable WebSocket support - otherwise the website loads, but players and other live updates don't move. The website, the API (/kafkatorio/) and the WebSocket (/kafkatorio/ws) are all on that one port.

Don't expose the other ports. Kafka and the Kafka UI have no authentication.

Unraid

kafkatorio-platform/unraid has a setup for running Kafkatorio on the same Unraid server as the Factorio server. It's the same as the example above, but keeps the data in appdata, and doesn't run a Factorio server.

  1. Install the Compose Manager plugin (or Compose Manager Plus) from the Apps tab.

  2. Build the images, and copy them to Unraid (step 1).

  3. Create /mnt/user/appdata/kafkatorio-platform/ containing

  4. Create a Compose Manager stack, with the Unraid docker-compose.yml, and example.env as its .env. Set KAFKA_DATA_DIR to a direct disk or pool path (e.g. /mnt/cache/... or /mnt/disk1/...), not /mnt/user/...: Unraid's FUSE layer is slow for Kafka. Start the stack.

  5. Edit the Factorio container, enable Advanced View, and set Extra Parameters to

    --log-driver=syslog --log-opt syslog-address=tcp://127.0.0.1:12035 --log-opt syslog-format=rfc5424 --log-opt mode=non-blocking --log-opt tag=<KAFKATORIO_TOKEN>
    

    On the Docker tab, give the Factorio container an autostart wait of about 30 seconds, so Kafkatorio starts first.

  6. Continue from step 4 (installing the mod). Put the icons in /mnt/user/appdata/kafkatorio-platform/icons.

Updating

  1. Pull the changes, and rebuild the images and the mod (step 1). If the version in build.gradle.kts changed, update KAFKATORIO_VERSION in .env.
  2. docker compose up -d recreates the containers with the new images. The map and other data are kept.
  3. If the mod changed, replace it on the server and give players the new zip. Factorio clients and the server must have exactly the same mod. Close the game before replacing the zip in your own mods directory.

Restart all the Kafkatorio services together when updating, not only some of them: events-server-syslog reads and re-writes every packet, so it drops anything it doesn't know about yet.

Troubleshooting

The website shows "Loading servers…" forever, or a new page shows the landing page. Reload without the cache (Ctrl+Shift+R). Older versions of the website could be cached by browsers after an update.

The website loads, but nothing moves. Check WebSocket support is enabled in the reverse proxy (step 7).

Nothing appears on the map.

  • Check the server's log reaches Kafkatorio: docker compose logs events-server-syslog. Could not decode JWT means the token wasn't signed with the jwt-secret.
  • Log lines that Kafkatorio rejects go to the kafkatorio.src.server-log.dlq topic, with the reason (look at it in the Kafka UI). no server ID for token means the token name (sub) isn't in kafkatorio-servers.
  • Run /kafkatorio prototypes and /kafkatorio chunks 5 in game (step 5).
  • After a large chunks command, processing takes minutes. The Kafka UI (http://localhost:18090 on the Docker host) shows the topics, and how far behind the processors are.

The Factorio container won't start: failed to initialize logging driver. Kafkatorio's syslog server isn't running yet. Start Kafkatorio first, wait a few seconds, then start the Factorio container.

Building icons, names, or the technology tree are missing. Export the icons and names (step 6) with the same mods as the server.

Developing

  • ./gradlew build builds and tests everything.
  • modules/modules.md shows how the modules depend on each other.
  • The packets are defined in events-library. Changing them regenerates the mod's TypeScript types (modules/events-mod/src/main/typescript/generated/kafkatorio-schema.ts).
  • For the website, ./gradlew :modules:web-map:browserDevelopmentRun --continuous serves it on http://localhost:3000. It forwards API requests to events-server-web on localhost:12080, so add ports: [ "12080:80" ] to events-server-web in your local docker-compose.yml. (Icons aren't forwarded.)