The syslog socket was bound as soon as SyslogSocketServer was created,
and the Kafka producer started collecting the messages afterwards. A
SharedFlow drops what's emitted while nothing collects it, so a client
that connected (or reconnected) while events-server-syslog started up
could lose its first messages.
Now the socket is bound in start(), after the producer is collecting.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
main.bundle.js and the CSS keep the same name in every version, and nginx
didn't send Cache-Control, so browsers could keep running the old bundle
after a deploy. Then new pages (like /research/tree) showed the landing
page, stuck on "Loading servers…".
Send Cache-Control: no-cache, so browsers revalidate each load; unchanged
files are still a cheap 304 via the ETag. Icons keep their 1 day expiry.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The research page has a new tree layout (/servers/{id}/research/tree),
next to the existing list, with List / Tree buttons to switch.
The tree goes from top to bottom: each technology is in the row of its
longest chain of prerequisites, and rows are ordered so technologies sit
near the ones they're linked to. Links are drawn between technologies and
their prerequisites, green once the prerequisite is researched.
- drag to move, scroll or pinch to zoom, and Fit to show the whole tree
- selecting a technology highlights all its prerequisites and links, and
what it leads to, and shows the science packs still needed to reach it
- search highlights matching technologies; Enter goes to each in turn
- double-click (or Details) opens the same details as the list, whose
details now have a 'Show in tree' link
- the selected technology is kept in the URL (?tech=), so it can be shared
The details pop-up is moved out of researchPage.kt into
TechnologyDetail.kt, so both layouts use it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every minute, the mod diffs the force's production statistics for each
surface against the totals it sent last time (kept in
storage.productionTotals), and sends what was produced and consumed as a
ProductionUpdate.
events-server-web keeps the statistics in ring buffers (1440 minutes and
168 hours), and serves them at
/kafkatorio/data/servers/{id}/production, for a range of game time and
optionally one surface.
The web map has a production page (/servers/{id}/production, linked from
the header) with science packs used per minute (in total and per pack),
production per minute as a line chart, and a searchable table of
everything produced and consumed.
export_building_icons.py now also exports every item and fluid icon and
name (items.json, fluids.json), for the production page.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The mod sends the research state of a force (researched technologies,
levels, current research, progress and queue) whenever research changes,
and every minute for the progress. events-server-web keeps the latest state
and serves it at /kafkatorio/data/servers/{id}/research.
The web map has a research page (/servers/{id}/research, linked from the
header), with:
- progress, the current research and the queue
- the science packs needed for everything that's left
- every technology, filterable by status, science pack and name, with its
cost (including levelled technologies' count formulas)
- details: the total cost to reach a technology, its prerequisites, what it
leads to and what it unlocks
The technology tree comes from export_building_icons.py, which now also
exports technologies.json, items.json and technology/science pack icons.
Kafka deleted data older than 7 days, but the building, resource and
research views are rebuilt by reading their topics from the beginning, so
buildings disappeared after a restart. These topics are now kept forever.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A dark theme in the style of Factorio's UI, with a card per server that
shows a preview of the map around the origin and the server's surfaces.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The services now run on Java 17, with a periodic G1 GC that returns unused
heap to the OS when idle. Previously Java 11 kept the committed heap, so
events-processors held 5.5 GB while its live data was ~250 MB.
All RocksDB state stores now share one bounded block cache and write buffer
(ROCKSDB_TOTAL_MEMORY_MB, default 256 MB), instead of each of the ~45 stores
having its own.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Belts are drawn like the in-game map: a strip in the colour of their tier
(taken from the icon of the transport belt of the same tier), with an arrow
per lane in the direction items go. Splitters are drawn at their rotated
2x1 footprint with a bar across the back.
The mod now sends whether an underground belt is an entrance or an exit.
Entrances are capped on the side where items go underground, exits on the
side where they come out, and the hover tooltip says which it is.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The mod now sends each building's direction and recipe (furnaces: the last
recipe, since they only have one while working).
- Rotated buildings are drawn at their real footprint.
- Crafting machines show their recipe icon, like Factorio's alt mode, and
the hover tooltip includes the recipe.
The web map has a surface drop-down (e.g. Space Exploration planets and
orbits), backed by a new surfaces endpoint. Every layer, the resource labels
and the player markers follow the chosen surface.
Resource labels use the in-game names (e.g. "Iridite" instead of
"Se iridium ore"). export_building_icons.py also exports recipe icons, plus
recipe and resource names.
The three in-memory views in events-server-web now share one helper for
following a Kafka topic.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
events-server-web keeps the latest state of every building in memory (read
from the map-chunk-entity topic, like the resource patches) and serves them
per area. The web map draws the icons at their real size onto canvas tiles,
so large bases stay fast, and shows each building's name, status and health
on hover. Ghosts are ignored, and buildings are removed when they're mined.
The icons belong to Wube and the mod authors, so they aren't in the repo.
export_building_icons.py builds them from a Factorio client's
--dump-data/--dump-prototype-locale/--dump-icon-sprites output, and the
web-map nginx serves them from a mounted volume.
Also label single-well oil fields "1 well".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On Factorio 2.0 (with Space Exploration + Krastorio 2), crude oil and
mineral water report infinite_resource=false and a nil initial_amount,
but only infinite resources have a minimum_resource_amount > 0.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Group nearby resource tiles of the same type into patches, and label
them on the map, e.g. "Iron ore / 2.4M", or "Crude oil / 12 wells".
- events-library: ResourcePatch model, shared by the server and map.
- events-server-web: ResourcePatchService reads the map-chunk-resource
topic from the beginning (read_committed) into memory, and on request
groups tiles into connected patches (join distance 2 tiles for ore,
12 for infinite resources, whose wells are spread out). Results are
cached until new data arrives.
New endpoint: GET .../servers/{id}/map/resources/s{surface}/patches
- web-map: toggleable "Resource patches" overlay with labels at each
patch centre and a hover tooltip, refreshed every minute. Patches
under 10 tiles are hidden. (Surface 1 only, like the rest of the map.)
- events-mod: mark resources infinite based on the prototype's
infinite_resource. Entity initial_amount is nil for e.g. crude oil, so
oil fields were treated as hundreds of tiny ore patches.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Under exactly_once_v2, Streams defaults the producer transaction timeout
to 10s. With commit.interval.ms raised to 10s (665cd1a), transactions on
the heavy map-aggregation topology timed out, got fenced
(ProducerFencedException / TaskMigratedException ~130 times during one
large chunk import), and their work was redone. Default the transaction
timeout to 60s, overridable via KAFKA_STREAMS_TRANSACTION_TIMEOUT_MS.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
'/kafkatorio chunks' only re-sent chunk terrain: resources were only
sent when Factorio first generated a chunk, and buildings only when
built or mined. A save that predates the mod therefore showed no
resources or buildings at all.
Re-sent chunks now also go through the entity chunk handler, which now
also includes the chunk's existing player-force buildings (excluding
characters). Newly generated chunks contain no buildings, so that path
is unaffected.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The zoom-out map layers aggregate 32x32 chunks into 256x256 and 512x512
chunks, holding up to 65k/262k tiles. Every incoming chunk rebuilt the
whole aggregate, and with exactly_once_v2's default 100ms commit
interval nearly every intermediate result was forwarded downstream,
cascading into the next level. A small map also has few 512-level keys,
so this work landed on one partition and one thread.
Make commit.interval.ms and cache.max.bytes.buffering configurable
(defaults unchanged), and in the Unraid example use a 10s commit
interval with a 128MB cache, so repeated updates to the same chunk are
merged in the record cache first. Measured on a live backlog: 41 -> 575
chunks/minute. Map updates now appear within ~10s instead of instantly.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- config.yml: num.stream.threads can be set with
KAFKA_STREAMS_NUM_THREADS (default unchanged: 1). Map processing was
bottlenecked on a single stream thread while the host was mostly idle.
- Unraid example: run 3 stream threads (topics have 3 partitions), and
cap each JVM's heap via JAVA_OPTS. Without a cap each JVM defaults to
1/4 of the host's RAM, and the stack grew to ~14 GB.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The syslog parser reuses its result map for each line it parses, and
SyslogMsg kept a reference to that map instead of a copy. The producer
consumes messages concurrently with the reader, so by the time a message
was checked, its map had sometimes already been overwritten by the next
line; the message then failed the 'KafkatorioPacket:::' filter and was
dropped without any log. Under real load (bursts of map chunk packets)
most packets were lost.
Copy the parsed map, and log Kafka producer send failures instead of
ignoring them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When a client closed its connection, handleConnection threw on the
null line, closed the socket, and then logged socket.description() -
which reads the address of the now-closed socket and throws
ClosedChannelException. That escaped the handler and crashed the
whole server, dropping every other client's connection. With the
Docker syslog log driver in non-blocking mode, the Factorio server's
logs were then silently discarded, so most packets never arrived.
Treat a null line as a normal client disconnect, capture the socket
description while it's still open, and always close the socket in
'finally', so one connection can't take down the server.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- EventDataQueue.enqueue: `key ! in storage.store` parses as a non-null
assertion followed by `in`, so the check was inverted: a key was only
queued if it was already in the store. A first enqueue stored the
event but never queued it, so it was never sent.
- The chunks command keyed the queue with `${surface}${x}${y}`, so
different chunks collided (e.g. surface 1, (1,12) and (11,2) both
became "1112"). Separate the parts with '_'.
Saves that ran the broken version keep orphaned queue entries; run
`/kafkatorio init_global` once to clear them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- web-map: main() was bound to the whole site state, so every state
update (e.g. each player position update over the websocket)
re-rendered the page and re-attached the Leaflet map to a new
element, blanking the tiles. Only re-render when the view, server
list, or map instance changes.
- web-map: reconnect the websocket when it closes, instead of silently
stopping live updates.
- events-server-web: send websocket pings every 30s, so idle
connections aren't closed by reverse proxies (nginx defaults to a
60s read timeout).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Getting the backend running locally in Docker surfaced several issues:
- geedeecee: REGISTRY_HOST came from the unset Gradle property
'dockerContainerRegistryHost'. A missing value in a MapProperty makes
the whole map absent, so every generated .env was empty and all
docker-compose builds failed. Default it to 'localhost'.
- kafka image: dlcdn.apache.org only hosts current releases, so Kafka
3.3.1 404s. Use archive.apache.org. Replace the deprecated openjdk:11
base with eclipse-temurin:11-jre, bump the download stage from EOL
Debian 11 to 12, and drop 'COPY --chmod' so the image builds without
BuildKit/buildx.
- web-map: the map calls the API and websocket on its own origin
(/kafkatorio/...), which previously relied on an external reverse
proxy. Proxy /kafkatorio/ to events-server-web in the map's nginx so
one port serves everything, and publish that port in the example
platform compose file.
- docs: the example secret config mapped server IDs to descriptions,
but the schema is token (JWT 'sub') -> server ID, and both must match
[a-zA-Z0-9-_]. Descriptions with spaces crashed every service.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Caught by another in-game crash: "Entity is not resource." when
reading entity.amount. The previous fix (collision_mask: "resource")
was still wrong - collision layers and entity type are decoupled in
Factorio 2.0, so find_entities_filtered({collision_mask: "resource"})
can return non-resource entities that merely opted into the same
collision layer (likely something from Space Exploration or
Krastorio 2). entity.amount/initial_amount only exist on entities
whose prototype type is actually "resource".
Filtering by type: "resource" directly targets the right entities
and is simpler than inferring it via collision mask in the first
place.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Caught by an in-game crash on real Factorio 2.0.77 (with Space
Exploration + Krastorio 2 loaded):
Error while running event kafkatorio-events::on_chunk_generated
Unknown collision-layer name: resource-layer
Factorio 2.0 made collision layers fully prototype-defined (previously
a fixed set of built-in names). Checked the official factorio-data
2.0.77 tag's core/lualib/collision-mask-defaults.lua: resource-type
entities now use layer name "resource", not "resource-layer". Fixed
both find_entities_filtered() call sites in EntityUpdateHandler.ts
that scan for resource entities around mined/placed tiles and newly
generated chunks.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Caught by actually running the mod on a real Factorio 2.0.77 server:
Error while loading mod-setting prototype
"kafkatorio:cache-expiration-seconds_kafkatorio-packet-keyed-PlayerUpdate"
(double-setting): Invalid prototype name. Only characters
A-Z a-z 0-9 _- are allowed.
cacheDurationSettingName() built names with a ":" separator
("kafkatorio:cache-expiration-seconds_..."), which Factorio 2.0
rejects outright. Replaced it with "-", matching the mod-name-prefix
convention the engine's own docs recommend. This is a pure runtime
validation issue invisible to the TypeScript compiler - exactly the
kind of thing only actually loading the mod in-game catches.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Without an explicit rootDir, tstl computed output paths relative to
tsconfig.json's own directory (modules/events-mod/) rather than the
actual TypeScript source root (src/main/typescript/), so control.lua
and settings.lua ended up nested under src/main/typescript/ in the
packaged mod zip instead of at its root - where Factorio requires them
to be. The mod would not have loaded at all as packaged.
This went uncaught because packageFactorioMod never previously
produced real output (blocked by the now-fixed kxs-ts-gen-core/JitPack
issue). Verified by actually running packageFactorioMod and inspecting
the resulting zip: control.lua, settings.lua and info.json are now at
the root, and control.lua's require() calls ("main.global-init" etc.)
correctly resolve against that flat layout.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Its only coordinate (org.jetbrains.kotlinx:kotlinx-nodejs:0.0.7) was
published to Bintray's "kotlinx" repository, which has been shut down
since 2021 - it's unresolvable from any repo configured here (Maven
Central, JitPack, Gradle Plugin Portal, or JetBrains Space all 404 it),
blocking configuration of the whole build whenever a task touched
web-map. Turns out nothing in web-map's source actually imports
kotlinx.nodejs.* - the one require() call nearby is webpack's own
bundler global, unrelated to this library. It was also duplicated next
to a second, redundant kotlinx-html line. Just deleting the dependency
is the correct fix, not re-hosting a defunct artifact.
Verified: `:modules:events-mod:build :modules:events-library:check`
now gets past configuration and all JVM-side tasks succeed (the JS
browser test target still needs a locally installed Chrome for Karma,
which is an unrelated environment requirement, not a dependency issue).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The version catalog pointed kxs-ts-gen-core at a JitPack coordinate
(com.github.adamko-dev.kotlinx-serialization-typescript-generator) that
no longer serves real artifacts there, despite JitPack's build-status
API still reporting it as built. The library is properly published to
Maven Central under dev.adamko.kxstsgen - a commented-out line with
that exact coordinate was already sitting in the catalog. Swapping to
it fixes dependency resolution for every module depending on
versions-platform (which previously blocked any Gradle task touching
events-library, including all of its tests).
That unblocked two more pre-existing, unrelated bugs that had never
been caught because this module never previously compiled:
- LuaJsonList<T> : List<T> by list triggered a longstanding Kotlin
compiler bug (KT-3546, "accidental override" on generic toArray)
specific to delegating java.util.List via `by`. Replaced with a
hand-written Collection<T> implementation (no delegation codegen,
no bug), which still satisfies Kotest's forOne/forAll inspectors -
the only thing any code actually needs from it.
- The kxs-ts-gen Kotlin->TypeScript generator (kt2ts2.kt) still emitted
the pre-2.0 bare `uint`/`int` names and no import statement, since it
predates typed-factorio's move to virtual modules. Updated its
primitive-type mappings to uint32/int32 and added the "factorio:runtime"
import to its generated header, so the regenerated schema matches what
the rest of events-mod now expects.
Verified via real Gradle tasks (not just manual tooling): events-library
compiles and all tests pass; events-mod's full build - Kotlin schema
generation, npm install, tstl compile, and info.json generation (with
factorio_version "2.0" and the space-exploration/Krastorio2 optional
dependencies) - succeeds end-to-end.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
typed-factorio 2.0.0+ stopped declaring types as ambient globals - they
now live in virtual modules ("factorio:runtime", "factorio:common",
"factorio:settings") and must be explicitly imported. Adds those imports
throughout events-mod. Also:
- Bumps typescript-to-lua 1.11.1 -> 1.25.2 and typescript 4.9 -> 5.4.2,
since tstl pins an exact peer TypeScript version and typed-factorio
3.x needs a newer one than 4.9 supports (a real compiler bug in
TS 4.9 against the generated defines.segmented_unit_ai_state enum).
- Bumps the pinned isolated Node (14.19.1 -> 24.21.0), since
typescript-to-lua 1.25.2 requires Node >=16.10.0.
- Renames bare uint/int -> uint32/int32 (import aliases), since the
unsized aliases no longer exist.
- Drops OnEntityDestroyedEvent from an unused documentation-only union
type - that event no longer exists in the 2.0 API.
- LuaEntityPrototype.max_health is now get_max_health() (quality-aware).
- Fixes a branded-type mismatch between our generated schema's
ForceIndex and typed-factorio's own ForceIndex.
Verified by actually running tstl end-to-end (clean compile, Lua
emitted) after installing a real JDK and Node toolchain.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Bumps the Factorio engine pin and typed-factorio (1.11.0 -> 3.36.0,
targeting 2.0.75) and fixes the resulting breaking API changes:
global -> storage, game.*_prototypes -> prototypes.*, game.table_to_json/
encode_string -> helpers.*, OnBuiltEntityEvent.created_entity -> .entity,
removed TileRead/ColorTable/MapPositionTable-style write-form aliases,
new CollisionMask shape, restructured settings-stage type imports, and
new defines.entity_status values (added to the Status enum, generated
schema, and EntityStatusRecord mapping). Also declares optional mod
dependencies on space-exploration and Krastorio2.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>