Commit graph

535 commits

Author SHA1 Message Date
e183ccd1fc Raise Kafka Streams transaction timeout above the commit interval
Under exactly_once_v2, Streams defaults the producer transaction timeout
to 10s. With commit.interval.ms raised to 10s (665cd1a), transactions on
the heavy map-aggregation topology timed out, got fenced
(ProducerFencedException / TaskMigratedException ~130 times during one
large chunk import), and their work was redone. Default the transaction
timeout to 60s, overridable via KAFKA_STREAMS_TRANSACTION_TIMEOUT_MS.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 07:00:06 +02:00
9067694172 Send resources and existing buildings when re-sending chunks
'/kafkatorio chunks' only re-sent chunk terrain: resources were only
sent when Factorio first generated a chunk, and buildings only when
built or mined. A save that predates the mod therefore showed no
resources or buildings at all.

Re-sent chunks now also go through the entity chunk handler, which now
also includes the chunk's existing player-force buildings (excluding
characters). Newly generated chunks contain no buildings, so that path
is unaffected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 20:04:50 +02:00
665cd1a7d6 Speed up map processing 14x by merging aggregate updates before forwarding
The zoom-out map layers aggregate 32x32 chunks into 256x256 and 512x512
chunks, holding up to 65k/262k tiles. Every incoming chunk rebuilt the
whole aggregate, and with exactly_once_v2's default 100ms commit
interval nearly every intermediate result was forwarded downstream,
cascading into the next level. A small map also has few 512-level keys,
so this work landed on one partition and one thread.

Make commit.interval.ms and cache.max.bytes.buffering configurable
(defaults unchanged), and in the Unraid example use a 10s commit
interval with a 128MB cache, so repeated updates to the same chunk are
merged in the record cache first. Measured on a live backlog: 41 -> 575
chunks/minute. Map updates now appear within ~10s instead of instantly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 19:43:17 +02:00
f02e1a783e Make Kafka Streams thread count configurable, cap JVM heaps on Unraid
- config.yml: num.stream.threads can be set with
  KAFKA_STREAMS_NUM_THREADS (default unchanged: 1). Map processing was
  bottlenecked on a single stream thread while the host was mostly idle.
- Unraid example: run 3 stream threads (topics have 3 partitions), and
  cap each JVM's heap via JAVA_OPTS. Without a cap each JVM defaults to
  1/4 of the host's RAM, and the stack grew to ~14 GB.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 19:21:22 +02:00
fa28cd9713 Fix syslog server silently dropping packets under load
The syslog parser reuses its result map for each line it parses, and
SyslogMsg kept a reference to that map instead of a copy. The producer
consumes messages concurrently with the reader, so by the time a message
was checked, its map had sometimes already been overwritten by the next
line; the message then failed the 'KafkatorioPacket:::' filter and was
dropped without any log. Under real load (bursts of map chunk packets)
most packets were lost.

Copy the parsed map, and log Kafka producer send failures instead of
ignoring them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 18:58:21 +02:00
2944d52843 Fix syslog server crashing whenever a client disconnects
When a client closed its connection, handleConnection threw on the
null line, closed the socket, and then logged socket.description() -
which reads the address of the now-closed socket and throws
ClosedChannelException. That escaped the handler and crashed the
whole server, dropping every other client's connection. With the
Docker syslog log driver in non-blocking mode, the Factorio server's
logs were then silently discarded, so most packets never arrived.

Treat a null line as a normal client disconnect, capture the socket
description while it's still open, and always close the socket in
'finally', so one connection can't take down the server.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 18:46:13 +02:00
da9751c089 Fix '/kafkatorio chunks' not sending most chunks
- EventDataQueue.enqueue: `key ! in storage.store` parses as a non-null
  assertion followed by `in`, so the check was inverted: a key was only
  queued if it was already in the store. A first enqueue stored the
  event but never queued it, so it was never sent.
- The chunks command keyed the queue with `${surface}${x}${y}`, so
  different chunks collided (e.g. surface 1, (1,12) and (11,2) both
  became "1112"). Separate the parts with '_'.

Saves that ran the broken version keep orphaned queue entries; run
`/kafkatorio init_global` once to clear them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 18:16:20 +02:00
d6d9444723 Fix web map blanking and websocket dropping behind a reverse proxy
- web-map: main() was bound to the whole site state, so every state
  update (e.g. each player position update over the websocket)
  re-rendered the page and re-attached the Leaflet map to a new
  element, blanking the tiles. Only re-render when the view, server
  list, or map instance changes.
- web-map: reconnect the websocket when it closes, instead of silently
  stopping live updates.
- events-server-web: send websocket pings every 30s, so idle
  connections aren't closed by reverse proxies (nginx defaults to a
  60s read timeout).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 18:09:03 +02:00
cae2eb09c6 Fix Docker image builds and local platform setup
Getting the backend running locally in Docker surfaced several issues:

- geedeecee: REGISTRY_HOST came from the unset Gradle property
  'dockerContainerRegistryHost'. A missing value in a MapProperty makes
  the whole map absent, so every generated .env was empty and all
  docker-compose builds failed. Default it to 'localhost'.
- kafka image: dlcdn.apache.org only hosts current releases, so Kafka
  3.3.1 404s. Use archive.apache.org. Replace the deprecated openjdk:11
  base with eclipse-temurin:11-jre, bump the download stage from EOL
  Debian 11 to 12, and drop 'COPY --chmod' so the image builds without
  BuildKit/buildx.
- web-map: the map calls the API and websocket on its own origin
  (/kafkatorio/...), which previously relied on an external reverse
  proxy. Proxy /kafkatorio/ to events-server-web in the map's nginx so
  one port serves everything, and publish that port in the example
  platform compose file.
- docs: the example secret config mapped server IDs to descriptions,
  but the schema is token (JWT 'sub') -> server ID, and both must match
  [a-zA-Z0-9-_]. Descriptions with spaces crashed every service.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 17:27:11 +02:00
05389d5954 Filter resource entities by prototype type, not collision layer
Caught by another in-game crash: "Entity is not resource." when
reading entity.amount. The previous fix (collision_mask: "resource")
was still wrong - collision layers and entity type are decoupled in
Factorio 2.0, so find_entities_filtered({collision_mask: "resource"})
can return non-resource entities that merely opted into the same
collision layer (likely something from Space Exploration or
Krastorio 2). entity.amount/initial_amount only exist on entities
whose prototype type is actually "resource".

Filtering by type: "resource" directly targets the right entities
and is simpler than inferring it via collision mask in the first
place.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-12 16:36:12 +02:00
8040abfd82 Fix collision layer name: "resource-layer" -> "resource" (Factorio 2.0)
Caught by an in-game crash on real Factorio 2.0.77 (with Space
Exploration + Krastorio 2 loaded):

  Error while running event kafkatorio-events::on_chunk_generated
  Unknown collision-layer name: resource-layer

Factorio 2.0 made collision layers fully prototype-defined (previously
a fixed set of built-in names). Checked the official factorio-data
2.0.77 tag's core/lualib/collision-mask-defaults.lua: resource-type
entities now use layer name "resource", not "resource-layer". Fixed
both find_entities_filtered() call sites in EntityUpdateHandler.ts
that scan for resource entities around mined/placed tiles and newly
generated chunks.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-12 16:32:57 +02:00
1e8ed08ca2 Fix invalid mod-setting prototype name (colon not allowed)
Caught by actually running the mod on a real Factorio 2.0.77 server:

  Error while loading mod-setting prototype
  "kafkatorio:cache-expiration-seconds_kafkatorio-packet-keyed-PlayerUpdate"
  (double-setting): Invalid prototype name. Only characters
  A-Z a-z 0-9 _- are allowed.

cacheDurationSettingName() built names with a ":" separator
("kafkatorio:cache-expiration-seconds_..."), which Factorio 2.0
rejects outright. Replaced it with "-", matching the mod-name-prefix
convention the engine's own docs recommend. This is a pure runtime
validation issue invisible to the TypeScript compiler - exactly the
kind of thing only actually loading the mod in-game catches.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-12 16:12:39 +02:00
d0c76b5c85 Fix changelog version number to match the actual project version
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-12 15:53:50 +02:00
c8de66247a Fix mod zip layout: set tsconfig rootDir so control.lua lands at mod root
Without an explicit rootDir, tstl computed output paths relative to
tsconfig.json's own directory (modules/events-mod/) rather than the
actual TypeScript source root (src/main/typescript/), so control.lua
and settings.lua ended up nested under src/main/typescript/ in the
packaged mod zip instead of at its root - where Factorio requires them
to be. The mod would not have loaded at all as packaged.

This went uncaught because packageFactorioMod never previously
produced real output (blocked by the now-fixed kxs-ts-gen-core/JitPack
issue). Verified by actually running packageFactorioMod and inspecting
the resulting zip: control.lua, settings.lua and info.json are now at
the root, and control.lua's require() calls ("main.global-init" etc.)
correctly resolve against that flat layout.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-12 15:53:03 +02:00
8c03f502db Remove dead kotlinx-nodejs dependency from web-map
Its only coordinate (org.jetbrains.kotlinx:kotlinx-nodejs:0.0.7) was
published to Bintray's "kotlinx" repository, which has been shut down
since 2021 - it's unresolvable from any repo configured here (Maven
Central, JitPack, Gradle Plugin Portal, or JetBrains Space all 404 it),
blocking configuration of the whole build whenever a task touched
web-map. Turns out nothing in web-map's source actually imports
kotlinx.nodejs.* - the one require() call nearby is webpack's own
bundler global, unrelated to this library. It was also duplicated next
to a second, redundant kotlinx-html line. Just deleting the dependency
is the correct fix, not re-hosting a defunct artifact.

Verified: `:modules:events-mod:build :modules:events-library:check`
now gets past configuration and all JVM-side tasks succeed (the JS
browser test target still needs a locally installed Chrome for Karma,
which is an unrelated environment requirement, not a dependency issue).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-12 15:46:04 +02:00
509832d26d Fix kxs-ts-gen-core dependency resolution and unblocked compile bugs
The version catalog pointed kxs-ts-gen-core at a JitPack coordinate
(com.github.adamko-dev.kotlinx-serialization-typescript-generator) that
no longer serves real artifacts there, despite JitPack's build-status
API still reporting it as built. The library is properly published to
Maven Central under dev.adamko.kxstsgen - a commented-out line with
that exact coordinate was already sitting in the catalog. Swapping to
it fixes dependency resolution for every module depending on
versions-platform (which previously blocked any Gradle task touching
events-library, including all of its tests).

That unblocked two more pre-existing, unrelated bugs that had never
been caught because this module never previously compiled:

- LuaJsonList<T> : List<T> by list triggered a longstanding Kotlin
  compiler bug (KT-3546, "accidental override" on generic toArray)
  specific to delegating java.util.List via `by`. Replaced with a
  hand-written Collection<T> implementation (no delegation codegen,
  no bug), which still satisfies Kotest's forOne/forAll inspectors -
  the only thing any code actually needs from it.
- The kxs-ts-gen Kotlin->TypeScript generator (kt2ts2.kt) still emitted
  the pre-2.0 bare `uint`/`int` names and no import statement, since it
  predates typed-factorio's move to virtual modules. Updated its
  primitive-type mappings to uint32/int32 and added the "factorio:runtime"
  import to its generated header, so the regenerated schema matches what
  the rest of events-mod now expects.

Verified via real Gradle tasks (not just manual tooling): events-library
compiles and all tests pass; events-mod's full build - Kotlin schema
generation, npm install, tstl compile, and info.json generation (with
factorio_version "2.0" and the space-exploration/Krastorio2 optional
dependencies) - succeeds end-to-end.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-12 15:40:33 +02:00
e296a9aa90 Fix compile errors surfaced by actually building against Factorio 2.0.77
typed-factorio 2.0.0+ stopped declaring types as ambient globals - they
now live in virtual modules ("factorio:runtime", "factorio:common",
"factorio:settings") and must be explicitly imported. Adds those imports
throughout events-mod. Also:

- Bumps typescript-to-lua 1.11.1 -> 1.25.2 and typescript 4.9 -> 5.4.2,
  since tstl pins an exact peer TypeScript version and typed-factorio
  3.x needs a newer one than 4.9 supports (a real compiler bug in
  TS 4.9 against the generated defines.segmented_unit_ai_state enum).
- Bumps the pinned isolated Node (14.19.1 -> 24.21.0), since
  typescript-to-lua 1.25.2 requires Node >=16.10.0.
- Renames bare uint/int -> uint32/int32 (import aliases), since the
  unsized aliases no longer exist.
- Drops OnEntityDestroyedEvent from an unused documentation-only union
  type - that event no longer exists in the 2.0 API.
- LuaEntityPrototype.max_health is now get_max_health() (quality-aware).
- Fixes a branded-type mismatch between our generated schema's
  ForceIndex and typed-factorio's own ForceIndex.

Verified by actually running tstl end-to-end (clean compile, Lua
emitted) after installing a real JDK and Node toolchain.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-12 15:27:50 +02:00
072d1a2845 Port events-mod to Factorio 2.0.77, declare SE/Krastorio 2 compatibility
Bumps the Factorio engine pin and typed-factorio (1.11.0 -> 3.36.0,
targeting 2.0.75) and fixes the resulting breaking API changes:
global -> storage, game.*_prototypes -> prototypes.*, game.table_to_json/
encode_string -> helpers.*, OnBuiltEntityEvent.created_entity -> .entity,
removed TileRead/ColorTable/MapPositionTable-style write-form aliases,
new CollisionMask shape, restructured settings-stage type imports, and
new defines.entity_status values (added to the Status enum, generated
schema, and EntityStatusRecord mapping). Also declares optional mod
dependencies on space-exploration and Krastorio2.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-12 15:06:30 +02:00
Adam
76c47d9756 tidy up after refactoring Gradle Factorio plugin 2022-12-19 16:53:18 +01:00
Adam
a99ed5bb85 remove generated lua (it's stable - moved it to temp dir) 2022-12-19 16:52:49 +01:00
Adam
9f2b9cbd5c big ol' refactor of the Gradle Factorio Mod
- handle launching in multiple OSes
- specific tasks for specific mod operations
- extension object for settings
- ground work for multiple mods
2022-12-18 01:41:14 +01:00
Adam
2c9c442595 typo dequed -> dequeued 2022-12-06 00:24:17 +01:00
Adam
fcb870072a version bumps 2022-12-06 00:23:52 +01:00
Adam
dd9c9500d3 lenient Docker Env properties 2022-12-05 23:20:54 +01:00
Adam
76d553a6bf version 0.10.1 2022-10-30 19:27:50 +01:00
Adam
ce4df2081a bump versions: factorio, typed-factorio 2022-10-30 19:26:43 +01:00
Adam
394acc5ca0 version 0.10.0 2022-10-30 19:22:38 +01:00
Adam
30d5a67a2a update license to be gpl-3.0 2022-10-30 18:58:50 +01:00
Adam
76a661e444 move logo, add function to auto-init the project logo 2022-10-30 18:44:47 +01:00
Adam
f0b080a7fe optimise logo 2022-10-30 18:44:11 +01:00
Adam
531b8e193e tidy up gradle plugins (mostly removing comments) 2022-10-30 17:13:03 +01:00
Adam
3f771f9838 fix bootstrap server, remove http:// 2022-10-23 21:54:38 +02:00
Adam
ccd08c87c8 version 0.9.15 2022-10-22 23:37:15 +02:00
Adam
5562e86f09 [docker-files] remove user config 2022-10-22 23:36:00 +02:00
Adam
e05557c97c [web-map] fix missing css/image in dockerfile 2022-10-22 20:54:40 +02:00
Adam
51f44f1fd2 version 0.9.14 2022-10-22 20:36:47 +02:00
Adam
82c681e320 [dockerfiles] remove volumes 2022-10-22 20:35:52 +02:00
Adam
14e204d197 version 0.9.13 2022-10-22 20:28:22 +02:00
Adam
45691a049b [dockerfiles] fix user name/id variables 2022-10-22 20:27:17 +02:00
Adam
c9d21fa25a try fixing user permissions in containers 2022-10-22 20:24:27 +02:00
Adam
bb5e98f740 version 0.9.12 2022-10-22 19:56:33 +02:00
Adam
9b331cc105 update dockerfiles
- define user uid/gid
- use new geedeecee files (hopefully means smaller images, faster building)
2022-10-22 19:55:02 +02:00
Adam
ce0d4b7378 [events-processors] change where topo description is written 2022-10-22 19:52:02 +02:00
Adam
de24a62635 version 0.9.11 2022-10-18 23:23:09 +02:00
Adam
f99208b5b2 more fixing of Kafkatorio packets... 2022-10-18 23:22:26 +02:00
Adam
77d0dc27ca fix local mod deploy Gradle tasks 2022-10-18 21:36:22 +02:00
Adam
dda48bac78 refactor the KafkatorioPacket tag that the mod logs, and the syslog-server parses 2022-10-18 21:35:49 +02:00
Adam
ecf680d9ce [syslog-server] minor styling 2022-10-18 21:26:30 +02:00
Adam
557dc97635 [syslog-server] refactor logging and error handling (avoid runCatching) in producer 2022-10-18 21:25:18 +02:00
Adam
8342a923a8 kt2ts2.kt - minor style refactor 2022-10-18 21:22:06 +02:00