Every minute, the mod diffs the force's production statistics for each
surface against the totals it sent last time (kept in
storage.productionTotals), and sends what was produced and consumed as a
ProductionUpdate.
events-server-web keeps the statistics in ring buffers (1440 minutes and
168 hours), and serves them at
/kafkatorio/data/servers/{id}/production, for a range of game time and
optionally one surface.
The web map has a production page (/servers/{id}/production, linked from
the header) with science packs used per minute (in total and per pack),
production per minute as a line chart, and a searchable table of
everything produced and consumed.
export_building_icons.py now also exports every item and fluid icon and
name (items.json, fluids.json), for the production page.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The mod sends the research state of a force (researched technologies,
levels, current research, progress and queue) whenever research changes,
and every minute for the progress. events-server-web keeps the latest state
and serves it at /kafkatorio/data/servers/{id}/research.
The web map has a research page (/servers/{id}/research, linked from the
header), with:
- progress, the current research and the queue
- the science packs needed for everything that's left
- every technology, filterable by status, science pack and name, with its
cost (including levelled technologies' count formulas)
- details: the total cost to reach a technology, its prerequisites, what it
leads to and what it unlocks
The technology tree comes from export_building_icons.py, which now also
exports technologies.json, items.json and technology/science pack icons.
Kafka deleted data older than 7 days, but the building, resource and
research views are rebuilt by reading their topics from the beginning, so
buildings disappeared after a restart. These topics are now kept forever.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The services now run on Java 17, with a periodic G1 GC that returns unused
heap to the OS when idle. Previously Java 11 kept the committed heap, so
events-processors held 5.5 GB while its live data was ~250 MB.
All RocksDB state stores now share one bounded block cache and write buffer
(ROCKSDB_TOTAL_MEMORY_MB, default 256 MB), instead of each of the ~45 stores
having its own.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Under exactly_once_v2, Streams defaults the producer transaction timeout
to 10s. With commit.interval.ms raised to 10s (665cd1a), transactions on
the heavy map-aggregation topology timed out, got fenced
(ProducerFencedException / TaskMigratedException ~130 times during one
large chunk import), and their work was redone. Default the transaction
timeout to 60s, overridable via KAFKA_STREAMS_TRANSACTION_TIMEOUT_MS.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The zoom-out map layers aggregate 32x32 chunks into 256x256 and 512x512
chunks, holding up to 65k/262k tiles. Every incoming chunk rebuilt the
whole aggregate, and with exactly_once_v2's default 100ms commit
interval nearly every intermediate result was forwarded downstream,
cascading into the next level. A small map also has few 512-level keys,
so this work landed on one partition and one thread.
Make commit.interval.ms and cache.max.bytes.buffering configurable
(defaults unchanged), and in the Unraid example use a 10s commit
interval with a 128MB cache, so repeated updates to the same chunk are
merged in the record cache first. Measured on a live backlog: 41 -> 575
chunks/minute. Map updates now appear within ~10s instead of instantly.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- config.yml: num.stream.threads can be set with
KAFKA_STREAMS_NUM_THREADS (default unchanged: 1). Map processing was
bottlenecked on a single stream thread while the host was mostly idle.
- Unraid example: run 3 stream threads (topics have 3 partitions), and
cap each JVM's heap via JAVA_OPTS. Without a cap each JVM defaults to
1/4 of the host's RAM, and the stack grew to ~14 GB.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>