Commit graph

18 commits

Author SHA1 Message Date
Lazzexus
0e1b315a6d Add a production dashboard
Every minute, the mod diffs the force's production statistics for each
surface against the totals it sent last time (kept in
storage.productionTotals), and sends what was produced and consumed as a
ProductionUpdate.

events-server-web keeps the statistics in ring buffers (1440 minutes and
168 hours), and serves them at
/kafkatorio/data/servers/{id}/production, for a range of game time and
optionally one surface.

The web map has a production page (/servers/{id}/production, linked from
the header) with science packs used per minute (in total and per pack),
production per minute as a line chart, and a searchable table of
everything produced and consumed.

export_building_icons.py now also exports every item and fluid icon and
name (items.json, fluids.json), for the production page.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-22 10:26:25 +01:00
Lazzexus
8770a0617b Add a research page, and keep the topics that views are rebuilt from
The mod sends the research state of a force (researched technologies,
levels, current research, progress and queue) whenever research changes,
and every minute for the progress. events-server-web keeps the latest state
and serves it at /kafkatorio/data/servers/{id}/research.

The web map has a research page (/servers/{id}/research, linked from the
header), with:
- progress, the current research and the queue
- the science packs needed for everything that's left
- every technology, filterable by status, science pack and name, with its
  cost (including levelled technologies' count formulas)
- details: the total cost to reach a technology, its prerequisites, what it
  leads to and what it unlocks

The technology tree comes from export_building_icons.py, which now also
exports technologies.json, items.json and technology/science pack icons.

Kafka deleted data older than 7 days, but the building, resource and
research views are rebuilt by reading their topics from the beginning, so
buildings disappeared after a restart. These topics are now kept forever.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 10:46:09 +01:00
Lazzexus
ba9cbe5afe Reduce memory use of the JVM services
The services now run on Java 17, with a periodic G1 GC that returns unused
heap to the OS when idle. Previously Java 11 kept the committed heap, so
events-processors held 5.5 GB while its live data was ~250 MB.

All RocksDB state stores now share one bounded block cache and write buffer
(ROCKSDB_TOTAL_MEMORY_MB, default 256 MB), instead of each of the ~45 stores
having its own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 10:31:04 +01:00
Lazzexus
d5f722b998 Raise Kafka Streams transaction timeout above the commit interval
Under exactly_once_v2, Streams defaults the producer transaction timeout
to 10s. With commit.interval.ms raised to 10s (665cd1a), transactions on
the heavy map-aggregation topology timed out, got fenced
(ProducerFencedException / TaskMigratedException ~130 times during one
large chunk import), and their work was redone. Default the transaction
timeout to 60s, overridable via KAFKA_STREAMS_TRANSACTION_TIMEOUT_MS.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 07:00:06 +02:00
Lazzexus
424e12309d Speed up map processing 14x by merging aggregate updates before forwarding
The zoom-out map layers aggregate 32x32 chunks into 256x256 and 512x512
chunks, holding up to 65k/262k tiles. Every incoming chunk rebuilt the
whole aggregate, and with exactly_once_v2's default 100ms commit
interval nearly every intermediate result was forwarded downstream,
cascading into the next level. A small map also has few 512-level keys,
so this work landed on one partition and one thread.

Make commit.interval.ms and cache.max.bytes.buffering configurable
(defaults unchanged), and in the Unraid example use a 10s commit
interval with a 128MB cache, so repeated updates to the same chunk are
merged in the record cache first. Measured on a live backlog: 41 -> 575
chunks/minute. Map updates now appear within ~10s instead of instantly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 19:43:17 +02:00
Lazzexus
5ad3e6f27b Make Kafka Streams thread count configurable, cap JVM heaps on Unraid
- config.yml: num.stream.threads can be set with
  KAFKA_STREAMS_NUM_THREADS (default unchanged: 1). Map processing was
  bottlenecked on a single stream thread while the host was mostly idle.
- Unraid example: run 3 stream threads (topics have 3 partitions), and
  cap each JVM's heap via JAVA_OPTS. Without a cap each JVM defaults to
  1/4 of the host's RAM, and the stack grew to ~14 GB.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 19:21:22 +02:00
Adam
3f771f9838 fix bootstrap server, remove http:// 2022-10-23 21:54:38 +02:00
Adam
ce0d4b7378 [events-processors] change where topo description is written 2022-10-22 19:52:02 +02:00
Adam
f99208b5b2 more fixing of Kafkatorio packets... 2022-10-18 23:22:26 +02:00
Adam
dda48bac78 refactor the KafkatorioPacket tag that the mod logs, and the syslog-server parses 2022-10-18 21:35:49 +02:00
Adam
1cc5afa6e2 set default socket-server host to 0.0.0.0 2022-10-17 00:12:03 +02:00
Adam
b0722487a0 fix appProps in events-server-web 2022-10-08 14:20:01 +02:00
Adam
6035c2eaa0 move kafka topics creator into events-processors 2022-10-08 14:16:43 +02:00
Adam
46db151f6e rename SERVER_PORT -> WEB_SERVER_PORT 2022-10-02 14:03:51 +02:00
Adam
f12a505d22 migrate buildSrc plugins to includedBuild plugins 2022-09-18 20:49:32 +02:00
Adam
bb973c7b9b fix tests 2022-09-16 22:31:32 +02:00
Adam
873983f9a3 big ol' refactor to split up the events-server and events-processors 2022-09-13 17:56:57 +02:00
Adam
c8815726b1 start splitting up events-server 2022-09-11 00:48:16 +02:00