Commit graph

29 commits

Author SHA1 Message Date
Lazzexus
ba9cbe5afe Reduce memory use of the JVM services
The services now run on Java 17, with a periodic G1 GC that returns unused
heap to the OS when idle. Previously Java 11 kept the committed heap, so
events-processors held 5.5 GB while its live data was ~250 MB.

All RocksDB state stores now share one bounded block cache and write buffer
(ROCKSDB_TOTAL_MEMORY_MB, default 256 MB), instead of each of the ~45 stores
having its own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 10:31:04 +01:00
Lazzexus
ce50d9f1bb Fix syslog server silently dropping packets under load
The syslog parser reuses its result map for each line it parses, and
SyslogMsg kept a reference to that map instead of a copy. The producer
consumes messages concurrently with the reader, so by the time a message
was checked, its map had sometimes already been overwritten by the next
line; the message then failed the 'KafkatorioPacket:::' filter and was
dropped without any log. Under real load (bursts of map chunk packets)
most packets were lost.

Copy the parsed map, and log Kafka producer send failures instead of
ignoring them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 18:58:21 +02:00
Lazzexus
f6501a5020 Fix syslog server crashing whenever a client disconnects
When a client closed its connection, handleConnection threw on the
null line, closed the socket, and then logged socket.description() -
which reads the address of the now-closed socket and throws
ClosedChannelException. That escaped the handler and crashed the
whole server, dropping every other client's connection. With the
Docker syslog log driver in non-blocking mode, the Factorio server's
logs were then silently discarded, so most packets never arrived.

Treat a null line as a normal client disconnect, capture the socket
description while it's still open, and always close the socket in
'finally', so one connection can't take down the server.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 18:46:13 +02:00
Adam
5562e86f09 [docker-files] remove user config 2022-10-22 23:36:00 +02:00
Adam
82c681e320 [dockerfiles] remove volumes 2022-10-22 20:35:52 +02:00
Adam
45691a049b [dockerfiles] fix user name/id variables 2022-10-22 20:27:17 +02:00
Adam
c9d21fa25a try fixing user permissions in containers 2022-10-22 20:24:27 +02:00
Adam
9b331cc105 update dockerfiles
- define user uid/gid
- use new geedeecee files (hopefully means smaller images, faster building)
2022-10-22 19:55:02 +02:00
Adam
f99208b5b2 more fixing of Kafkatorio packets... 2022-10-18 23:22:26 +02:00
Adam
ecf680d9ce [syslog-server] minor styling 2022-10-18 21:26:30 +02:00
Adam
557dc97635 [syslog-server] refactor logging and error handling (avoid runCatching) in producer 2022-10-18 21:25:18 +02:00
Adam
08f23eebd7 [syslog-server] fix server host/port mixup in dockerfile 2022-10-17 00:18:07 +02:00
Adam
0216d068c5 [syslog-server] ignore, but log, non-syslog messages. Minor logging tweaks. 2022-10-17 00:14:27 +02:00
Adam
1cc5afa6e2 set default socket-server host to 0.0.0.0 2022-10-17 00:12:03 +02:00
Adam
33255f2d1b [syslog-sever] rm unnecessary @PublishedApi from internal jsonMapper 2022-10-17 00:10:21 +02:00
Adam
0f9e51df31 refactor syslog server to use Dispatchers.IO and SupervisorScope (try to fix connection refused error) 2022-10-16 12:34:07 +02:00
Adam
6a439b334d try fixing syslog port to be 9002, and expose it using an env-var 2022-10-15 20:41:31 +02:00
Adam
18bdfb70b1 fix dockerfile entrypoint run script 2022-10-08 13:50:16 +02:00
Adam
4c5ed2dc73 give +x to docker apps, tidy up docker conf 2022-10-08 13:05:24 +02:00
Adam
6507f13462 fix docker image building 2022-10-05 08:52:39 +02:00
Adam
b61ee6bf5c apply geedeecee plugin, or update config. Update kvision dependency helper. 2022-10-04 22:52:19 +02:00
Adam
562296aa96 Dockerfiles: add syntax=docker header 2022-10-04 22:28:04 +02:00
Adam
0044f0cb98 create and use APP_NAME, APP_GROUP in docker compose files 2022-10-04 22:27:43 +02:00
Adam
99d1f92bbb update docker image names to include REGISTRY_HOST, and group, and correct image name 2022-10-02 14:51:00 +02:00
Adam
f804c39a3b migrate docker-compose convention plugin out of buildSrc 2022-10-02 12:11:06 +02:00
Adam
9071148944 start dockerising the modules... 2022-09-18 23:38:15 +02:00
Adam
f12a505d22 migrate buildSrc plugins to includedBuild plugins 2022-09-18 20:49:32 +02:00
Adam
873983f9a3 big ol' refactor to split up the events-server and events-processors 2022-09-13 17:56:57 +02:00
Adam
c8815726b1 start splitting up events-server 2022-09-11 00:48:16 +02:00