First the …
Server setup (the receiver)
On your central log host (Debian 12 / Ubuntu 24.04 or newer):
sudo apt update sudo apt install systemd-journal-remote
Create the storage directory and set ownership:
sudo mkdir -p /staorage/journal/remote sudo chown systemd-journal-remote:systemd-journal-remote /storage/journal/remote
Enable the socket (recommended over the service for on-demand behavior):
sudo systemctl enable --now systemd-journal-remote.socket
For production, add a drop-in to control listening mode and output path:
sudo systemctl edit systemd-journal-remote.service
Add:
[Service]
ExecStart=
ExecStart=/lib/systemd/systemd-journal-remote --listen-http=-3 --output=/storage/journal/remote
ReadWritePaths=/storage/journal/remote
If you want HTTPS (strongly recommended across untrusted networks), switch to –listen-https and supply certificate paths owned by the systemd-journal-remote user.
Firewall example (UFW) for the server
Open poort
sudo ufw allow 19532/tcp comment "systemd-journal-remote"
Retention and maintenance on the server
Zorg dat de journals niet teveeel ruimte cosumeren
Voorbeeld: maximaal 10 GB remote journals bewaren en bestanden ouder dan 30 dagen opruimen. Dat is ruim voldoende voor het onderzoeken van een Raspberry Pi die af en toe vastloopt.
Op de server (hier zoneminder):
du -sh /storage/journal/remote ls -lh /storage/journal/remote
Maak eerst een opruimscript:
sudo vi /usr/local/sbin/cleanup-remote-journals
Zet hierin:
#!/bin/bash
REMOTE_DIR="/storage/journal/remote"
MAX_SIZE=$((10 * 1024 * 1024 * 1024)) # 10 GiB
MAX_AGE_DAYS=30
# Verwijder remote journals ouder dan 30 dagen
find "$REMOTE_DIR" -type f -name '*.journal' -mtime +"$MAX_AGE_DAYS" -delete
# Als er nog steeds meer dan 10 GiB staat,
# verwijder dan de oudste journal-bestanden totdat het onder 10 GiB zit.
while [ "$(du -sb "$REMOTE_DIR" | awk '{print $1}')" -gt "$MAX_SIZE" ]; do
OLDEST=$(find "$REMOTE_DIR" -type f -name '*.journal' -printf '%T@ %p\n' \
| sort -n | head -n 1 | cut -d' ' -f2-)
[ -z "$OLDEST" ] && break
rm -f -- "$OLDEST"
done
Maak het uitvoerbaar:
sudo chmod 755 /usr/local/bin/cleanup-remote-journals
Systemd service maken
sudo vi /etc/systemd/system/cleanup-remote-journals.service
Inhoud:
[Unit] Description=Cleanup remote systemd journals [Service] Type=oneshot ExecStart=/usr/local/sbin/cleanup-remote-journals
Timer instellen
sudo vi /etc/systemd/system/cleanup-remote-journals.timer
Inhoud:
[Unit] Description=Daily cleanup of remote systemd journals [Timer] OnCalendar=daily Persistent=true [Install] WantedBy=timers.target
Daarna:
sudo systemctl daemon-reload sudo systemctl enable --now cleanup-remote-journals.timer
Controleer:
systemctl list-timers cleanup-remote-journals.timer --no-pager
Meteen één keer testen. Voordat we hem aan de automatische timer overlaten:
sudo systemctl start cleanup-remote-journals.service
Daarna:
du -sh /storage/journal/remote
En:
systemctl status cleanup-remote-journals.service --no-pager
Or …. run periodic vacuuming via a systemd timer:
journalctl --directory=/storage/journal/remote --vacuum-time=90d --vacuum-size=80G
Ik ga voor het script LOL
Query received journals
Received journals appear as remote-<hostname>.journal files. Query them with on the server (zoneminder) via:
sudo journalctl --directory=/storage/journal/remote --since "1 hour ago" sudo journalctl --file=/storage/remote/remote-web-01.journal -u nginx
If the storage mounted over nfs to another system then replace the path to /sotorage/remote accordingly, exemple:
arjan@arjan:/ $ sudo journalctl --directory=/mnt/zoneminder/storage/journal/remote --since "1 hour ago" arjan@arjan:/ $ sudo journalctl --file=/mnt/zoneminder/storage/remote/remote-web-01.journal -u nginx
Client setup (the uploader)
On every machine you want to forward logs:
sudo apt install systemd-journal-remote
Edit the upload configuration:
sudo systemctl edit systemd-journal-upload.service --full --force
Or simply edit /etc/systemd/journal-upload.conf:
[Upload] # The central server is URL=http://zoneminder.thuis.net:19532 # For TLS client certificate auth (recommended) # ServerKeyFile=/etc/ssl/journal/client.key # ServerCertificateFile=/etc/ssl/journal/client.crt # TrustedCertificateFile=/etc/ssl/journal/ca.crt
Enable and start:
sudo systemctl enable --now systemd-journal-upload.service
The service maintains /var/lib/systemd/journal-upload/state. If you ever need a full resend, delete that file.
TLS best practices
- Use a proper CA (step-ca, smallstep, or Let’s Encrypt for the server cert)
- Generate per-client certificates when possible
- Set chmod 600 on private keys and correct ownership
- On the server side, use –trust to point at your CA bundle
Self-signed certs work fine for homelabs; just distribute the CA and client certs securely (Ansible, scp with strict modes, or even systemd-creds).
Verification and troubleshooting
On a client:
journalctl -u systemd-journal-upload -f
On the server, watch incoming connections and check that new journal files appear with the expected hostname.
Common gotchas
Hier kun je tegenaan lopen
- Certificate permission errors (the journal-remote user must read the key)
- Firewall blocking 19532
- Time skew between client and server (journal entries have monotonic timestamps, but wall-clock correlation still matters)
When to layer something else on top
This setup gives you centralized, structured logs you can query with the same tools you already know. For full-text search, alerting, or long-term retention beyond a few months, point a lightweight shipper (Vector, Fluent Bit) at the remote journal directory or use the HTTP gateway (journal-gatewayd) to feed a proper backend.
The native tools keep the operational surface tiny and the data format lossless. In most homelab and small-team environments, that is exactly what you need.
References
systemd-journal-remote.service(8) and journal-remote.conf(5) man pages
DigitalOcean: “How To Centralize Logs With Journald on Ubuntu 20.04”
systemd source and issue tracker discussions on resume behavior and TLS handling
Orgineel: https://dev.to/lyraalishaikh/stop-scattered-logs-centralize-linux-journald-with-systemd-journal-remote-30ik