Remote systemd journald logging
TL;DR: `echo DESTINATION=http://10.0.254.102 > /cfg/journald-remote` now
activates sending of everything into a remote server.
This was more complex than I expected (ha ha):
- By default, this unit pulls in network-online.target, and that one is
FUBAR on our systemd-networkd setup. That in turn leads to a "failed
unit" check, which transitions the whole system into `degraded`, and
that one leads to a slow and painful watchdog death.
- Even if everything worked, this is a network thingy, and as such it's
rather fragile -- it can die at any time (a broken connection terminates
this service with a failure). That's once again something which leads to
a `degraded` state.
So, let's fix everything by:
- auto-restarts (with a sufficient timeout, etc)
- ignoring network,
- some conditional enablement.
This appears to work, which is awesome if you ask me.
Change-Id: I83ae10d52b35eafd633e77b78b02580c041496b4
diff --git a/board/czechlight/clearfog/overlay/usr/lib/systemd/system/multi-user.target.wants/systemd-journal-upload.service b/board/czechlight/clearfog/overlay/usr/lib/systemd/system/multi-user.target.wants/systemd-journal-upload.service
new file mode 120000
index 0000000..f70515b
--- /dev/null
+++ b/board/czechlight/clearfog/overlay/usr/lib/systemd/system/multi-user.target.wants/systemd-journal-upload.service
@@ -0,0 +1 @@
+../systemd-journal-upload.service
\ No newline at end of file
diff --git a/board/czechlight/clearfog/overlay/usr/lib/systemd/system/systemd-journal-upload.service b/board/czechlight/clearfog/overlay/usr/lib/systemd/system/systemd-journal-upload.service
new file mode 100644
index 0000000..985b328
--- /dev/null
+++ b/board/czechlight/clearfog/overlay/usr/lib/systemd/system/systemd-journal-upload.service
@@ -0,0 +1,43 @@
+# CzechLight-specific configuration:
+#
+# - do not record a failed uint when that thing dies (likely due to a network issue)
+# - only start when a custom config file exists
+# - do not bring in a network-online.target because that might trigger
+# extra failure reports (and a watchdog action)
+# - keep retrying upon failure(s)
+
+[Unit]
+Description=Journal Remote Upload Service
+
+[Service]
+DynamicUser=yes
+LockPersonality=yes
+MemoryDenyWriteExecute=yes
+PrivateDevices=yes
+ProtectControlGroups=yes
+ProtectHome=yes
+ProtectHostname=yes
+ProtectKernelModules=yes
+ProtectKernelTunables=yes
+RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
+RestrictNamespaces=yes
+RestrictRealtime=yes
+StateDirectory=systemd/journal-upload
+SupplementaryGroups=systemd-journal
+SystemCallArchitectures=native
+User=systemd-journal-upload
+WatchdogSec=3min
+LimitNOFILE=524288
+
+# - ignore failures
+# - read (one) location from the env file
+ExecStart=-/usr/lib/systemd/systemd-journal-upload --save-state --url=${DESTINATION}
+EnvironmentFile=-/cfg/journald-remote
+
+# run forever and ignore any network issues
+Restart=always
+RestartSec=5
+
+[Unit]
+# shared as an env file
+ConditionFileNotEmpty=/cfg/journald-remote
diff --git a/configs/czechlight_clearfog_defconfig b/configs/czechlight_clearfog_defconfig
index 67ce188..1a2119c 100644
--- a/configs/czechlight_clearfog_defconfig
+++ b/configs/czechlight_clearfog_defconfig
@@ -55,6 +55,7 @@
BR2_PACKAGE_DDRESCUE=y
BR2_PACKAGE_RAUC=y
BR2_PACKAGE_RAUC_NETWORK=y
+BR2_PACKAGE_SYSTEMD_JOURNAL_GATEWAY=y
# BR2_PACKAGE_SYSTEMD_HWDB is not set
BR2_PACKAGE_SYSTEMD_RANDOMSEED=y
# BR2_PACKAGE_SYSTEMD_VCONSOLE is not set
diff --git a/package/czechlight-rauc/rauc-hook.sh b/package/czechlight-rauc/rauc-hook.sh
index c701c3e..b6ace5c 100755
--- a/package/czechlight-rauc/rauc-hook.sh
+++ b/package/czechlight-rauc/rauc-hook.sh
@@ -7,10 +7,11 @@
# whitelist so that we don't copy cruft or lost+found
for ITEM in \
etc \
+ journald-remote \
random-seed \
ssh-user-auth \
; do
- if [[ -d /cfg/${ITEM} ]]; then
+ if [[ -d /cfg/${ITEM} || -f /cfg/${ITEM} ]]; then
cp -a /cfg/${ITEM} ${RAUC_SLOT_MOUNT_POINT}/
fi
done