System Software Architecture of Czech Light Embedded Linux

These are important "features" or "design decisions" which drove the overall architecture:

  • Users are supposed to only interact with these devices via YANG data. If some functionality is not available through one of enabled models, that's a bug which should be (eventually) fixed.

  • Devices are updated in one step. There are no separate user-visible or individually updateable packages. There's one software image which drives "the whole device".

  • "Nobody" is expected to log in into the system. SSH connections are supported, but users should not have a local shell. Remote SSH connection should go to a CLI which manipulates the YANG datastores.

Configuration

All configuration "should be" stored in sysrepo as YANG-formatted data. Of course there are still (Q2 2020) exceptions:

  • hostname
  • timezone
  • SSH keys for system accounts
  • passwords for direct login, which should not be supported anyway
  • ...

YANG

The system always boots with an empty sysrepo database -- no YANG modules, and no configuration is installed. The required modules (and configuration) is added in several steps:

  • the YANG modules for netopeer2-server are added via netopeer2-install-yang.service (via our Buildroot patches),
  • CzechLight-specific YANG modules and their initial data are added via czechlight-install-yang.service,
  • system configuration is restored from the persistent location in /cfg via cfg-restore-sysrepo.service,
  • configuration of the Netopeer server gets re-checked via netopeer2-setup.service (once again in our Buildroot patches); this is needed especially during the first boot with no previous configuration to restore,
  • configuration of NACM is applied via nacm-restore.service,
  • finally, any daemons that use sysrepo are started.

We are also using a tmpfs mount at /run/sysrepo that gets wiped out whenever a sysrepo service fails. This is currently (Q1 2021) needed because of the risk of corrption of the in-memory sysrepo SHM data.

Factory data

MAC addresses and some identification data and in future also calibration data. As of 2020-04, MAC addresses and device type information (czechlight=...) are in U-Boot env.

Boot Flow

1) Early boot:

  • Hardware watchdog timer gets activated, and U-Boot starts pinging it.

  • U-Boot: there's a script that determines what RAUC slot to use. This part is supposed to be immutable. Nothing updates this script as it's hard-embedded into the bootloader, and nothing updates the bootloader in the field because that is hard to do in a failsafe manner.

  • U-Boot environment is marked as "I'm trying to boot an A/B slot now" while keeping track of how many attaempts are remaining.

  • Once an A/B boot slot is chosen, another U-Boot script (which is a part of every system release and therefore updateable) changes LED blinking patterns, loads the device tree (such as this one) which provides Linux-level device description, and launches the kernel.

2) Linux Kernel:

  • HW watchdog keeps running, but U-Boot has now ceased its periodic ping. We have about 90 seconds to start pinging it, otherwise the system reboots once the watchdog fires.

  • An init shell script sets up overlayfs so that rootfs' /etc is writable, but all changes are discarded on reboot.

  • Some parts of /etc are persistent (sysrepo configuration in the startup datastore, SSH keys for local shell). These are copied from /cfg, our R/W configuration partition, into their "final" destination.

  • Bootup is controlled via systemd and its targets; it's a pretty stock configuration.

3) Useful services:

The boot is driven by device information passed from U-Boot via the czechlight=... kernel command line parameter.

Hacks and Development Access

Root Access

Login as root over the serial console (µUSB port at the front panel, serial emulation via an FTDI chip, 115'200 Baud). On Linux, the command typically looks like picocom -b 115200 /dev/ttyUSB0. There's no password.

SSH into the System

Unlike the NETCONF access which has its own SSH key store (but uses regular system-wide accounts for UIDs, etc), there's a separate "store" of SSH keys for direct shell access. Put public keys into /cfg/ssh-user-auth/$USERNAME (configured here).

Debugging, Installing Custom Software, Packages, Utilities, etc

There's no package manager, so it's impossible to install anything. Buildroot always generates a full filesystem image that can be loaded to a device, for example via RAUC.

Development of on-device software

Buildroot supports local.mk along with *_OVERRIDE_SRCDIR. This is the mechanism that is used to select the specific version of our internal packages (e.g., cla-sysrepo) during the build process via dev-setup-git.sh. Either edit the local.mk to point, e.g., CLA_SYSREPO_OVERRIDE_SRCDIR to your cla-sysrepo checkout, or work in the checkout directly.

Debugging

Sometimes /tmp could be useful for fast prototyping. Here's how one could be debugging code on device:

user@laptop ~/build/br-cfb $ make cla-sysrepo-reconfigure
user@laptop ~/build/br-cfb $ scp per-package/cla-sysrepo/target/usr/bin/cla-sysrepod root@10.10.10.228:/tmp/
user@laptop ~/build/br-cfb $ ssh root@10.10.10.228 gdbserver :33666 /tmp/cla-sysrepod ...some args here...
user@laptop ~/build/br-cfb $ ./host/usr/bin/arm-linux-gdb \
  per-package/cla-sysrepo/target/usr/bin/cla-sysrepod --ex 'set sysroot target/' \
  --ex 'target remote 10.10.10.228:33666' --ex c

The core "NETCONF-related" packages are not stripped, so that some debugging should be possible.