| commit | 0d31208af5094313ed978389b9e0ae75802ccd9d | [log] [tgz] |
|---|---|---|
| author | Jan Kundrát <jan.kundrat@cesnet.cz> | Wed Jan 17 18:28:20 2024 +0100 |
| committer | Jan Kundrát <jan.kundrat@cesnet.cz> | Thu Jan 18 04:37:47 2024 +0100 |
| tree | 8745961983e5281b53bfb8608b626aaeaaf5488d | |
| parent | b0b2dc58dfd0863a1d72938ce564e09c877fae9d [diff] |
improve tracing This started by me not being able to get `perf top` working, but in the end I got additional tools as well. The `rtla timerlat` in particular was useful to show me that a kernel thread waited up to 220ms at some time, which means that it's about 10 times longer than the cutoff threshold for reading 128 bytes out of the HW FIFO of MAX14830 running at 57600 Baud. That might very well explain the FIFO overruns we saw on the EDFAs. Anyway, `perf top` still doesn't produce any useful output by default, but the `-e cpu-clock` option appears to do the trick -- even if it's apparently a SW metric. At least I see *something*. Enabling CoreSight (BR2_PACKAGE_OPENCSD) had no effect, apparently because our kernel doesn't enable that. Looking at the config options in the kernel, that thing appears to be used on arm64, with some scary mentions of "not being supported for 32bit programs". OTOH, the SoC funciton specification *does* mention some coresight bits, but I have no clue what it's all about, and the DTS appears to have no references to anything CoreSight-related, so, well, let's not use that. The kernel update was needed to enable the kernel's `perf` tool to find `pkg-config` under Buildroot (branch `cesnet/2024-01-17`). Buildroot was then updated so that these optional libs are indeed available. Change-Id: I4779badd3b7df452b402815629edcef06abdb9b4
This repository contains CzechLight-specific bits for Buildroot. Buildroot is a tool which produces system images for flashing to embedded devices. They have a nice documentation which explains everything that one might need.
The system architecture is described in another document. This is a quick build HOWTO.
Everything is in Gerrit. One should not need to clone anything from anywhere else. The build will download source tarballs of various open source components, though.
By default, each change of this repo uploaded to Gerrit causes the CI system to produce a firmware update. On Gerrit, the change will get a comment from Zuul with a link to the CI log server. Next to the logs, a file named artifacts/update.raucb can be used for updating devices.
Behind the scenes, the system uses Zuul with a configuration tracked in git.
Here's how to reproduce the build on a developer's workstation:
git clone ssh://$YOUR_LOGIN@cesnet.cz@gerrit.cesnet.cz:29418/CzechLight/br2-external czechlight pushd czechlight git submodule update --init --recursive popd mkdir build-clearfog cd build-clearfog ../czechlight/dev-setup-git.sh make czechlight_clearfog_defconfig make -j8
A full rebuild takes about half an hour on a modern laptop.
WARNING: Buildroot is fragile. It is not safe to perform incremental builds after changing an "important" setting. Please check their manual for details.
Apart from the traditional way of re-flashing the SD card or the eMMC from scratch, it's also possible to use RAUC to update. This method preserves the U-Boot version and the U-Boot's environment. Apart from that, everything starting with the kernel and the DTB file and including the root FS is updated. Configuration stored in /cfg is brought along and preserved as well.
To install an update:
# build node make rsync -avP images/update.raucb somewhere.example.org:path/to/web/root # target, perhaps via an USB console or over SSH rauc install http://somewhere.example.org/update.raucb reboot
Once the updated FW slot boots, the configuration in /cfg will be automatically upgraded ("migrated") to the newest layout. A downgrade to an incompatible OS version might therefore fail during the next reboot. Completely removing all data in the newly updated slot's cfg partition will restore functionality, but it is effectively a factory reset.
On a regular Clearfog Base with an eMMC, one has to bootstrap the device first. If recovering a totally bricked board (or one that is fresh from factory), use the kwboot command to upload the initial, new enough U-Boot via the console. Ensure that the jumpers are set to 0 1 0 0 1 (default for eMMC boot is 0 0 1 1 1), and then use U-Boot's kwboot tool:
./host/bin/kwboot -b ./u-boot-spl.kwb -t -p /dev/ttyUSB0
Prepare a USB flash disk with a raw bootable image, images/usb-flash.img. Use a tool such as dd to overwrite the raw block device, do not copy the image file. Once in U-Boot, plug the USB flash disk and execute:
usb start; fatload usb 0:1 00800000 boot.scr; source 00800000
The system will boot and flash the eMMC from the USB drive. Once the status LED starts blinking in yellow, data are being transferred to the eMMC. The light changes to solid yellow in later phases of the flashing process. Once everything is done, the status LED shows a solid white light and the system reboots automatically.
Turn off power, remove the USB flash, re-jumper the board (0 0 1 1 1), power-cycle, and configure MAC addresses at the U-Boot prompt. The MAC addresses are found on the label at the front panel.
=> setenv eth1addr 00:11:17:01:XX:XX => setenv eth2addr 00:11:17:01:XX:YY => setenv eth3addr 00:11:17:01:XX:ZZ
Also set up the system type:
| Model | czechlight variable value |
|---|---|
| ROADM Line Degree | sdn-roadm-line-g2 |
| WSS Add/Drop | sdn-roadm-add-drop-g2 |
| Hi-resolution Add/Drop | sdn-roadm-hires-add-drop-g2 |
| Coherent Add/Drop | sdn-roadm-coherent-a-d-g2 |
| Inline EDFA Amplifier | sdn-inline-g2 |
Some prototypes have deprecated PCBs (blue). On these, skip the -g2 suffix. All red PCBs are -g2.
=> setenv czechlight sdn-roadm-line-g2 => saveenv Saving Environment to MMC... Writing to redundant MMC(0)... OK => boot
Once the system boots (which currently requires a reboot for some unknown reason -- fsck, perhaps?), configure hostname, plug in the network cable, and update SW:
# hostnamectl set-hostname line-XYZSERIALNO # cp /etc/hostname /cfg/etc/ # rauc install http://somewhere.example.org/update.raucb # reboot
Obtain a reasonable Linux distro image for BBB and flash it to a µSD card. Unlock eMMC boot partitions (echo 0 > /sys/class/block/mmcblk1boot0/force_ro; echo 0 > /sys/class/block/mmcblk1boot1/force_ro). Clean the eMMC data (blkdiscard /dev/mmcblk1). Flash the content of images/emmc.img to device's /dev/mmcblk1. Flash what fits into /dev/mmcblk1boot0 and /dev/mmcblk1boot1. Fetching the image over web (python3 -m http.server and wget http://...:8000/emmc.img -O - | dd of=/dev/mmcblk1 conv=sparse) works well.