A clear demarcation between a green and a seasoned firmware (FW) engineer is the level of understanding, care, and planning that is evident in their work. The green engineer typically believes that “it works” is the only criterion of importance in their work product. It’s not entirely their fault, as many college courses, and even inexperience managers, often believe this is the only important criterion. Of course, it is critically important that things work, but there are other attributes of a professional firmware project that a seasoned engineer has learned, often the hard way, are vitally important. The green engineer eventually learns that a poorly planned and hastily implemented solution probably does not actually work all that well in the long run. The solution may perform its main function fine, but if it is not maintainable, extensible, and supportable by someone else, the solution may not have any long- term value for the company.
Build System Replication
One aspect of delivering a professional firmware solution is the ability to control the build toolchain configuration so that it can be reliably replicated. There are many reasons why this is important:
- If there is more than one developer on the team, it is critical that they use exactly the same toolchain. When this doesn’t happen, defects can arise that only show up when built by one of the engineers. Defects that cannot be replicated are extremely difficult to resolve. This type of issue arises more often than one might think.
- Modern development practices rely on a continuous test and integration process to identify defects as soon as possible, reducing the effort required to resolve them. These processes are typically run on a continuous integration server. If that server runs a different version of components of the toolchain than the developer(s), then the same sort of issues described in the previous bullet will arise.
- If the developer leaves the company and there is no way to replicate their build environment, it may be impossible to spin up a new developer to work on the firmware. Given the short length of employment spans found on many resumes today, the wise manager makes sure that the build system can be replicated for the next engineer that who will own the code.
- In industry, it is quite common to need to perform maintenance updates or implement new features months or even years after the initial product release. If updated tools break your build process or change the FW binary in a meaningful way, then these types of updates may be impossible.
- With Graphical User Interface (GUI) based build systems in particular, the build configuration can be quite fragile. In the internal Eclipse build system for example, the build configuration is stored in a huge XML file along with many other settings. It is really easy for a developer to change a simple setting in a dialog nested several layers down and break the build in some manner. Without text configuration files that can be easily compared, hunting down these changes are an exercise in futility.
In order to provide reliable, repeatable replication of FW build environments, Simplexity has selected a combination of Linux and Docker for our FW development system.
In order to provide reliable, repeatable replication of FW build environments, Simplexity has selected a combination of Linux and Docker for our FW development system.
Linux as a Build System
Simplexity has chosen Linux for a build system for a variety of reasons:
- Linux is widely known as a developer friendly environment. While most people eschew command line interfaces for GUI’s (and they are superior for many uses), the fact is that most FW development GUI’s are just GUI’s on top of command line tools. In addition, FW tool usage is rarely exploratory. There’s little or no advantage to being able to click your way through different steps in order to find an optimal solution. Most FW tool usage is simply repeating the same operations over and over.
- Command line tools are much easier to automate and chain together than are GUI tools, and the automation tools are typically very familiar to developers. These automation scripts are easy to revision control, and analyzing differences is as simple as comparing two text files, a standard development process. The Linux environment offers rich support for command line and text file operations.
- Linux is generally high performance, and many tools support parallel operations to speed up builds, allowing for rapid iteration.
- Simplexity uses Atlassian’s Bitbucket as our git source code repository hosting service. Bitbucket uses Linux machines for its hosted continuous integrated server.
- Bitbucket uses Docker on top of Linux for continuous integration.
So, what is this Docker thing? Simply put, Docker is a lightweight virtualization system. It is a system to allow you to configure and run virtual machines on your PC. Virtual machines allow one to recreate a full machine configuration easily and exactly, the key to development environment reproducibility. Docker in particular, is a tool originally created to allow companies to easily replicate web servers so they can scale web service bandwidth up and down as needed. Docker supports a text file-based configuration tool for a virtual machine, allowing for easy configuration control. As noted above, Docker was originally created to run Linux virtual machines on Linux servers, so it plays nicely with our desire to use Linux as our firmware development environment.
Until mid-2020, there was one serious wrinkle in Simplexity’s Linux/Docker based firmware development environment: Simplexity is a Windows shop. Recent data shows that approximately 75% of all desktop PC systems run some form of Microsoft Windows. In companies where Windows is standard, IT prefers to only have to maintain one flavor of OS, so many developers face this challenge. Running a Linux development environment in a Windows world was not a trivial endeavor. We had tried various flavors of Linux-like development environments on our Windows PC’s and found them all lacking:
- Cygwin – Provides a command line Linux emulator for the Windows command prompt. We found it hard to control the configuration of the Cygwin environment itself, and, at the time, we were home brewing software install scripts, which was error prone and resource intensive.
- VirtualBox – VirtualBox is a full-blown virtual machine program. VirtualBox VM’s were easier to replicate than Cygwin installs, and we were able to get build system configuration control by using Ansible. On the downside, the full VM’s were resource intensive, access to USB devices was not ideal, and mounting of file systems between the VM and Windows host required a lot of manual setup.
- Docker Desktop – The next step for Simplexity was the move to Docker Desktop. The virtualization technology was similar to VirtualBox, but a bit less resource intensive as Docker VM’s are “lightweight” in that they share a set of common OS components, then layer installation specific components on top of that. This dramatically reduced the disk space required per VM. Like VirtualBox, Docker Desktop can use the Hyper-V technology to virtualize the machine at the CPU level, providing high CPU performance. The big benefit of the move to Docker was the machine configuration control provided natively by Docker, eliminating the need for the Ansible configuration. The downside to this setup was with accessing source and build artifacts. Because of the way Docker Desktop worked prior to the release of Microsoft’s second version of the Windows Subsystem for Linux (discussed below), file system performance was either painfully slow (if hosting source code on the Windows side), or you had to re-clone your repo every time you fired the VM up (Docker always strives to provide a “fresh” machine). Simplexity operated with this setup for almost two years, but the disk I/O performance was a constant issue.
Problems Solved
In 2020, Microsoft released a major upgrade to its Windows Subsystem for Linux (WSL) with version 2. With WSL2, Microsoft offered a Hyper-V enabled Linux VM as a Windows application. Using WSL2, you can now run a full-blown Linux installation right alongside Windows with little or no performance penalty. In addition, the Docker team released a version of Docker Desktop that runs on top of WSL2 instead of providing its own copy of the Linux system. With this combination, there was now an essentially “native” Docker on the Linux system available on Windows offering close to Linux native CPU and file system I/O performance. In addition, by storing the source repository tree within the WSL Linux filesystem it is retained between runs of the Docker VM. In Simplexity’s testing, we found no meaningful FW build performance differences between running on a WSL2+Docker and a native Linux install running the same toolchain.
In addition, with WSL2, Microsoft has provided some additional tools that simplify working with the Linux file system with standard Windows tools:
- The WSL2 file systems (you can have more than one Linux distro installed) are exposed as network machines at the UNC path “\\wsl$”. Navigation to the files is fairly simple, and can be made even simpler by directly mapping a standard Windows drive to any point in that file system. The WSL2 files can be treated like any Windows network drive.
- With Microsoft’s “Remote – WSL” and “Remote – Container” plugins for their VS Code editor, you can run a Windows native editor/ that can directly work on remote file systems. This alleviates the need to install GUI toolkits on the WSL2 system to get a high-quality IDE experience.
By utilizing WSL2 and Docker, Simplexity has a FW development environment that supports easy configuration control, high performance, and best-in-class command line and IDE tools. We’re able to replicate our tools for developers by running a single script, use the exact same toolchain as our Bitbucket Continuous Integration server, as well as archive our build configurations for the future, all using standard, open-source tools that are developed and managed by third parties. This allows us to get the configuration control that we require without a large investment in tools so that we can focus on our core Firmware development. In addition, our customers can easily replicate the toolchain and take it with them as they spin up their own firmware development teams.
Software: Design Complete
Hardware: Pre-production units for design verification
MFG. Readiness: CM schedule and budget, Unit build tracking