BETTER FIRMWARE FASTER

A team from Simplexity recently attended Jack Ganssle’s “Better FW Faster” class. If you’re involved in the practice of embedded systems firmware engineering, you’ve probably heard of Jack and likely get his weekly newsletter. For those of you that haven’t heard of him, I highly recommend that you check out the many articles and white papers available on the Ganssle Group website. There’s a wealth of information available for free. While you’re at it, check out the great content available at EmbeddedGurus also.

Working in an engineering services company, the product we sell is our time. Taking four revenue generating engineers away from billable work for a full day is an expensive proposition. So, was it worth it? I’d answer that with a resounding “yes”. Mr. Ganssle’s rapid fire slide deck of high-quality material and intelligent delivery kept the team engaged and imparted a lot of wisdom. One of our team members was up half the night with his nine-month old, accidentally drank decaf coffee all morning, and was still leading our team’s debrief of the morning session over lunch.

Of the wide ranges of topics related to embedded firmware engineering presented in the class, here are some of our favorites.

Scope Probes

Embedded software engineering is a contact sport; you’re going to be in close contact with your hardware during development. There is a lot of crossover with electrical engineering. Since embedded systems rarely have a screen (a modern car has approximately 50 processors and one or two screens), other tools are required to support debugging. It is very common to use an oscilloscope to probe different pins that have been allocated to support debugging. If you’re not careful, simply applying a scope probe to a pin can blow a part.

Microprocessor clock speeds are continually increasing. With that speed increase comes an increase in the frequency content of the signals. If you’re probing a high-speed signal with a probe that has relatively high capacitance, its impedance is going to be very low for any high frequency current. This can lead to pulling more current than most microcontroller pins can source, causing incorrect electrical states or burned up electronics. It’s important that the EE and firmware engineer work closely to make sure the correct probes are used for the electronics.

Metrics

Engineers generally understand that feedback stabilizes a system. We often forget that human processes like product development can also benefit from feedback. To implement a feedback process, we need to take measurements. Collecting metrics on the code base and development process allows you to assess the present situation, take corrective action, and verify that those actions (usually new tools and processes) are helping.

Some of the recommended metrics for embedded firmware development are:

• Defect counts – Delivering quality firmware requires elimination of defects. If you’re not keeping track of your defects, it’s hard to know if you’ve resolved them all. You should know how many open defects you have, who discovered them, and the phase of development in which they were discovered.
• Defect rates (and resolution rates!) – Combining the defect counts with information about development phases or lines of code allows you to assess the defect rates or defect density for your process. This has the effect normalizing metrics for project size. In turn, you can now use project size estimates to estimate the expected defect count.
• Lines of code (LOC) – While this one is heavily debated as to whether it’s the optical metric for project size, there’s no debating that it’s easy to calculate. Though imperfect, LOC is still indicative of project size. In this case, the simplicity trumps perfection.
• McCabe Cyclomatic Complexity – This metric measures the complexity of a function based on how many paths can be taken through the code. High complexity has been shown to be highly correlated with defect generation. Putting bounds on the acceptable complexity metric helps your team write understandable code. Plus, the cyclomatic complexity number gives you a lower bound on the number of test cases you need to test a function!
• Available Memory – Your compiler tool chain will spit this out for you. Keep an eye on how much memory you have as your project progresses. You need to know when you start running short. Project costs increase dramatically as you run out. Also, if you have a history of memory used during development, it’s often easy to go back and see where the big increases occurred. At Simplexity, we gather this information on our Jenkins continuous integration (CI) server. We also perform a compile time stack size analysis to guarantee that we won’t run out of memory at run time.
• Available CPU Bandwidth – The simple measure of this is CPU idle time in the main loop or idle task, depending on how your project is set up. At Simplexity we track this during development, but then double check schedulability with a custom tool we’ve built for performing Rate Monotonic Analysis to make sure our code doesn’t have any corner cases that can cause missed deadlines.

Start collecting those metrics! Like most engineers, I’ve found that as soon as I have a data set, I can’t help but start to analyze it. Interesting things always fall out once you have some data to play with.

Code Reuse

The Holy Grail of any code development effort is code reuse. Write it once, use it on all your projects! While it definitely can be done, it’s not a trivial endeavor. Jack made some of interesting points about reuse:

• If you must change more than 30% of the code, it’s not really code reuse. Sure, we can debate the number there, but it’s an interesting concept. If you’re required to change too much, you’re not reaping the benefits of reuse.
• Code needs to be reused three times before it’s reusable. Sounds a little counterintuitive, right? The point here is that when you attempt to reuse things, you’ll find you need to make changes to make it truly reusable. After about three attempts, you’ll probably get it right. I’ve got a chunk of servo code (motion profile validation, generation, and feedback control) that I’ve developed and reused several times. It was after attempting to reuse the third time that I finally figured out how the interfaces really needed to be structured to make it easy to leverage.
• A modular architecture is critical for reuse. I learned this one during my time at HP. The HP InkJet firmware architecture is component-based and has been reused on literally hundreds of projects over about 20 years. When Simplexity created our mechatronics product firmware code base, we adopted this architecture and have successfully delivered a range of products without touching most of the components.
• Use wrappers and abstraction layers to minimize dependencies. To support the ability to leverage your code to other processors, things like hardware abstraction layers (HAL’s) are critical to avoid rewriting all of the hardware peripheral access code. With a well-structured HAL, swapping processors is not a “bet the company” proposition.

Firmware Quality

Of course, everything discussed above all supports getting to high quality firmware. While everyone understands the concept of quality, the road to getting to a quality product is something that is often misunderstood. Clearly, to prove you have a quality product, you have to do a lot of testing. While that is necessary, it is not sufficient. A quality product can only be built if quality was designed in. Hard-won experience tells us that taking shortcuts in the design leads to longer, more expensive projects. If you don’t do the hard work to design in quality at the front, it’s going to take much longer to shoehorn it in at the end of the project through test/fix cycles that are notoriously difficult to estimate for cost and schedule.

“Coders” don’t build quality firmware. They jump in quick, code stuff up and show early prototypes that demonstrate some of the core system behaviors. Then they sit at “80% complete” for 110% of the schedule. Software engineers, following a disciplined process, are what is required to create quality firmware. The first code typically doesn’t come out as quickly when you follow a disciplined process, but when it does start to come out, you spend less time debugging, and your overall schedule is shorter. Some of the key aspects to a disciplined process:

Do design up front, before coding. Review your design with the team. The more people that will interact with your code, the more important this is to do. I’ve been in multiple design reviews where every person on the team had a different assumption about how the system should be designed. If that isn’t discovered early, there will be a considerable volume of code that gets thrown away.

Have an arsenal of tools to help you filter out defects and defect prone code structures from your source before you push it out to the rest of the team. Crank up all the warnings and errors on your compiler and treat warnings as errors. Use a static analyzer. Use McCabe cyclomatic complexity to look for error prone functions.

Use good software engineering practices to help generate well structured, readable code: have a coding standard, write short functions, decompose your code into modules, don’t use global variables, and eliminate side effects from functions. Most importantly: hold code reviews regularly to get more eyes on the code. Automated tools can only catch certain classes of errors, others require experienced engineers to find.

Finally, track metrics as described above. Metrics help you understand how well your process is performing. Your metrics will tell you when something is going wrong with your process and help guide you to what needs fixing.

Like any engineering discipline, it takes hard work and discipling to build robust, quality firmware for a product. Unlike mechanical and electrical engineering, embedded firmware engineering is a relatively new field. The processes required to engineer quality firmware on time and on budget are not taught to most embedded firmware engineers in school. Our industry is lucky to have people like Jack Ganssle who have been through the battles, earned the experience, and are willing to share it with others. If you get the chance, I highly recommend you and your team spend a day and take his class.