In this article, we are going to be explaining the very basic of embedded continuous integration, also known as CI, and look at how it can transform our workflows for faster and more reliable software delivery.
We are first going to define what continuous integration are in the traditional software development sense and afterwards, we will look at what makes embedded continuous integration different to those approaches. We will then look at the key tools for embedded continuous integration and then explain a typical use case we see for embedded teams. Finally, we will look at the challenges in adopting embedded continuous integration and the strategies we see in overcoming these challenges.
You can also watch the video version of the article below:
What is Continuous Integration (CI)?
Continuous integration is the practise of frequently merging developer code into a single central repository that is then automatically built and tested. This single central repository tends to be using a Git-based service such as GitLab or GitHub. These central repositories are version control systems that track the changes and handle merges.
Before the adoption of continuous integration, developers would have their own versions of source code that they would work on until a couple of months before release. The entire team would then attempt to merge code into a single source code for release. This inevitably would lead to ‘merge hell’ where code clashes and bugs arose as systems attempt to interact with each other. Projects get delayed by months to squash bugs and integrate properly.
Continuous integration emerged as a solution to this by continuously integrate the code throughout development, instead of all the engineers integrating at the end. This would mean that any integration bugs are caught earlier in the process where they are less likely to be damaging.
Once that code is centralized we can then also repeatably build and test that code to ensure that the code that is being introduced is not going to cause any clashes. When then form this positive feedback loop, where developers are constantly able to update their code, get feedback on it, and use that to inform the next update .
What is embedded continuous integration? The challenges
Okay so what makes embedded continuous integration unique to traditional continuous integration? The answer is that embedded systems development faces unique challenges that most software engineers never need to worry about. The first and most obvious is that you are dependent on physical hardware which makes automating the testing process far more difficult than software-only environment. You need to worry about managing external connections like USB or SSH. How are you updating these devices? Are you going to be updating at the hardware, firmware or software level?
Physical devices introduce challenges such as limited availability of prototypes. Building prototypes can be time consuming and costly so you want to be able to effectively manage the devices you do have.
Another one of these difficulties is that you are updating firmware, which means we need to ensure the boot process of the device works correctly. We need to be able to detect a system crash and be able to recover that device.
The next difficulty is limited resources. In software development, if you need to run your software faster, you can just rent a more powerful machine. With embedded devices you have constrained memory, processing power and battery life, which can require tailoring your test environment to.
Finally, in embedded continuous integration it is common to test the same software across multiple different platforms, meaning that for the same software you could need multiple hardware setups.
We have a lot of challenges here but the benefits of embedded continuous integration are worth overcoming these challenges.
What is embedded continuous integration? The benefits
The benefits of embedded continuous integration are the same three as ordinary continuous integration. Faster bug detection, reduced integration costs, and quicker development iterations for your device. But these advantages are so crucial for embedded products.
With software, if there is a bug, you can always just release an update, but that’s not always the case with embedded systems. 50% of customers won’t connect smart appliances, which means many users never update their products. They will just return a product as broken if the firmware or software does not work. On top of that if your embedded device controls mechanical systems, bugs could present a physical hazard to users. This makes fast and thorough bug detection is absolutely crucial.
Reducing your integration costs is important because you want developers to regularly perform system level tests because a bug in firmware can be an absolute nightmare to debug. Small and regular changes that can easily be reversed will make it far easier to isolate those nasty bugs.
Quicker development iterations is also a bigger deal than it is in the software world because in software development, there are people who get shocked if it takes more than half an hour to run through unit tests. In hardware and embedded development, we have talked to clients who have told us it takes them literally a week to go through all their manual testing tasks, whereas using continuous integration could reduce that down to half a day. There are huge time savings to be had.
Key Tools & Technologies for embedded continuous integration
So the there are three key tools that any CI system needs:
- A Version Control System and these tend to be Git based, so you have your GitHub, GitLab or BitBucket
- A continuous integration System like Jenkins, GitLab CI or our very own BeetleboxCI
- An isolated testing environment. You will need a software tools that allows your embedded software to be built and tested in an independent, isolated environment. This was done before using Virtual Machines but nowadays is mostly handled by containers like Docker
Let’s now look at how we can combine these tools into a typical setup that we see from our customers using BeetleboxCI.
Example Use Case
In a typical embedded continuous integration use case, you see typically a five-stage system and to look at that we will be imaging an embedded software developer has just finished with their latest changes to the code and now wants to run through the automated testing sequence. Let’s also say that they are using a custom version of embedded Linux to run on a device.
The first thing the developer needs to do is push their tool to the version control system. Once it is there their BeetleboxCI can gain access to the code. It can then detect the code changes and begin running.
In CI, the entire automated process from taking those code changes to testing, building and deployment is called a pipeline. A single pipeline will be split into multiple workflows and each workflow defines the specific jobs and processes involved in automating a task. For instance, in our example, we have a single workflow in our pipeline. This workflow then splits into several jobs and each job defines a specific task that is to be performed by the pipeline.
Our first task is to compile our application from our source code. Running the device using simulators or on device could be quite slow, especially if we first want to run some basic unit tests, so instead we can compile on the continuous integration server and run basic some basic tests.
If these steps pass, we then move onto testing on emulators. We can cross compile our device and then run it on an emulator, which means we can replicate running on our device in a cost-effective and scalable way.
The next stage is hardware in the loop testing. Hardware in the loop is where physical hardware is attached to BeetleboxCI so that tests can be performed on them. BeetleboxCI has unique features that allow devices to be easily attached, programmed and tested that makes setting these tests up far less painful. In this case, the device is reprogrammed with our embedded linux. We can then run boot tests and then run tests on device to ensure it is completely functional.
If all these tests pass, then we can finally package the system so that is then ready to be deployed. If errors have occurred, developers will be notified by BeetleboxCI and they can check logs and extract the data form the device. This can then provide that feedback needed to complete the continuous integration cycle that we saw earlier.
How to start
Much of the time people are reluctant to start using embedded continuous integration because they look at how complex the automated pipelines can be and they get scared about the months it will take to properly set up. The best way to start continuous integration is to go with the absolute smallest part that you can automate, so if you only have a singular module that might be running bare metal start there. Get your team on board by showing them the value in just that section and start slowly expanding from there.
Embedded continuous integration can be far more of a challenge to set up but it is ultimately worth it and hopefully the example we provided here can help inspire your own work.
