New-generation embedded test tools enable software developers to be confident of delivering high-quality software, making a significant contribution to new products that work first time
Magnus Unemyr and Mark Moran
Saturday, June 29, 2013: Releasing a product with bugs is potentially very expensive, especially considering the cost of field ugrades, recalls and repairs. Less quantifiable, but equally important is the damage done to the reputation and consequent loss of customer-goodwill by any company. Notwithstanding, many embedded products are indeed released without all the necessary and desirable testing. Consequently, software quality is receiving increasing focus in embedded application development.
Most existing software testing tools can only execute the tests on a PC, the use of which is limited when testing an embedded application. The actual system has numerous differences including hardware interfaces, timing issues and memory constraints among others. Therefore it is important to run as many software tests as possible on the actual embedded hardware to minimise inter-platform differences.
Unit testing
A key element of the software testing process is ‘unit testing.’ Unit tests call up ‘C’ programing functions with different combinations of input parameter values to drive the code through different execution paths. Other than for trivial functions, writing unit tests by hand is labour-intensive, costly, tedious, and might leave many possible execution paths untested. A detailed analysis of how input parameters drive the code is therefore required, so that important execution paths throughout a function are actually exercised by a test suite.
Another challenge faced by development teams is maintaining the synchronisation of unit tests with code under development. This problem is often exacerbated when delivery schedules become compressed with project progression. If source code and unit tests become out-of-sync, the tests will become less useful, especially when these are needed the most.
The challenge is to findautomated in-target software testing tools that can be easily integrated into developers’ embedded development environments. These tools should be able to analyse the application source code, autogenerate a test suite and automatically run it on the target board. However, a test report showing 100 per cent success may not be sufficientas the test procedure itself may not have been applied to more than a small fraction of the code. Understanding the quality of test procedures thus becomes critical in judging whether a product is ready for release.
However, new and advanced tools for embedded test automation and test quality measurement are now emerging.
The unit test tools available to PC developers are less useful to embedded developers as these rarely manage compilation, downloading and execution of the test suites on embedded boards. Tools that create the unit tests automatically, build these into the embedded application and allow to run on the target via a convenient pathway, such as a joint test action group (JTAG) debugger, result in optimal usage of time and maximum productivity.
An effective approach is to use a fully-embedded test automation system that is integrated right into the C/C++ integrated development environment. Thus synchronisation is easy to maintain and ancillary tools such as JTAG debug probes can be used to full effect. Such tools were not common previously, but new tools now meet these demanding criteria and bring powerful in-target test automation capabilities to software developers.
Unit test automatic generation. The new breed of professional embedded test automation tools can analyse the source code and generate and execute suitable test suites automatically on the target board. For example, these auto-generate a test suite and make many function calls with different combinations of input parameters, thus driving a large number of different execution paths.
Unit test automatic execution. Once the test suite has been generated, it must be compiled, linked and executed on the target system. Many low-cost unit test tools exist that run unit tests only on a Windows PC. The inability of these tools to integrate with the embedded tool development environment results in additional inefficiency to execute the tests on the target board. However, the new breed of professional embedded test automation tools integrate into a professional embedded IDE, enabling easy synchronisation of test with code development. These also take advantage of integration with tools such as JTAG debug probe.
Measuring test quality
Once a code has been developed and tested, the focus shifts to quantitatively understanding what actually happened during testing. Code-coverage measurement, which is performed using dynamic execution flow analysis, i commonly used to study which parts of the code have been tested.
There are many different types of code-coverage analysis—from the very simple to very stringent types. Some types are:
Statement or block coverage. This measures how many of the C-state-ments or code blocks have been executed during a test session. It does not measure how branches affect which C-statements or code blocks are executed.
Function coverage. This only measures which or how many of the C-functions have been called during a test session. It does not measure which or how many of the function calls in a code section are actually executed, or the quality of the testing of the function itself.
Function call coverage. This measures which or how many of the function calls in a code section have actually been made during a test session.
Branch coverage. This measures whether all code blocks and alternate branch paths have been executed in a code section (such as both the ‘if’ and ‘else’ part in an if-else statement). Branch coverage typically requires a code section to be executed a sufficien number of times, so that all alternative branch directions are tested. As all branch paths must be executed, all corresponding code blocks are executed as well.
Modified condition/decision coveage. This is a very advanced type of code-coverage analysis, which is often used for measuring the test quality of safety-critical software—generally in flight-controland aerospace applications. However, many other projects can also benefit from this advanced test analysis.
This kind of code-coverage is applied to applications through which the highest reliability is expected. It extends branch coverage with the additional requirement that all sub-expressions in complex decisions (such as in a complex if-statement) must drive the branch decision independently of the other sub-expressions.
Until recently, there have been few test-quality measurement tools available. These are limited by one or more of the following problems:
1. Testing using weak types of code-coverage analysis only
2. Testing in PC environments and not on the embedded target
3. Very expensive
4. Difficult to us
5. Lack of integration with other embedded development tools
New tools are highly integrated supporting a wide range of code-coverage analysis types. One of their most powerful features is that the analysis is conducted on the embedded target. Connecting to the target via a JTAG probe, the embedded application is run directly on the target board with the code-coverage information recorded for later inspection and quantitative analysis.
As an example of these tools’ capabilities, Fig. 2 shows a trivial code section that contains three blocks—a red code block, which is always executed; a green block, sometimes executed depending on the branch decision made in the if-statement; and a blue block, which again is always executed.
The branch decision in the if-statement drives the selection of one out of the two possible execution paths—either directly from the red to the blue code block; or passing from red to blue via the green code block.
Using only a few mouse-clicks, the easy-to-use in-target analyser tools can detect all the different types of code coverage including statement/block coverage, function coverage, function call coverage, branch coverage and modified condition/decision coverage.
Both the verification and analysis tools are integrated in the integrated development environment. It’s this new generation of advanced embedded test tools that will truly enable software developers to be confident of delivering high-quality software, making a significant contribution to new products that work first time.
The authors are from Atollic—a provider of ad-vanced development tools and expert services within the field of embedded systems and technical PC applications.
Article shared by Embedded Systems Solutions, Bengaluru—Atollic’s authorised distributor in India
