Menu
The published record
Why 100% MC/DC coverage costs so much
None of the following is our data. It is the published record from NASA and the FAA on what modified condition/decision coverage costs the people who have to achieve it.
The requirement
MC/DC is the standard
Structural coverage is not optional for the safety-critical parts of a code base.
100 percent code test coverage using the Modified Condition/Decision Coverage (MC/DC) criterion for all identified safety-critical software components.
The difficulty
Practitioners find it hard and expensive
The NASA and FAA Streamlining Software Aspects of Certification survey reached more than 400 people and received around 300 responses.
78.5%
of experienced Level A respondents rated MC/DC moderately or extremely difficult
74.5%
rated its cost substantial or nearly prohibitive
| Finding | Respondents |
|---|---|
| MC/DC was moderately difficult | 55.5% |
| MC/DC was extremely difficult | 23% |
| Combined moderately or extremely difficult | 78.5% |
| MC/DC cost was substantial | 67.5% |
| MC/DC cost was nearly prohibitive | 7% |
| Combined substantial or nearly prohibitive cost | 74.5% |
| Would spend substantially less if it were not required | 61% |
| Would spend nothing on it if it were not required | 10% |
The complexity curve
Complexity punishes manual closure
NASA NESC data shows test effort rising sharply as structural complexity rises.
31 hours
to test a single complexity-28 function
30%
estimated developer confidence that coverage was complete
Data table
| Cyclomatic complexity | Hours to test |
|---|---|
| 5 | 0.75 hrs |
| 9 | 1.65 hrs |
| 28 | 31 hrs |
The honest part
In practice, 100% is not always reached by unit tests
In practicality, all possible MC/DC tests may not be achievable… conditions/decisions not achievable through unit testing should be listed and waived.
Some conditions genuinely cannot be reached through unit testing, and no tool changes that. What changes is who finds them and when. Today that list emerges late, from engineers investigating coverage gaps one at a time. COVE produces it automatically and completely, as gap analysis, at the point the tests are generated — so the dispositions you still have to write are known early and are the only ones left.
The cost
The coverage-closure loop
When coverage gaps remain, this is the cycle that consumes schedule and certification attention.
- Requirements-based tests
- Measure MC/DC
- Coverage gaps
- Engineering investigation
- Supplement or justify
- Waiver or disposition
What changes
What COVE changes
COVE takes the coverage-closure loop off the critical path. The tests, the gap analysis and the coverage record are produced together, early, instead of being assembled by a dedicated team at the end.
Sources
References
- NASA NPR 7150.2D, Chapter 3, SWE-219 100% MC/DC for identified safety-critical components.
- NASA/TM-1999-209519 Hayhurst et al., Report on the Streamlining Software Aspects of Certification Survey.
- NASA Software Engineering Handbook, Multi-Logic Testing Guidance on conditions not achievable through unit testing.
- NASA/TM-2001-210876 Hayhurst et al., A Practical Tutorial on Modified Condition/Decision Coverage.
- NASA/TM-20205011566 Squire et al., Cyclomatic Complexity and Basis Path Testing Study.
- AUTOSAC avionics study; SmartUnit, ICSE-SEIP 2018 Manual test burden and automated test throughput.