Home

Welcome to the COSA wiki!

COSA is a research project to explore the viability of performing security compliance in a DevOps lifecycle when many activities cannot easily be automated.

Risk-based Control Compliance - a short review

There is a historical tendency in DevOps to think only of the automatable path and to strive for 100% automation. But what if you cannot? COSA acknowledges that not every control can be fully automated but that making consistent and measurable progress towards 100% compliance is a worthy goal.

Consider a hypothetical security control that states "Access to the physical computers must be controlled with 24x7 uniformed security guards by each entry point to the data center". This is completely made-up but consider what it takes for an auditor to assert that the control is being met.

First, you need a map of the facility that shows entry points. Counting up the entry points, the auditor sees that there is only one. Therefore, there must be one guard at that entry point. Upon accessing the facility, the auditor goes to the entry point to see if a guard is posted. They may return at night to ensure that there is still a guard posted. They may examine security camera footage, if it covers the guard post. They may check the security guard badge has been swiped at the beginning of the shift. They could check the camera live to see the guard is in the frame. If the guard has an Apple airtag, they could verify that the airtag is located where expected. These are all hypothetical solutions.

Each of these is called a "Test" in COSA. So, a Control has one or more Tests. Each Test provides an additional degree of certainty that the control is being met. For the previous example, having 8 Tests gives the auditor greater confidence that the control is being met than if they simply check the employee badge swipe log. How many Tests should be used? An infinite number of Tests would provide 100% confidence that the control is being met. But is that realistic and is it a good idea? Probably not. Control compliance is a risk-based activity. That is, no amount of testing provides 100% confidence. However, you can reach enough confidence to cover the risk of non-compliance. That, of course, is a policy decision by the organization. But it is a reasonable one.

Now, consider a Catalog of one or more Controls where each Control has one or more Tests. That's the basis for compliance testing in COSA.

Each time a test is executed (whether it is by a human or by a computer), it has a limited lifetime. That is, the result has an expiration date. That's because we often need to re-test to ensure that the current condition is still secure. But Controls are often written such that the re-testing period need not be continuous, because the level of risk and the cost of compliance allows for a lower frequency. So, for the example above, we may retest only annually. Or quarterly. Or Monthly. Also, the expiration is at the Test level, not the Control level. So, some tests are executed very frequently while others are not. Automated tests are typically executed nearly continuously, because the cost of testing is so low (in most cases). However, if an automated test is expensive (for example, uses a Satellite data link or perhaps ties up an expensive resource), we may set an expiration date that balances the need for testing with the need for security. Again, the risk-based assessment mindset prevails.