Compliance as code: turning controls into automated checks auditors accept

Your engineers wrote a script that checks every cloud account for multi-factor authentication each night. It works. Then the auditor asks three questions: which control does it test, who approved its logic, and can you prove it ran on every day of the report period? Nobody has a clean answer, and the enterprise renewal is waiting on the SOC 2 report. That gap, between a script that works and evidence an auditor accepts, is what compliance as code has to close.

This guide is for heads of compliance, GRC leads and CTOs at UK B2B tech firms who already run automated checks and now have to defend them. It shows how compliance as code turns a control into a check an auditor can rely on: the anatomy of a check, how auditors test system-generated evidence, what to codify first, and a worked example.

What is compliance as code?

Compliance as code is the practice of expressing control requirements as automated, version-controlled checks that test your systems on a schedule and record each result as evidence. Policy as code blocks bad changes before they happen; compliance as code detects drift and proves the control held. The checks only count when each is mapped, reviewed and complete.

The two ideas share tooling, which is why they get confused. A policy engine in a deployment pipeline stops a change that breaks a rule. A compliance check looks at the live estate, compares it with the control requirement and writes down what it found.

Most firms need both. This page covers the detective side and the evidence it produces.

The term also gets used for writing policies in Markdown and storing them in Git. That is good document control, and it is not what an auditor means by a coded check.

What does a compliance as code check need to contain?

A compliance as code check needs eight parts: the control objective, the framework reference by ID, the data source, the test logic, the pass and fail criteria, the frequency, a named owner and the evidence record it writes. Missing any one of them leaves a question the auditor will ask, and the script cannot answer.

We recommend writing these parts down as a short specification that sits beside the code, in the same repository:

The specification is what the auditor reads first. The code is what they test against it.

A check with no control objective is not a control. It is a script with an opinion.

How do auditors decide whether automated evidence counts?

Auditors treat a check's output as information produced by the entity, so they test whether it is complete and accurate before relying on it. They then test the check's design and whether it operated across the whole period. A check that passes all three tests covers the whole population, where a screenshot sample covers a handful of items.

The UK auditing standards, written for financial statement audits, set out the logic plainly. ISA (UK) 500, paragraph 9, requires the auditor to evaluate whether information produced by the entity is sufficiently reliable, including evidence about its accuracy and completeness. Paragraph A61 adds that the auditor may get that evidence by testing the controls over how the information is prepared and maintained.

For a coded check, that means three things the auditor will look for:

1. Completeness of the population. The check read every account, user or resource in scope, not only the ones it could reach. A count of what was tested, reconciled to an independent list, answers this.
2. Accuracy of the logic. The test does what its specification says. The auditor may re-run it, read the code or test it against a known failure.
3. Integrity of the output. The result record could not have been edited after it was written.

Design and operation are separate tests. ISA (UK) 330, paragraph 11, requires the auditor to test controls throughout the period they intend to rely on them. The same idea sits behind a SOC 2 Type 2 report, which covers how controls operated over a period rather than at one date. ISA (UK) 402 defines a type 2 service organisation report in the same terms.

Automation helps here. ISA (UK) 330 paragraph A29 notes that an automated control can be expected to function consistently unless the application is changed. The auditor may then test that it continues to work, including the general IT controls around it. That shifts the weight onto the next question: who can change the check?

Why is the compliance check a control in its own right?

The compliance check is a control in its own right because the evidence it produces is only as trustworthy as the process that changes it. If anyone can edit the logic, switch it off or delete a failed result without review, the auditor cannot rely on the output, however good the code looks.

ISA (UK) 330 paragraph A36 says an auditor may check whether an automated control has changed by inspecting logs that show what was changed. Your repository history is that log, if you run it properly.

We recommend treating the check repository like production code under ISO 27001 Annex A 8.32 change management:

• Peer review. Every change to test logic or pass criteria goes through a reviewed pull request, approved by someone other than the author.
• Restricted access. Few people can merge, and the people whose systems are being tested cannot change the checks that test them.
• Traceable releases. Each result record names the version of the check that produced it.
• Protected history. Results and run logs are kept in storage that the check's own operators cannot overwrite.

The same discipline covers the SOC 2 change management criteria in the CC8 series. One change process can serve both frameworks.

A coded control that anyone can quietly edit gives no more assurance than a spreadsheet.

How should you handle exceptions and false positives?

Handle exceptions and false positives through a recorded process, not by editing the check. A genuine exception needs an owner, a reason, a compensating control and an expiry date. A false positive is a defect in the check, so the fix goes through the same reviewed change as any other logic change, with the fail result kept.

Most coded checks will flag something you decide to accept: a service account that cannot hold a second factor, or a legacy system with a retirement date. We recommend an exception register that records:

• the check and the resource it excepts;
• why the exception exists, and who approved it;
• the compensating control, such as network restriction or extra logging;
• the expiry date, after which the check fails again.

The check should read the register and report excepted items as a separate result, not as passes. The auditor then sees every exception, who approved it and whether it lapsed.

False positives are different. If the check fails something that meets the control, correct the logic, and keep the original fail alongside a note that links to the fix.

An exception with no expiry date is a policy change that nobody approved.

Which controls should you codify first, and which stay manual?

Codify the controls that test system state first: configuration, access, encryption, logging and patching. Their requirements are objective, their data sits in systems you can query, and they change often enough for a point-in-time sample to miss drift. Controls that rest on human judgement stay manual, with automation supporting the record.

Judgement controls stay manual because the auditor is testing the quality of a decision, not the state of a system. A script can prove a management review meeting was held. It cannot prove the meeting weighed the right risks.

Your own reviews of whether you meet your policies, under ISO 27001 Annex A 5.36, sit across both groups. Coded results feed the review, and a person signs it.

What format should compliance as code evidence take?

Compliance as code evidence should be a structured, dated record that names the control, the check version, the population tested, the result and any exceptions, kept unaltered for the whole report period. NIST's OSCAL is the open standard for that record, so your results can move between tools and assessors without rework.

OSCAL, the Open Security Controls Assessment Language, is a NIST-led initiative with machine-readable formats in XML, JSON and YAML. It is organised in three layers:

• Control layer: catalogues of controls and profiles that select and tailor them.
• Implementation layer: system security plans and component definitions that say how a system meets the controls.
• Assessment layer: assessment plans, assessment results, and plans of action and milestones.

The OSCAL assessment results model is the one that matters for coded checks. It holds observations with their evidence, findings that can show each control objective's status, and a log of assessment actions with start and end timestamps. NIST says it also supports reporting continuous assessment information.

You do not have to adopt OSCAL on day one. We recommend at least matching its shape: one record per run, with the control reference, the timestamps, the observation and the finding. Converting later is then a mapping exercise.

Retention follows the audit, not the tool. We recommend keeping every result record and run log for the full report period, plus enough time for the auditor to finish, and a gap in that run history needs an explanation on file. The idea of ongoing control monitoring is set out in NIST SP 800-137, NIST's guide to information security continuous monitoring.

What does compliance as code look like for one control?

Compliance as code for one control runs from the requirement to the check to a stored evidence record. Taking MFA enforcement as the example: the requirement comes from ISO 27001 Annex A 8.5 and the SOC 2 CC6 series, the check tests every in-scope user daily, and each run writes a record the auditor can sample.

Step 1: the requirement. Your access control policy says all staff signing in to production systems use multi-factor authentication. That supports ISO 27001 Annex A 8.5 secure authentication and the logical access criteria in the CC6 series of the SOC 2 Trust Services Criteria.

Step 2: the specification. The eight parts from the table above, written down and approved by the control owner before any code is written.

Step 3: the check. A small job reads the identity provider, lists every active user in the production access group, and confirms each one has a registered second factor and an enforcement rule. It counts the population, reads the exception register and writes the result.

Step 4: the evidence record. Each run stores one record:

These values are illustrative, not drawn from a real system.

Step 5: what the auditor tests. They reconcile the population count to the directory, read the specification against the code, re-run the check or test it with a known failing account, and sample records across the period. Then they check the repository history for changes to the logic.

What are the common mistakes with compliance as code?

The common mistakes with compliance as code are checks with no control mapping, scripts that change without review, gaps in the run history and dashboards that nobody signs off. Each one leaves the auditor with output they cannot rely on, so the team falls back to screenshots in the middle of fieldwork.

• No control mapping. A check that tests something useful but maps to no control objective produces data, not evidence.
• Unreviewed scripts. One engineer wrote it, nobody else has read it, and the logic changed twice during the period.
• Gaps in the run history. The job failed silently for three weeks and nobody noticed. Under ISA (UK) 330 paragraph 11, a period you cannot evidence is a period the auditor cannot rely on.
• Population not reconciled. The check passed every account it could reach, and nobody asked how many it could not.
• Dashboards nobody signs off. A live dashboard shows the state today. The auditor needs the record for each day, and evidence that someone acted on the fails.

A green dashboard with gaps in its history proves only that it was green when someone looked.

Frequently asked questions