Control testing: design vs operating effectiveness, and how to automate it

Your first SOC 2 Type 2 period opens next quarter, and the enterprise deal behind it depends on a clean report. The auditor will test your controls. Nobody on your team has tested them first, and you have no internal audit function to ask. Control testing is how you find the failures before they become exceptions in a report your customer reads.

This guide is for compliance leads and heads of security at UK B2B tech firms of 25 to 250 people facing a first SOC 2 Type 2 or an ISO 27001 surveillance audit. It covers the two tests every control needs (design and operating effectiveness), the four test methods, sampling, the test record, exceptions, and which tests can run automatically.

What is control testing?

Control testing checks two things about a security control. A test of design asks whether the control, as written, would address its risk. A test of operating effectiveness asks whether it worked consistently across a period. Testers use four methods: inquiry, observation, inspection and re-performance. For security controls, much operating-effectiveness testing can be automated.

The discipline comes from audit. The UK auditing standard ISA (UK) 330, written for financial audits, defines a test of controls as a procedure "designed to evaluate the operating effectiveness of controls". The same logic carries straight across to access reviews, backups and change approvals. Run all year rather than once before the auditor arrives, it becomes part of GRC engineering.

Control testing is not substantive testing. A substantive test looks for errors in the output itself; a control test checks whether the safeguard that should prevent those errors is working.

What is the difference between a test of design and a test of operating effectiveness?

A test of design asks whether a control could work: is it capable of meeting its objective, and is it in place? A test of operating effectiveness asks whether it did work, consistently, across a defined period. Design comes first, because there is no point testing the operation of a control that could never meet its objective.

ISA (UK) 315 makes that order explicit. Evaluating design means considering whether a control "is capable of effectively preventing, or detecting and correcting" the problem it targets. The standard adds that "the auditor evaluates the design of a control first".

A well-designed control can still fail in operation. A quarterly access review that is perfectly written but run twice in a year passes the first test and fails the second.

How do you test whether a control is well designed?

Test design by tracing the control from the risk it addresses to the record it leaves. Confirm the control has a named owner, a defined frequency, a clear trigger and an output a tester can inspect. Then walk through one real occurrence end to end. If any link is missing, the control fails design, however well it runs.

Five questions settle most design tests:

1. Which risk does it address? Name the risk register entry or the SOC 2 criterion it supports.
2. Who performs it? A named role, not "the team".
3. When does it run? A frequency or an event, such as each leaver or each production release.
4. What would count as a failure? The control needs a condition someone can check.
5. What record does it leave? A ticket, log entry, approval or report with a date.

A control with no record is a control nobody can test.

The walkthrough matters because documents describe intent. ISA (UK) 315 lists inquiry, observation and inspection as ways to check design and implementation, then adds: "Inquiry alone, however, is not sufficient for such purposes."

How do you test operating effectiveness?

Test operating effectiveness by selecting occurrences of the control from across the whole period and checking each one ran as designed: on time, completely and by the right person. ISA (UK) 330 frames the three questions as how the control was applied, the consistency of its application, and by whom or by what means.

For a monthly vulnerability scan review, that means pulling scan reports from across the period, checking each was reviewed within the agreed window, and confirming the reviewer was the named owner. A review signed off by someone without the authority to accept the risk is a deviation, even if the date is right.

The period matters as much as the sample. A SOC 2 Type 2 report, as the Journal of Accountancy's guide to SOC 2 explains, includes "a description of the service auditor's tests of operating effectiveness and the results of those tests". Your customer reads those results, so every occurrence inside the window counts.

What are the four control testing methods?

The four control testing methods are inquiry (asking the owner), observation (watching the control run), inspection (examining records it leaves) and re-performance (running the control again to see if you get the same result). They give increasing assurance in roughly that order. Inquiry alone never proves a control works, so every test combines it with at least one other method.

ISA (UK) 330 explains the ranking. Inquiry combined with inspection or re-performance "may provide more assurance than inquiry and observation, since an observation is pertinent only at the point in time at which it is made."

Security teams who work to US federal guidance will meet the same ideas under different names. NIST SP 800-53A Rev. 5 uses three methods: examine, interview and test. Its test method exercises a control "under specified conditions to compare the actual state of the object to the desired state", which is re-performance by another name.

How many samples does control testing need?

Control testing samples by control frequency: the more often a control runs, the more occurrences a tester selects. Your auditor applies its own sampling methodology, so ask for it during planning and test to the same standard. Automated controls need fewer samples, provided changes to the system behind them are themselves controlled.

ISA (UK) 330 lists the frequency of the control and the expected rate of deviation among the factors that set the extent of testing. The Financial Reporting Council's 2023 thematic review of audit sampling shows what that looks like in practice. It compared the sample sizes seven major UK audit firms set for testing controls in financial audits:

Treat those figures as a reference point, not a rule. A SOC 2 service auditor sets its own sample sizes, and an ISO 27001 internal audit programme decides its own depth of testing.

Two principles hold whatever the numbers:

• Define the population first. ISA (UK) 530 defines it as "the entire set of data from which a sample is selected". A sample drawn from an incomplete list proves nothing about the controls that were left off it.
• Sample across the whole period. Ten occurrences from the last fortnight say nothing about the first nine months.

Automated controls are the exception to the sample-size rule. ISA (UK) 330 notes that an automated control "can be expected to function consistently unless the IT application ... is changed". Test it once, then rely on your change management controls to show it has not been altered.

What evidence holds up in control testing?

Evidence holds up when it is complete for the population, accurate, dated inside the period and drawn from the system of record rather than recreated by hand. A screenshot taken the week before fieldwork shows the state today, not the state across the period. Records generated by the system that runs the control are the strongest evidence.

Four tests apply to every piece of evidence you attach:

• Complete: it covers every item in the population, or the full sample.
• Accurate: it matches the source system when re-checked.
• Dated: each record shows when the control ran, inside the period.
• From the system of record: exported from the identity provider, ticketing tool or cloud console, not typed into a spreadsheet.

Building that evidence so it collects itself is a separate discipline. Our guide to automated evidence collection covers evidence catalogues, collection methods and retention.

What goes in a control test record?

A control test record captures enough detail for someone else to repeat the test and reach the same result. It names the control, the test steps, the population and sample, the evidence examined, the result and any exception. Without a record, a test is an opinion; with one, it becomes evidence for your internal audit and your auditor.

We recommend one record per control per test period. Keep it in the same place as your ISO 27001 internal audit records, so one set of working papers serves both frameworks.

How do you handle exceptions found in control testing?

Handle a control testing exception in four steps: confirm it is a genuine deviation, find the root cause, fix both the instance and the cause, then retest. An exception is not a failure of the testing programme. It is the testing programme working, provided the cause is fixed before the auditor samples the same control.

ISA (UK) 330 recognises that some deviations will occur, caused by "changes in key personnel, significant seasonal fluctuations in volume of transactions and human error". What matters is whether the rate of deviation shows the control cannot be relied on.

1. Confirm the deviation. Check the evidence again. A missing record can mean the control failed or that the export was incomplete.
2. Find the root cause. A leaver's account left active for three weeks might trace to HR telling IT by email instead of through a ticket.
3. Remediate. Fix the instance (disable the account) and the cause (route leavers through the ticketing tool).
4. Retest. Select new occurrences after the fix and confirm the control now operates.

An exception found in an ISO 27001 internal audit that shows a requirement was not met is a nonconformity. Clause 10.2 on nonconformity and corrective action then asks you to evaluate its causes, act on them and review whether the action worked.

An exception you find yourself is a problem you can fix. One the auditor finds is a line in a report your customer reads.

How does control testing feed the ISO 27001 internal audit and a SOC 2 Type 2?

Control testing supplies the evidence both frameworks rely on. For ISO 27001, test records are evidence for the clause 9.2 internal audit, which must establish whether the ISMS is effectively implemented and maintained. For SOC 2 Type 2, testing inside the period shows management whether controls will survive the service auditor's tests before the report is written.

The two uses differ in who tests. Under ISO 27001, your internal auditor must be objective and impartial, so at a firm with no audit function the tester should come from outside the control's own team. Our guide to the ISO 27001 clause 9.2 internal audit covers the programme, independence and reporting.

Under SOC 2, management's own testing is not a substitute for the service auditor's. It gives management a basis for the assertion it signs which, as the Journal of Accountancy guide describes, states for a Type 2 that controls were operating effectively throughout the period.

We recommend one test calendar that serves both. Each control is tested once per cycle, and the same record feeds the internal audit report and the SOC 2 readiness check.

Which control tests can be automated?

Automated control testing suits controls whose evidence is machine-readable and whose pass condition is objective: configuration, access, logging and backups. Controls that rely on judgement stay manual, including governance, training effectiveness and physical security. The realistic answer is a mix, and the automated share grows as more controls produce system records.

NIST SP 800-137 puts it plainly: organisation-wide monitoring "cannot be efficiently achieved through manual processes alone or through automated processes alone".

• Good candidates: MFA coverage, leaver deprovisioning, encryption at rest, logging enabled, backup success and restore tests.
• Stay manual: whether a risk assessment is sound, whether training changed behaviour, whether a visitor log matches the door.

Design effectiveness stays a human judgement in every case. An automated test proves a control ran; only a person can judge it was the right control.

For the method, see our guides to continuous control monitoring, which covers test cards, frequencies and failure handling, and compliance as code, which covers writing checks auditors accept.

What are the common mistakes in control testing?

The most common control testing mistakes are relying on inquiry, testing only the last few weeks of the period, drawing samples from an incomplete population and closing exceptions without retesting. Each one produces a clean-looking test record that would not survive an auditor re-performing it.

• Accepting "yes, we do that". Inquiry is where testing starts, not where it ends.
• Testing operation before design. A control that could never meet its objective will pass every operating test and still leave the risk open.
• Sampling from what was easy to find. If the population came from the same team that runs the control, check it against an independent source.
• Testing your own control. The person who performs a control is the least likely to see its gaps.
• Fixing the instance, not the cause. The account gets disabled, the email route stays, and the same exception reappears next quarter.

A control nobody has tested is a hope with a policy number.

Frequently asked questions