Automated evidence collection: build an audit evidence catalogue that collects itself

Fieldwork for your SOC 2 Type 2 starts in three weeks, and a prospect's security review is waiting on the report. The evidence is a shared folder of screenshots, half of them undated, and the access review for the second quarter is missing. Automated evidence collection fixes that before the next period starts, not the week before the auditor arrives.

This guide is for compliance leads, security engineers and CTOs at UK B2B tech firms facing a SOC 2 Type 2 period or an ISO 27001 surveillance audit. It sets out what auditors check in evidence, the evidence catalogue that comes first, how to collect, protect and package each item, and how one collection run serves every audit and customer review.

What is automated evidence collection?

Automated evidence collection is pulling proof that controls operate, such as configuration states, logs, tickets and reports, straight from source systems on a schedule. Each item lands in a timestamped, tamper-evident store mapped to the control it proves. Auditors prefer system-generated records to screenshots when completeness and integrity can be shown. The catalogue comes first.

The tooling is the easy part. The hard part is deciding, control by control, what proof an auditor will accept, where it lives and how often to take it. That decision is the evidence catalogue, and the rest of this guide builds on it.

Automation without a catalogue collects data. Automation with one collects evidence.

What do auditors check in audit evidence?

Auditors check that evidence is relevant to the control, reliable, complete for the population it describes and spread across the whole audit period. They prefer records produced by systems to copies made by people, and they want to know the records were not altered. They gather it by examining documents, interviewing staff and testing controls directly.

The UK standard on audit evidence, ISA (UK) 500, is written for financial audits. Its reliability tests are still a useful yardstick for any control audit. Paragraph 7 asks the auditor to consider the relevance and reliability of information used as evidence. Paragraph A35 adds that reliability is influenced by where the evidence comes from, what it is, and the controls over how it is made and kept.

Five checks follow from that, and each one shapes how you collect:

• Relevance. The item proves the control it is mapped to, not a neighbouring one. A list of user accounts is not proof of an access review.
• Completeness of the population. When you hand over a listing, such as every change merged in the period, the auditor needs to know nothing is missing. ISA (UK) 500 paragraph 9 asks the auditor to evaluate whether information the entity produces is reliable, including its accuracy and completeness.
• Period coverage. A SOC 2 Type 2 report covers operating effectiveness across a period, so evidence from one month cannot speak for the whole period.
• System-generated over screenshot. Under paragraph A35, an original document is more reliable than a copy or a digitised version, whose reliability may depend on the controls over its preparation. A screenshot is a copy.
• Integrity. The auditor wants to see who could change the record after it was created.

NIST SP 800-53A names the three ways an assessor gathers evidence: examine (review documents and records), interview (talk to the people running the control) and test (exercise the control and compare the result with the expected behaviour). Automated collection serves examine and supports test. It never replaces interview, because an auditor still wants to hear the control owner explain what they do.

What goes in an evidence catalogue?

An evidence catalogue lists, for each control, the evidence item that proves it, the source system, the collection method, the frequency, the retention period and how integrity is kept. It is one table, owned by one person, and it turns "gather the evidence" into a list of named jobs that run on a schedule.

We recommend these fields as a minimum:

Here are nine worked rows across six control areas. Treat them as illustrations to adapt, not a standard set:

Retention and integrity are usually the same for every row. Set them once for the store, not per item. Where an item needs longer retention, such as supplier contracts, note the exception on its row.

Which collection methods should you use for automated evidence collection?

Use the most direct method each evidence item allows, in this order: an API pull from the source system, a scheduled export, a ticket workflow that records the human step, and owner attestation only for what cannot be pulled. The higher the method sits, the less the record depends on memory or trust.

1. API pull. A scheduled script queries the source system and stores the result with a timestamp. Configuration states, user lists and change records suit this best. It is the closest thing to the original record.
2. Scheduled export. The system produces a report on a schedule and delivers it to the evidence store. Use it where no API exists, or where the system's own report is what the auditor expects to see.
3. Ticket workflow. For controls with a human decision, such as an access review or a restore test, the ticket is the evidence. It records who did what, when, and with what result. Build the template once so every instance captures the same fields.
4. Owner attestation. A dated, signed statement from the control owner, ideally referencing a record held elsewhere. Keep it for the few controls where nothing can be pulled, and expect the auditor to probe it at interview.

The method matters less than the schedule. A control that runs quarterly needs evidence every quarter, collected when the control runs, not reconstructed later.

How do you keep audit evidence intact and retained?

Keep audit evidence intact by stamping each item with its collection time, writing it to storage that is versioned or immutable, and limiting who can change or delete it. Keep it for at least the full audit period, plus enough time for the auditor's queries. Make integrity part of the store, not a promise from the team.

ISA (UK) 500 ties the reliability of internally generated evidence to the controls over its preparation and maintenance. In practice that means four things:

• Timestamps from the system, not the person. The collection job records when it ran. A file named "March final" proves nothing about March.
• Versioned or immutable storage. Object storage with versioning or a retention lock keeps every version, so a later change is visible rather than silent.
• Separated write access. The people whose controls are evidenced should not be able to edit the evidence. Service accounts write; humans read.
• A hash or checksum where it helps. For exported reports, a stored hash shows the file the auditor sees is the file the job produced.

On retention, we recommend keeping evidence for the current audit period and the one before it, so a returning auditor or a customer query can always be answered. In ISO 27001 terms, the evidence store holds documented information, so clause 7.5 documented information applies: retention and protection are decisions you make, write down and follow.

This is audit evidence, not forensic evidence. Collecting evidence of an incident for disciplinary or legal action is a separate discipline, covered in our guide to ISO 27001 Annex A 5.28.

How do you collect once and map evidence to many frameworks?

Collect once by tying every evidence item to your own internal control, then mapping that control to each framework requirement it satisfies. The evidence never knows which framework it serves. When a new framework arrives, you add mappings to existing controls rather than new collection jobs, and one run serves ISO 27001, SOC 2 and every customer review.

The method has three layers:

1. Evidence items are collected against internal control IDs, as in the catalogue above.
2. Internal controls describe what your organisation does, in your own words: "access to production is reviewed quarterly by the system owner".
3. Framework mappings link each internal control to the requirements it meets, for example an ISO 27001 Annex A control and the relevant SOC 2 Trust Services Criteria.

Build the mapping layer yourself, from the frameworks you hold. Published crosswalks can inform it, but many carry licence terms. And your auditor tests your controls against the framework, not against someone else's map.

If you want the catalogue and mappings in a machine-readable form, NIST's Open Security Controls Assessment Language (OSCAL) provides open formats in XML, JSON and YAML for control and assessment information. It is optional. A well-kept table does the same job at this scale.

How do you package evidence for the auditor?

Package evidence for the auditor by period, population and request. State the period each item covers, give full populations with the query that produced them, and let the auditor pick their own samples. Then answer each sample request from the store, with the original record and its collection timestamp, rather than gathering it afresh.

Sampling is where automated collection pays off. ISA (UK) 530, the UK standard on audit sampling, requires the auditor to select items so that every sampling unit in the population has a chance of selection. Its guidance adds that the auditor obtains evidence that the population is complete. Both depend on a population you can produce on demand.

We recommend a short index for each control, with four parts:

• the control ID, its owner and the framework requirements it maps to;
• the period covered and the collection frequency;
• the population listing and how it was produced;
• where each sample item sits in the store.

The same pack serves your own assurance. Under ISO 27001 clause 9.2, your internal auditor can work from the same index, so the certification body sees evidence that has already been tested once.

How does the same evidence answer customer security questionnaires?

The same evidence answers customer security questionnaires because questionnaires ask, in different words, whether the controls your auditors test are operating. Point each common question at the catalogue row that proves it. You answer with a dated, current record instead of a fresh promise, and the security lead stops rebuilding answers for every deal.

In practice:

• Map common questions to controls once. Most questionnaires ask about access, encryption, backups, vulnerability handling, suppliers and staff training. Each maps to rows you already collect.
• Share summaries, not raw logs. A customer wants to know the access review happened and who signed it, not a full user export. Summarise from the store and offer the underlying record under NDA if asked.
• Quote the date. "Last access review completed 30 June, next due 30 September" is stronger than "we review access regularly", and it is only possible when collection runs on a schedule.

What are the common mistakes with automated evidence collection?

The common mistakes with automated evidence collection are keeping screenshots without dates or scope, collecting evidence that misses part of the audit period, collecting data that no control or auditor tests, and storing evidence where the people being audited can edit it. Each one makes evidence look complete while leaving the auditor unable to rely on it.

• Screenshots without date or scope. A screenshot of a settings page with no date, system name or account shown proves nothing about when or where. If a screenshot is unavoidable, capture the date, the system and the scope in the image.
• Gaps in the period. A job that failed silently for six weeks leaves a hole the auditor will find. Monitor the collection jobs themselves, and treat a failed run as a control issue.
• Collecting what nobody tests. Pulling every log a system produces fills the store without proving anything. Collect what the catalogue says each control needs.
• Editable evidence. If a control owner can overwrite their own evidence, the auditor has to trust them, which defeats the point of automating.
• Automating the pull but not the review. An access list collected quarterly is not an access review. The human decision still needs its own record.

The catalogue prevents most of these, because each one is a row that was never written down.

Frequently asked questions