Do Not Wait for Perfect Data Before You Start

Use assessment AI first, then improve the information that actually matters

About this article
This article presents practical observations from GFLOPS as the provider of AskDona Batch Assessment. It is intended for knowledge sharing and does not describe any specific customer, organization, system, external service, assessment result, or operating environment. The vulnerability and system-risk assessment examples are fictional and have been generalized from issues that can arise across many organizations. In this article, “vulnerability assessment” is used broadly to include document- and evidence-based reviews of weaknesses and control implementation, not only automated vulnerability scanning or penetration testing.

Conversations about assessment AI often begin with data quality.

Organizations are told to remove obsolete documents, eliminate duplicates, standardize terminology, rebuild inventories, add metadata, and create better evidence. All of these activities can be valuable.

But if every improvement becomes a prerequisite, the AI initiative turns into another large data-preparation program. Teams that expected to reuse existing documents are instead asked to redesign the entire information environment before they can evaluate anything.

From our experience providing Batch Assessment, the order should usually be reversed.

Do not wait for perfect data. Run an assessment with the checklists and evidence you already have. Then use incorrect results, indeterminate outcomes, and missing-information outputs to decide what is actually worth improving.

Data improvement should not be a one-time preparation project completed before assessment AI can be used. It should be an iterative process guided by real assessment results.

AskDona Batch Assessment compares existing Excel-based assessment items with policies, design documents, specifications, inventories, evidence records, and prior responses stored in RAG knowledge. It organizes a first-pass result, rationale, quotations, sources, and missing information for each item. A human reviewer then confirms the evidence, adds context, corrects the result where necessary, and makes the final decision.

The missing-information output is particularly important. It provides a concrete starting point for deciding what the organization should improve next.

The chicken-and-egg problem of assessment data

Reliable assessment requires reliable information. At the same time, an organization often cannot know which information matters most until it runs a real assessment.

Stable assessment requires better information
                  ↕
You cannot know what to improve until you assess

A practical way forward is not to clean up everything first. It is to proceed in stages:

Select an assessment sheet that is already used in practice.
Register the documents that are currently available.
Run a first-pass assessment on representative targets.
Classify the causes of incorrect and indeterminate results.
Improve the recurring, high-impact information gaps first.
Re-run the same assessment and confirm the effect.

This sequence keeps the improvement scope tied to actual business needs.

Do not add documents every time the AI is wrong

When an AI-generated assessment differs from the expected result, it is tempting to conclude that the RAG knowledge base is incomplete. That may be true, but it is only one possibility.

What happened	What to inspect first
The boundaries between ratings are unclear	Rating definitions and decision criteria
One question contains several independent conditions	The wording and structure of the assessment item
Relevant text was found but interpreted incorrectly	Prompt instructions and terminology
A registered document was not retrieved	Search scope, filters, and retrieval settings
The AI applied the wrong assumptions about the target	Shared target context
A document exists but was never registered	Registration and update process
A control is performed but no record is retained	Evidence-generation process
The control does not exist	Actual remediation or risk treatment

The AskDona usage guide recommends a similar order: review the instructions and rating definitions first, then the assessment question, and only then change the underlying knowledge data.

Before adding more content, identify what the AI misunderstood. Otherwise, the knowledge base may accumulate more documents with overlapping meanings without resolving the decision problem.

Shared target context is often the highest-value first improvement

In system-risk assessment, some of the most consequential errors occur before the AI evaluates a control. The AI misunderstands what the assessment target is.

Common examples include:

Whether the target is internally developed, packaged software, SaaS, or a hybrid
Whether an internal application uses part of an external service
Whether it handles personal, customer, or confidential information
Whether it connects to the internet or another external network
Whether system-to-system communication exists
Which component manages standard and privileged identities
The target’s criticality classification
The environments and locations in scope
External products and shared platforms on which it depends
Whether a control is the responsibility of the organization, the provider, or both

If the AI has to infer these facts from fragments of design documentation every time, applicability errors are likely.

For example, an external platform may be mentioned throughout a design document, causing the AI to treat the entire target as an externally provided service. A corporate privacy policy may describe personal-data controls, causing the AI to assume that every system holds personal data. A file server may be assumed to manage its own identities even when authentication is delegated to a shared platform.

A central IT, security, or governance function can reduce these errors by maintaining shared attributes in a consistent target master.

Example fields for an assessment-target master

Field	Purpose
Target ID	Unique organizational identifier
Official name	Approved system or service name
Alias and former names	Names used in existing documents
Business owner	Department responsible for the service
Technical owner	Development or operations owner
Delivery model	Internal, SaaS, package, or hybrid
Information handled	Personal, customer, confidential, or other categories
External connectivity	Internet, private network, external API, and so on
External dependencies	Providers, shared platforms, industry networks
Identity-management owner	Target, shared identity platform, or provider
Environments	Production, development, test, disaster recovery
Criticality	Organization-defined classification
Responsibility model	Internal, provider-owned, or shared
Effective date	Date to which the attributes apply
Confirming owner	Person or function that validated the record
Last confirmed	Latest review date

These attributes are not evidence that a particular control is implemented. They are controlled context that helps the AI determine applicability and select the right evidence.

Design CSV records so that one target can be understood from one block

When CSV data is registered in AskDona, each row is generally handled as an individual information block. For shared target attributes, it is therefore useful to place the main conditions for one target across the same row.

A fragmented structure

system_name,attribute,value
System A,delivery_model,internally developed with an external service
System A,personal_data,yes
System A,criticality,A
System A,external_connectivity,yes
System A,identity_owner,shared identity platform

Each row becomes a separate block. A retrieval may return the personal-data row without returning the delivery model or identity owner.

A structure that keeps shared context together

system_id,system_name,alias,delivery_model,personal_data,customer_data,criticality,external_connectivity,system_interfaces,identity_owner,external_dependencies,responsibility_model,effective_date,confirmed_by
SYS-A,System A,A Tool,internally developed with an external service,yes,no,A,yes,yes,shared identity platform,External Service X,application settings are internal and platform operation is provider-owned,2026-07-01,IT Governance

When this row is retrieved, the AI receives the main assumptions about System A in one context.

This does not mean that every detail should be forced into a single row. One-to-many relationships should be separated into their own CSV files, while repeating the target ID and name in every row.

system_master.csv
One row per system

system_external_services.csv
One row per system × external service

system_locations.csv
One row per system × location or environment

system_interfaces.csv
One row per system × communication path

The practical rule is simple: a row should be understandable on its own.

Blank values should also be avoided. A blank cannot distinguish “no,” “unknown,” “not yet confirmed,” “not entered,” and “not applicable.” Use explicit values such as:

yes
no
unknown
not confirmed
not applicable

System-specific design knowledge should remain with the field teams

A central governance function cannot realistically maintain every architectural detail and every reason behind every system change.

Field teams may be the only people who know:

The actual current architecture
Which functions of an external service are used
Which default settings were changed
Which data travels through which interfaces
What changed during discussions with a customer
Why a particular design was selected
Which constraints led to a compensating approach
Whether documentation differs from the current production state
Whether temporary exceptions remain in place

This information should remain owned by the teams closest to the system. The central function’s role is to define a minimum structure for recording it.

Useful categories of field-owned information

Record	What it should contain
Current architecture	Components, interfaces, data flows, enabled functions, settings
Architecture decision record	Options considered, selected approach, rationale, assumptions
Change record	Before and after, reason, affected scope, added controls
Responsibility matrix	Internal, provider, and shared responsibilities
Exception record	Reason, compensating control, approver, expiry date
Operational evidence	Test results, approvals, logs, inspection records, inventories

When customer requirements or operational constraints change a design, it is not enough to preserve only the final diagram. The organization should also retain what triggered the change, which alternatives were considered, and what additional controls were introduced.

Assessment often depends on both what the design is and why it became that design.

Do not solve every terminology problem by expanding the glossary forever

If different departments use different names for the same concept, the AI must infer the relationship.

A certificate category, for example, might appear as:

Public certificate
External CA certificate
Third-party certificate
Commercial certificate
Trusted certificate
A department-specific label

Adding every new phrase to a glossary can become an endless maintenance task.

A more sustainable approach has two parts:

Map legacy terms to a controlled organizational term.
Use the controlled term in newly created designs, inventories, requests, and evidence records.

For example, an organization might adopt the following categories:

certificate issued by a third-party certificate authority
certificate issued by an internal certificate authority
self-signed certificate

Existing labels can be mapped to these categories. The important improvement is not only a larger glossary. It is the gradual convergence of new information on a shared vocabulary.

Do not ask subject-matter experts to answer the assessment question directly

Some relevant facts exist only in the memory of engineers, operators, or business owners. They have to be elicited.

However, asking a complex question such as “Is this control compliant?” often produces inconsistent answers. The subject-matter expert may not know the assessment-specific definition or the evidence threshold.

Instead, decompose the requirement into factual questions.

Assume an assessment item asks whether a relevant communication path uses a certificate issued by a third-party certificate authority as part of a specified integrity control.

Useful interview questions include:

Which communication paths are in scope?
Is a certificate used on each path?
Where is the certificate configured?
Who issued it?
Is it third-party-issued, internal-CA-issued, or self-signed?
What is the certificate used for?
Which document or configuration can confirm the setting?
Are production, development, and recovery environments different?
Are any paths handled as exceptions?
Who owns renewal and configuration?

The field team provides target-specific facts. The assessment function decides whether those facts satisfy the requirement and whether the evidence is sufficient.

The interview output should also be structured rather than stored only as meeting minutes.

Assessment target:
In-scope communication:
Technology used:
Issuer or provider:
Purpose:
Environment:
Exceptions:
Available evidence:
Information provider:
Reviewer:
Confirmation date:
Assessment interpretation:

This structure separates the facts provided by the field team from the interpretation applied by the assessor.

AI can research external information, but people must decide whether it applies

AI is well suited to researching external services, products, standards, and industry platforms.

It can help with:

Finding official documentation and standards
Identifying the applicable version
Extracting security requirements
Comparing changes between versions
Summarizing the general capabilities of an external service
Suggesting possible mappings to internal documentation

But external information cannot, by itself, prove that an internal target actually uses, enables, or operates a capability.

People still need to confirm:

Which product or service is actually used
Which version is deployed
Which functions are enabled
Whether defaults were changed
Which external requirements apply to the internal target
Whether an external control can be inherited
Whether additional internal controls are required
Whether responsibility is internal, provider-owned, or shared

A practical division of work is:

AI
Research official external requirements and capabilities

Field team
Confirm the internal target's actual usage and configuration

Governance or assessment function
Determine applicability, responsibility, and assessment treatment

Create evidence when the work is performed, not when the assessment arrives

A person may remember that a control was performed, but without a record, a reviewer cannot verify it later.

Examples include:

A patch was tested before deployment
Privileged access was reviewed periodically
A responsible manager approved a device
An asset was entered into an inventory
A certificate expiry date was checked
Additional controls were applied to an exception

Rather than reconstructing these facts through interviews during every assessment, make evidence retention part of the completion criteria for the underlying work.

Activity	Evidence to retain at completion
Device introduction	Device ID, purpose, connection, requester, approver, approval date
Patch validation	Version, test environment, results, approver
Identity review	Accounts reviewed, date, reviewer, changes and removals
Certificate renewal	Issuer, target, purpose, expiry date, renewal record
Exception handling	Scope, reason, compensating control, approver, expiry
Periodic inspection	Scope, checks, result, corrective actions, completion date

This is not about producing extra documents for AI. It creates records that are also useful for audits, handovers, customer explanations, and future reviews.

Divide work between AI and people according to the type of decision

Work that AI can support well

Searching large document collections
Classifying policy, implementation, operation, and result evidence
Highlighting similarities and conflicts across documents
Listing mandatory conditions that could not be confirmed
Drafting follow-up questions for subject-matter experts
Researching official external specifications and standards
Aggregating recurring information gaps across many targets
Suggesting improvements to document or CSV structure

Work that remains a human responsibility

Defining the assessment target and scope
Establishing organizational terminology and applicability
Deciding what counts as sufficient evidence
Determining whether external requirements apply internally
Establishing responsibility boundaries
Evaluating exceptions and compensating controls
Accepting risk
Approving the final result

AI should not decide organizational policy on behalf of the organization. Its role is to investigate and organize information so that the points requiring human judgment become visible.

Turn “information not found” into a concrete improvement task

When Batch Assessment identifies missing information, route the gap to the correct owner and repository.

Missing information	Primary owner	Improvement destination
Shared target attribute	Governance function	Assessment-target master
Current architecture	Development or technical team	Current architecture record
Design rationale	Development or business team	Architecture decision record
Change context	Project or system owner	Change record
Operational evidence	Operations team	Logs, inventories, approvals
External specification	AI research plus service owner	Official external source set
Responsibility boundary	Governance, legal, and technical owners	Responsibility matrix
Inconsistent terminology	Standards owner	Controlled vocabulary and mapping
Unclear decision condition	Assessment-criteria owner	Evaluation specification
Knowledge held only by an individual	Assessor and field team	Decomposed interview record
Control does not exist	Business or technical owner	Remediation plan or risk treatment

A one-off gap may be handled locally. If the same gap appears across multiple targets, elevate it into a shared improvement: add a field to the target master, revise a template, adopt a standard term, or make evidence retention mandatory.

Build a small improvement loop, not a big-bang data program

A practical maturity path looks like this:

Phase 1 — Start with existing information

Run a first-pass assessment using the current checklist and available documents.

Phase 2 — Classify errors and gaps

Separate instruction problems, rating-definition problems, question structure, retrieval issues, shared context gaps, registration gaps, missing evidence, and real control gaps.

Phase 3 — Improve recurring prerequisites

Prioritize high-impact shared attributes such as delivery model, information handled, connectivity, criticality, external dependencies, and responsibility.

Phase 4 — Introduce field templates

Define minimum fields for current architecture, design decisions, changes, exceptions, and responsibility boundaries.

Phase 5 — Generate evidence as part of operations

Retain the necessary record when approvals, inspections, changes, and tests are completed.

Phase 6 — Reassess and scale

Re-run the same assessment, confirm the improvement, and promote only recurring cross-target issues into organization-wide rules.

This approach avoids making organization-wide data redesign a prerequisite for the first assessment.

Start assessing. The results will show what to improve.

Assessment AI does not need perfect data on day one.

What it needs is a process that makes it possible to see which facts were used, which assumptions were wrong, which conditions could not be confirmed, and which evidence was missing.

AskDona Batch Assessment uses existing assessment sheets and RAG knowledge to produce a first-pass result together with rationale, quotations, sources, and missing information. People review the result and make the final decision.

The output can then be fed back into information improvement:

Central functions maintain shared target context.
Field teams record system-specific design and change rationale.
Operations teams retain evidence when work is performed.
AI researches and organizes external official information.
People decide applicability, evidence sufficiency, exceptions, and risk.
CSV records keep the main conditions for one target together.
Only recurring, high-impact gaps become organization-wide improvements.

Data improvement is not something that must be completed before assessment AI can be used. It is a focused, continuous process guided by the results of real assessments.

The goal is not to create more documents for AI. It is to create an environment in which both people and AI can determine what is a fact, which target and time period it applies to, who owns it, and what evidence supports the conclusion.

Learn more on the AskDona Batch Assessment solution page.