From Simple Prompts to Evidence-Based Assessment

The Production Challenge Begins Before the Answer

Through our work with production chatbot systems, GFLOPS reached a more precise conclusion: model output quality is only one part of the problem.

Foundation models will continue to improve. However, a production system cannot delegate reliability to future model releases. An answer may still be incorrect, unsupported, inconsistent, or unsuitable for a particular business process. Model updates can also change behavior that was previously tested and accepted. Recent research on prompt underspecification found that behavior associated with unstated requirements was substantially less stable across prompt formulations and model updates. In the study’s model-update experiments, 5.9% of unspecified-requirement cases experienced performance declines of more than 20 percentage points, compared with 3.0% of cases in which the requirement was explicitly stated. [1]

The problem therefore is not that AI answers no longer matter. The more useful observation is that, before an answer can be reliable, the system must determine whether the user has provided enough information to define the task.

In many cases, the first thing a user types is not a complete question. It is only the beginning of one.

Users Begin with a Goal, Not a Complete Specification

People often express what they want in a few words:

What is the weather?

What is today’s date?

How much paid leave do I have left?

Each question appears simple, but even these examples contain unstated requirements.

A weather question requires a location, a relevant time period, and possibly a preferred unit of measurement. Today’s date depends on the user’s time zone. A paid-leave question requires the system to identify the employee, determine the type of leave, obtain current HR records, account for used and approved leave, apply accrual and carryover rules, and use the policy version effective on the relevant date.

The user should not need to know all of these dependencies before asking the question.

Years of interacting with search boxes may encourage people to submit brief, retrieval-oriented queries. However, the more important product principle is independent of why they do it: users naturally begin with the part of the problem that is visible to them. They do not necessarily know which facts, rules, or documents the system needs.

Research on conversational search similarly treats short, ambiguous, and underspecified queries as a recurring information-retrieval problem.

A production AI system should therefore not assume that the first message is a complete task specification.

Enterprise AI Receives Many Kinds of Requests

Enterprise interactions with AI cannot be reduced to three types of questions. In fact, many enterprise inputs are not questions at all. They are requests, instructions, or tasks:

Summarize this contract.

Compare these two policies.

Draft a response to the regulator.

Update the customer record.

Alert me when this limit is exceeded.

For this reason, enterprise requests is a more useful term than enterprise questions when discussing the full range of interactions between employees, customers, and AI systems.

Users may ask a system to retrieve information, explain a concept, summarize or compare documents, analyze data, diagnose a problem, create content, recommend an action, assess a case, execute a transaction, or monitor an ongoing process. Many requests combine several of these purposes.

Research on enterprise AI usage reflects this breadth. An analysis of approximately 5.5 million Microsoft 365 Copilot sessions identified uses including writing and editing, information retrieval, analysis, decision-making and strategizing, technical assistance, and the evaluation or diagnosis of programs and systems. [2]

There is therefore no single short taxonomy that describes every enterprise request. In customer-service environments, domain-specific intent taxonomies may contain more than one hundred categories, reflecting the detailed reasons customers contact an organization. [3]

At a high level, enterprise requests may include the following categories. This list is illustrative rather than exhaustive.

Request type	Primary purpose	Example
Retrieval	Find information or documents	”Where is the travel policy?”
Explanation	Explain a term, rule, or reason	”What does enhanced due diligence mean?”
Summarization	Condense lengthy material	”Summarize the changes in this policy.”
Comparison and synthesis	Compare and combine multiple sources	”How do the EU and Japanese requirements differ?”
Calculation and analysis	Calculate or analyze data	”How much paid leave do I have remaining?”
Troubleshooting and diagnosis	Identify the cause of a problem	”Why was this transaction rejected?”
Creation and transformation	Create or transform content	”Draft a customer notification based on this policy.”
Recommendation and planning	Recommend options or next steps	”What should we do before submitting this application?”
Assessment and classification	Apply criteria to evaluate or classify a case	”Does this customer require enhanced due diligence?”
Action and transaction	Execute an authorized operation	”Submit my leave request.”
Monitoring and audit	Monitor conditions or inspect ongoing work	”Which cases are overdue or missing evidence?”

A Single Request May Contain Multiple Intents

These categories are not mutually exclusive.

For example:

Why was this customer application rejected, and what evidence is needed for reconsideration?

This request may involve troubleshooting, explanation, information retrieval, assessment, and recommendation at the same time.

It is therefore more practical to treat many enterprise requests as multi-intent rather than assign each request a single label. The relevant system behavior depends not only on what the user asks, but also on which sources, permissions, calculations, judgments, and actions are required to complete the request safely.

Two Dimensions for Understanding Enterprise Requests

A more useful framework separates two different dimensions.

The first dimension is what the user is trying to achieve. The user may want to:

• find information;
• understand a concept or decision;
• compare alternatives;
• analyze data;
• create or transform content;
• solve a problem;
• make or support a judgment;
• execute an action; or
• monitor an ongoing condition.

The second dimension is what the system requires in order to respond or act reliably. This may include:

• public, internal, confidential, or personal information;
• one source or multiple sources;
• a general rule or a case-specific conclusion;
• current information or information valid at a particular historical date;
• one jurisdiction or multiple jurisdictions;
• a calculation, an exception analysis, or both;
• supporting evidence;
• access to another enterprise system;
• verification of the user’s role and authority;
• human approval; and
• additional safeguards where an error could affect health, rights, safety, employment, or financial well-being.

Using these two dimensions makes the nature of a request clearer.

For example:

How much paid leave do I have remaining?

can be described as:

User intent: Personalized calculation and information retrieval
Operational requirements: Authenticated personal data, current HR records, the applicable policy version, a date-sensitive calculation, and access control

Similarly:

Can we onboard this customer?

can be described as:

User intent: Assessment, classification, recommendation, and potentially transaction execution
Operational requirements: Jurisdiction-specific rules, customer data, identity evidence, ownership information, sanctions and politically exposed person checks, policy versioning, explainability, audit records, and authorized human approval

The second request is not merely a more complicated question. It is a controlled workflow involving multiple information sources, regulatory criteria, permissions, evidence requirements, and accountable decisions.

From Information Retrieval to Evidence-Based Assessment

Within this broader landscape, GFLOPS is particularly interested in one important progression:

source retrieval → case-specific contextualization → evidence-based assessment

This progression should not be understood as a complete classification of enterprise requests. It describes an increase in the amount of context, evidence, reasoning, control, and accountability required from the system.

Source Retrieval

A retrieval request asks what an approved source says:

What does the company policy say about carrying over unused leave?

The primary task is to identify the relevant document, locate the applicable passage, and present it with a reliable citation.

Case-Specific Contextualization

A contextualized request combines general rules with authorized information about a particular person, transaction, product, or event:

How many days of paid leave do I have remaining?

The system must identify the employee, retrieve current HR data, determine which leave policy applies, account for leave already used or approved, and calculate the result as of the relevant date.

This is no longer only a document-search problem. It also involves identity, access control, structured data, policy versioning, and calculation.

Evidence-Based Assessment

An assessment request asks how documented criteria apply to a particular case:

After transferring to another entity in the group and taking parental leave, am I eligible to carry over my unused paid leave?

The system may need to determine the nature of the transfer, continuity of employment, the applicable jurisdiction, the relevant policy version, treatment of the leave period, available exceptions, and the evidence contained in the employee’s records.

The output is not merely a piece of information. It is a preliminary conclusion whose supporting facts, governing criteria, assumptions, missing evidence, and unresolved issues must remain visible.

Controlled Action

Some enterprise workflows continue beyond assessment into an authorized action:

Submit the approved carryover request to the HR system.

At that point, the system must also verify authorization, preserve an audit record, handle failure conditions, and distinguish between a recommendation generated by AI and an action approved by an accountable person.

The important transition for GFLOPS is therefore not from one universal question type to another. It is the transition from answering isolated requests to supporting a structured process in which the system determines what information, evidence, rules, permissions, and human decisions are required before an outcome can be trusted.

GFLOPS does not claim that all enterprise interactions fall into three categories. We focus on one particularly difficult transition: moving from retrieving information to applying documented criteria to a specific case in a traceable and accountable way.

Users Should Not Need to Become Prompt Engineers

LLM outputs can be sensitive to apparently minor differences in wording, structure, formatting, and punctuation.

This has led to the idea that users must learn prompt engineering in order to obtain intelligent results. That is not an appropriate long-term design principle for production software.

A system that works reliably only when the user writes an expert-level prompt has transferred part of the application-design burden to the user.

The objective should not be to teach every employee or customer how to write longer prompts. Longer is not automatically better. Research on underspecified prompts has found that simply adding every possible requirement can make prompts difficult for models to follow, especially when requirements interact or conflict. [1]

Instead, the system should help discover what is missing.

It should be capable of saying:

I can assess this, but I first need to confirm your employment entity, the effective date of the transfer, and whether the unused leave was accrued before or after the transfer.

That is more useful than silently assuming the missing facts.

Human Conversation Builds the Question Collaboratively

When people ask another person for advice, they usually have an expectation of that person’s expertise. They may ask an accountant a different question from the one they would ask a doctor, lawyer, engineer, or colleague.

The question also changes during the conversation.

A listener may ask what a particular word means. The speaker may notice that the listener has misunderstood and rephrase the request. The listener may repeat part of the question to confirm it. In face-to-face communication, timing, tone, gaze, gesture, and facial feedback can provide additional—although imperfect—signals about whether shared understanding has been established.

Conversation research describes this as grounding and repair: participants collaboratively establish enough common understanding for the purpose of the interaction and correct problems of speaking, hearing, or interpretation when they occur.

A typical text-based enterprise chatbot does not have access to most of these signals. More importantly, it often behaves as though its role is to answer the latest message immediately rather than determine whether the message is sufficiently specified.

AI Can Clarify Questions, but It Does Not Do So Reliably by Default

It would be incorrect to say that AI cannot improve, expand, or clarify a user’s question.

LLMs can rewrite prompts, maintain conversational context, generate follow-up questions, and identify some forms of ambiguity. A study using real human–AI conversation data found that rewriting ineffective user prompts could improve subsequent responses while broadly preserving user intent. The improvements were generally stronger when more conversational context was available.

However, the ability is not yet reliable enough to be assumed.

Recent research has found a gap between recognizing ambiguity and acting on it. Models may identify that a question is ambiguous when explicitly asked to analyze the question, but when asked the same question normally, they often provide a direct answer instead of seeking clarification.

Another benchmark found that current models still struggle to identify different forms of ambiguity and generate consistently useful clarification questions.

Multi-turn conversation does not automatically solve the problem either. In one large simulation study, models often made assumptions in early turns, committed prematurely to a solution, and failed to recover when later information revealed that the initial direction was wrong.

The correct conclusion is therefore:

AI can participate in question clarification, but a generic chatbot should not be expected to perform this process reliably without explicit system design, grounding, evaluation, and monitoring.

A Rewritten Question Can Still Contain an Unverified Assumption

Automatically rewriting a short question into a detailed prompt can improve the response. It can also introduce assumptions that the user never confirmed.

For example, the system might rewrite:

Can I carry over my leave?

as:

Can a full-time employee in Japan carry over unused annual leave into the next calendar year under the current company policy?

This version is more detailed, but it has assumed the user is a full-time employee in Japan and that the relevant period is the next calendar year.

Those assumptions may be plausible, but they are not verified facts.

In a regulated or high-impact workflow, the system must clearly separate:

• facts stated by the user;
• facts retrieved from an authorized system of record;
• requirements retrieved from regulations or internal policies;
• inferences proposed by the model;
• facts that remain unverified; and
• missing evidence that could change the outcome.

The AI should not silently improve a question by inventing its missing conditions. It should make those conditions visible and ask the user or an authorized data source to confirm them.

Regulations Give Assessment Work a Structure

Regulations are not themselves user questions.

They establish obligations, prohibited conditions, thresholds, exceptions, procedural requirements, evidence standards, record-retention duties, and explanation requirements. Internal policies and operating procedures then translate those rules into organizational processes.

An assessment applies those requirements to a particular set of facts.

For example, European medical-device rules require manufacturers to conduct and document clinical evaluations and maintain technical documentation that allows conformity with applicable requirements to be assessed. [4]

In U.S. credit decisions, reasons given for certain adverse actions must accurately describe the factors that were actually considered or scored. [5]

International AML and counter-terrorist-financing standards provide a common framework that jurisdictions implement according to their legal, administrative, and financial circumstances. [6]

These examples differ substantially, but they share a basic structure:

1. determine which requirements apply;
2. collect the relevant case facts;
3. verify the required evidence;
4. identify missing or contradictory information;
5. apply the criteria and exceptions;
6. document the basis of the assessment; and
7. assign accountability for the result.

This is more than information retrieval. It is evidence-based assessment.

The Question Is Produced by More Than the User

In complex assessment work, the correct question does not come entirely from the person using the chatbot.

It is constructed from several elements:

  user objective
+ applicable regulation and policy
+ case-specific facts
+ evidence requirements
+ unresolved exceptions
─────────────────────────────────
= assessable question

A user may know the outcome they need, but not the regulatory conditions that determine it. The applicable documents may reveal questions that the user would never have thought to ask.

The system should therefore retrieve the relevant sources before deciding which follow-up questions are necessary. Clarification questions should themselves be grounded in the available documents. Otherwise, the model may ask plausible but irrelevant questions that are unsupported by the governing material. Research on retrieval-augmented clarification has identified corpus-grounding as an important requirement for faithful conversational search.

The purpose is not to ask as many questions as possible. It is to ask the minimum set of questions whose answers can materially affect the assessment.

From Chatbot to Assessment System

A conventional chatbot waits for a question and generates an answer.

An assessment-oriented system should perform a controlled process:

1. identify the user’s objective and the type of decision being considered;
2. determine the relevant jurisdiction, organization, product, and effective date;
3. retrieve the applicable regulations, policies, procedures, and precedents;
4. identify the criteria, exceptions, and evidence requirements;
5. ask source-grounded clarification questions;
6. retrieve authorized case data and supporting documents;
7. distinguish verified facts from model assumptions;
8. detect contradictions and missing evidence;
9. abstain or escalate when the available information is insufficient;
10. generate a preliminary assessment with citations, unresolved issues, and a traceable reasoning record; and
11. preserve human authority for decisions that require professional, legal, ethical, or organizational accountability.

This is not a single perfect prompt. It is a structured interview and evidence-review process.

GFLOPS’s Direction

At GFLOPS, we believe the next stage of enterprise AI is not simply another chatbot that answers more questions.

It is a system that helps construct the question that must be answered.

The system should not require the employee or customer to understand every regulation, internal policy, exception, or evidence requirement before beginning. It should use those sources to determine what information is necessary, what remains uncertain, and which questions should be asked next.

The objective is not to hide complexity behind a fluent answer. It is to make the complexity manageable and visible.

In regulated and high-impact work, intelligence is not demonstrated by how confidently an AI responds. It is demonstrated by whether the system can:

• identify the applicable criteria;
• collect the facts required by those criteria;
• distinguish evidence from assumption;
• recognize when information is insufficient;
• show which sources support the assessment; and
• preserve accountability for the final decision.

The right question is therefore not merely a better sentence written by the user.

It is a structured and verifiable specification produced from the user’s objective, the governing rules, the case record, and the evidence still required.

GFLOPS aims to move enterprise AI from simple question answering toward evidence-based assessment—helping organizations determine not only how to answer, but what must be asked before an answer can be trusted.

References

1. Yang, C., Shi, Y., Ma, Q., Liu, M. X., Kaestner, C., & Wu, T. (2026). “What Prompts Don’t Say: Understanding and Managing Underspecification in LLM Prompts.” Findings of the Association for Computational Linguistics: ACL 2026, 9072–9101. https://doi.org/10.18653/v1/2026.findings-acl.441
2. Analysis of approximately 5.5 million Microsoft 365 Copilot sessions, arXiv:2605.23958.
3. Domain-specific intent taxonomies for customer-service conversations, arXiv:2505.09436.
4. European Union, Regulation (EU) 2017/745 on medical devices.
5. Consumer Financial Protection Bureau, Regulation B, 12 CFR Part 1002 (Equal Credit Opportunity Act).
6. Financial Action Task Force, International Standards on Combating Money Laundering and the Financing of Terrorism and Proliferation (FATF Recommendations).