When your organization introduced generative AI, did you encounter problems like these?
- “The AI confidently provided contact information that was not included in the company rules.”
- “I asked about upcoming events, but it returned an announcement from last year.”
- “It found both old and new documents about the same policy and based its answer on the older one.”
RAG is a mechanism in which generative AI creates text by referring to internal information designated by an organization. As long as AI is used, the risk of hallucination cannot be eliminated completely. In particular, when instructions for handling reference information are ambiguous, the AI may interpret them in unintended ways and provide incorrect guidance.
In other words, these problems are not always caused only by the retrieval accuracy of RAG (retrieval-augmented generation). Even if retrieval finds the right documents, an appropriate answer is not guaranteed when the AI has not been clearly told how to use them.
This is why system prompts for RAG are important.
In a standard LLM chat, a system prompt is often used to define the model’s role, tone, and output format. RAG requires additional decisions: how the model should read retrieved reference information, how far it may go in answering, and when it should not answer.
GFLOPS, the provider of the enterprise RAG platform “AskDona,” supplies RAG infrastructure for the support site of RIKEN’s supercomputer Fugaku, the website of the National Institute of Informatics, and private-sector companies, including those in the financial industry. Through this implementation and operational support, GFLOPS has accumulated knowledge about how organizations can use RAG reliably.
Based on lessons from real-world operations, this article presents six important principles for designing RAG system prompts in companies and other organizations, together with practical prompt examples.
1. What Is a System Prompt?
A system prompt is an instruction that defines the AI’s role and the conditions it must follow before interacting with a user. It is treated separately from the question entered by the user in the chat interface, which is called the user prompt.
For example:
- You handle inquiries about internal company policies.
- Keep each answer within 400 characters.
- Do not output personal information.
- If you cannot find supporting evidence, do not speculate. Tell the user that the information could not be confirmed.
The technological developments that led to today’s chat-based LLMs are easier to understand when divided into two major streams.
The first is training models to follow human instructions. The InstructGPT paper published by OpenAI in March 2022 described a method for adjusting models using responses that people judged to be desirable. [1]
The second is structuring instructions according to who provided them. The Chat Completions API released by OpenAI on March 1, 2023 introduced a format in which conversations are passed as messages with roles, making it possible to distinguish application-level instructions from user input. [2]
In April 2024, an OpenAI research team proposed an Instruction Hierarchy that prioritizes higher-level instructions when they conflict with lower-level ones. The paper’s basic ordering was system messages, user messages, and third-party content obtained from tools and other sources. [3] The first Model Spec, released the following month, organized the priority order as Platform > Developer > User > Tool. [4]
However, three points are important to understand.
A system prompt is not an absolute command that will be followed 100% of the time
Even when instruction priorities are defined, the model will not always behave as expected. Prompts are an important means of control, but they alone cannot guarantee safety or accuracy. Evaluation and improvement using real questions are essential.
Some problems cannot be solved by the system prompt alone
In RAG, the LLM processes the user’s question together with a certain volume of reference information, response-format requirements, prohibitions, and other instructions. The ability to handle these inputs and follow instructions differs depending on how the model was trained and on its characteristics.
Therefore, even a carefully designed system prompt may not enable some models to handle reference information properly or follow multiple instructions consistently.
Priority and specificity matter more than length
Adding more rules does not necessarily reduce performance. However, packing the prompt with similar or contradictory instructions makes the priorities unclear. Longer prompts are also more likely to contain duplication and conflicting exceptions.
Deciding “what takes highest priority” and “what to remove” is just as important as deciding what to write.
2. Differences Between Standard LLM Chat and RAG
The following settings are commonly used in system prompts for standard LLM chats.
- Assign a role: “Respond as an experienced HR representative.”
- Define the output format: specify the length, headings, bullet points, writing style, and other requirements.
- Provide examples: when decision criteria are difficult to communicate in words alone, include examples of questions and answers.
- Define prohibitions: “Do not make medical diagnoses.” “Do not output personal information.”
These settings are also effective in RAG. However, RAG has one more important element: “reference information” retrieved from an external knowledge database.
RAG searches for documents related to a question, passes portions of them to the LLM as reference information, and generates an answer. The foundational 2020 RAG paper also described an architecture that retrieves from an external document index and uses the results for generation in addition to the model’s internal knowledge. [5]
A RAG system prompt must therefore define not only how the answer should be expressed, but also how the following decisions should be made.
- May the model fill in content that is absent from the reference information?
- How should it respond when supporting evidence is insufficient?
- Which source should take priority when multiple documents conflict?
- How should it distinguish past information from information that is currently valid?
- How should it show users which sources support the answer?
If these points remain ambiguous and a standard chat prompt is reused, the defining RAG behavior of producing “evidence-based answers” is likely to break down.
Figure 1. Standard LLM chat vs. RAG
In RAG, in addition to the standard chat settings, you must decide how to handle the retrieved reference information.
Standard LLM chat
Mainly controls role, expression, and output format
- Assign a role: “Respond as an experienced HR representative.”
- Define the output format: length, headings, bullet points, writing style
- Provide examples: add question-and-answer examples when criteria are hard to convey in words alone
- Define prohibitions: “Do not make medical diagnoses.” “Do not output personal information.”
RAG (retrieval-augmented generation)
The four items on the left apply to RAG too
- Role / output format / examples / prohibitions
+ Controls when to use, reject, prioritize, and cite reference information
- May the model fill in content absent from the reference information?
- How should it respond when supporting evidence is insufficient?
- Which source should take priority when documents conflict?
- How should it distinguish past information from currently valid information?
- How should it show users which sources support the answer?
If the five items on the right remain ambiguous and a standard chat prompt is reused, the defining RAG behavior of “evidence-based answers” tends to break down.
3. Six Key Principles for RAG System Prompts
3-1. Do Not Let the Model Fill In Information Beyond the References
One purpose of adopting RAG is to ground answers in documents managed by the organization. However, an LLM may fill gaps in the reference information with general knowledge learned during training.
The result may be correct in general but wrong for your organization’s policies. This discrepancy is a major problem for RAG systems used by companies and other organizations.
An instruction such as “Answer accurately” does not make the prohibited behavior clear. Specify the types of information that must not be generated.
[Scope of Reference Information]
- Create the answer based on the provided reference information.
- Do not generate email addresses, URLs, numbers, dates, policy names,
or procedures that are not in the reference information.
- If the reference information does not contain evidence that supports an answer,
do not fill the gap with speculation or general knowledge.
Tell the user, "The answer could not be confirmed from the reference information."Email addresses and URLs are called out specifically because their formats make plausible-looking strings easy to fabricate, while users may find it difficult to determine whether they are genuine.
A case where no supporting evidence exists must be handled separately from a case where retrieval or the system has failed. If the same standard message hides both cases, an outage may go unnoticed. Ideally, the application should be able to distinguish between zero retrieval results and a failure of the retrieval process itself.
3-2. Treat “Do Not Answer” and “Ask a Follow-up Question” as Normal Behavior
In a standard chat, the model is often expected to return some kind of answer. In RAG, however, not answering is the correct behavior when there is no supporting evidence.
When a question is ambiguous, asking a follow-up question is also faster and safer than forcing an answer.
[Follow-up Questions]
- If the reference information contains multiple candidates and a single answer
cannot be determined, do not speculate.
- Ask for one necessary detail at a time, such as the target, time period,
symptom, or policy name.
- When asking a follow-up question, do not use the standard answer template.For example, the statement “My PC is not working” does not reveal whether it will not power on, the user cannot log in, or it freezes during operation. Confirming the situation first avoids providing a long set of incorrect instructions.
The decision to ask a follow-up question can be difficult to define with prose alone. In such cases, one or two few-shot examples can help.
Example:
User: "My PC is not working."
Answer: "Which option is closest to the current state?
1. It will not power on 2. I cannot log in
3. It freezes during operation 4. Other"However, too many examples can cause the model to overfit to individual cases or bury the most important rules. Review the question patterns in your organization and begin with cases that are frequent or have a strong need for follow-up questions.
3-3. Define How Sources and Links Should Be Displayed
A major benefit of RAG is that users can trace the evidence behind an answer. However, source displays are not necessarily formatted automatically. Both the metadata passed with the answer and the output rules must be designed.
[Displaying Sources]
- List the names of the documents used for the answer under "Sources"
at the end of the response.
- Include a link only when the URL is present in the reference information.
- Output links in Markdown format: [Document Name](URL).
- Do not generate URLs that are absent from the reference information.Fixing the link format reduces problems such as long raw URLs appearing in the response or links being broken across lines.
If articles from external sites are collected through your own website, decide which URL should be shown. Whether to direct users to the original article or to an internal detail page should be specified as an operational rule, not left for the AI to infer.
3-4. Define Priorities When Multiple Documents Conflict
An internal knowledge database contains multiple documents on the same topic, such as an old policy and its revised version, or an official regulation and meeting notes from an explanatory session. Both may appear in the retrieval results.
The organization must define which source is the primary basis for an answer instead of leaving the decision to the LLM.
[Priority of Reference Information]
When multiple sources can support an answer, prioritize them in the following order.
1. Currently effective policies and regulations
2. Official notices and procedural documents
3. FAQs and explanatory materials
4. Meeting minutes and other materials
- If multiple documents are of the same type, prioritize the one with the most
recent effective date or update date.
- If you cannot determine which document is currently valid, do not make a
definitive statement. Tell the user that validity could not be determined.
- If currently valid documents conflict, do not hide the inconsistency.
Show the document names and dates.Simply instructing the model to “prioritize newer documents” may be insufficient. An older regulation may still be in force, while a newer document may only be explanatory material from a briefing. Metadata (categories, tags, and other attributes assigned to documents in the knowledge database), such as document type, effective date, expiration date, and amendment or repeal relationships, enables more reliable decisions without relying on the prompt alone.
3-5. Provide a Reference Date for “Upcoming,” “Current,” and “Latest”
Time is easy to overlook, but it can have a significant impact.
A knowledge database accumulates past events, closed application notices, and information from earlier fiscal years. Even when a user asks about “upcoming events,” retrieval may return older articles containing words such as “scheduled” or “held.”
[Handling Dates]
- Use {CURRENT_DATE} as the reference date.
- When the user asks about "upcoming," "next," or "from now on,"
consider only information dated after the reference date.
- Do not present completed events or closed applications as upcoming schedules.
- If the reference information contains no future information,
say that no published schedule could be confirmed.
- Display a date only when it is stated in the reference information,
using the Month D, YYYY format.The important point is to pass the correct runtime date into {CURRENT_DATE}. If a fixed statement such as “The current year is 2026” is written directly in the prompt, it may not be updated. Ideally, the application should inject the current date, time, and time zone on every run.
A newly published article does not necessarily describe a policy that is currently valid. Metadata that distinguishes publication date from event date, and update date from effective date, may also be useful.
3-6. Map Organization-Specific Terms and Abbreviations
Every organization has abbreviations used more often than official names, common names for systems, and former department names. Users ask questions using familiar terms, while the documents may contain only the official names.
[Terminology Map]
- {System A} (abbreviation: {Abbreviation A}): {Purpose}
- {System B} (common name: {Common Name B}): {Purpose}
Interpret an abbreviation or common name used in a question as the corresponding
official name.
If an abbreviation could refer to multiple official names, do not speculate.
Ask a follow-up question.Terminology maps can become lengthy. As a rule, manage them separately as background knowledge or a glossary instead of packing them into the same field as the highest-priority rules. Uploading a glossary to the knowledge database instead of including it in the system prompt can also be effective.
Do not casually use realistic department names, email addresses, or URLs in examples or mapping tables. The model may reuse those example strings as evidence in an answer. Clearly mark replacement fields, such as {Department Name}.
4. Do Not Mark Every Rule as “Absolute”
The six principles above should not all be presented with equal weight. Their importance should be differentiated.
| Level | Purpose | Examples |
|---|---|---|
| Highest-Priority Rules | Core rules whose violation would have a major business impact | Prohibit fabricated information; respond appropriately when evidence is insufficient; ask necessary follow-up questions |
| Mandatory Rules | Operational rules that should normally be followed | Display sources; response length; link format; classification tags |
| Recommended Rules | Supporting rules that improve quality | Wording; headings; notes; writing style |
Starting with about one to three highest-priority rules is usually easier to manage. Place rules with the greatest business impact at the top, review chat histories and answer evaluations, and improve the prompt during operation.
AskDona Separates Rules from Background Knowledge
AskDona divides system prompt settings into two fields.
| Field | Length | Primary Content |
|---|---|---|
| System Environment Settings | Up to 2,000 characters | Priority rules, answer templates, handling of reference information, and output format |
| AI Personality Settings | No limit | AI role, organization description, glossary, and inquiry contact maps |
Conceptually, System Environment Settings contain the “business rules,” while AI Personality Settings contain the “role and background knowledge.” Placing critical instructions in the former and longer definitions or mapping tables in the latter makes the priorities easier to retain.
When multiple settings are present, AskDona applies them in the following order, with later settings taking effect later in the sequence.
- AskDona default settings (users cannot view or change them)
- Organization-level System Environment Settings
- Organization-level AI Personality Settings
- User-level system prompt
If a setting is not reflected as expected, investigate which layer conflicts with or overrides it.
5. Configuration Examples by Use Case
5-1. Internal Inquiries Across Multiple Departments
This use case stores information from HR, General Affairs, Accounting, and other departments in one knowledge database and identifies the responsible department together with the answer.
[Highest-Priority Rules]
- If the subject of the question is ambiguous or could fall under multiple
policies, do not speculate. Ask a follow-up question.
- Do not generate email addresses, URLs, policies, or procedures that are
absent from the reference information.
[Mandatory Rules]
- Always output [Responsible Department] at the end of the answer.
- The responsible department must exactly match one of the following values:
Human Resources / General Affairs / Accounting / Information Systems / Other
- Use "Other" only when the department cannot be determined.
- Quote the contact point from the "Inquiry Contact Map" in the background knowledge.Place supporting rules for department classification and contact lists in AI Personality Settings.
Using [Responsible Department] at the end of each answer as a fixed classification tag makes it possible to aggregate chat histories and analyze which departments receive the most inquiries. The key is to prohibit variations in spelling.
5-2. A Chatbot Using a Public Website as Its Knowledge Database
This use case automatically collects your website and uses it as a knowledge database. Because past articles accumulate, controlling time references and link destinations is important.
[Highest-Priority Rules]
- Use {CURRENT_DATE} as the reference date.
- When asked about upcoming schedules, provide only information dated after
the reference date.
- Do not present past articles or completed events as upcoming schedules.
- If no future schedule can be confirmed from the reference information, say so.
Provide a listing page only when it is included in the reference information.
- When presenting information from an external site, use the original article URL
included in the reference information.
- Do not generate URLs, dates, or facts that are absent from the reference information.
[Link Format]
- Display links in this format: [Article Title (Publication or Organizer)](URL).Pass the current date as a runtime value instead of writing a fixed value in AI Personality Settings. This prevents outdated settings from remaining after the year changes and keeps the reference consistent, including the time zone.
6. System Prompts Should Be Updated During Operation
A RAG system prompt must control more than the tone of the answer. Its central purpose is to define how retrieved reference information should be handled.
- Do not fill in facts that are absent from the reference information.
- Do not answer when there is no supporting evidence, and ask a follow-up question when the request is ambiguous.
- Standardize source displays and link formats.
- Define priorities among multiple documents.
- Distinguish the past, present, and future.
- Map organization-specific terms and abbreviations.
Do not assign every instruction the same strength. Place rules with greater business impact higher and keep stylistic preferences lower.
You do not need to create a perfect prompt from the beginning. Actual chat histories will reveal unexpected questions, cases where retrieval succeeded but answer generation failed, and cases that required a follow-up question. Accumulate these examples as evaluation data, then separately improve the prompt, retrieval settings, document metadata, and the characteristics of the model used for answer generation. This is the shortest path to reliable RAG operations.
AskDona separates rules from background knowledge and clarifies the order in which settings are applied, enabling an organization’s own staff to manage system prompts. If you need help with prompt design or improvement, our team can review your chat history and recommend adjustments. Please contact your customer success representative.
References
- Ouyang, L. et al., Training language models to follow instructions with human feedback, arXiv:2203.02155, 2022.
- OpenAI, Introducing APIs for GPT-3.5 Turbo and Whisper, March 1, 2023.
- Wallace, E. et al., The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions, arXiv:2404.13208, 2024.
- OpenAI, Model Spec (2024/05/08), May 8, 2024.
- Lewis, P. et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, NeurIPS 2020.
- Anthropic, System Prompts - Claude Platform Docs.