Grounding Product Knowledge in AI Agents: Four Lessons

According to an article published by Salesforce, trust in an AI agent’s answers starts first and foremost with the content behind it. The company describes how it uses Data 360 to prepare and organize product knowledge, while the Agentforce platform puts it to work for customers. This process is defined as grounding: connecting the agent to corporate content so that its answers are sourced from trusted information. When a customer asks a question, the agent retrieves the relevant passages and builds its response using RAG (retrieval-augmented generation). This means the agent can only be as good as the content it retrieves. The company shared four lessons from its Customer Zero journey, in which it served as the first customer of its own products.

Lesson 1: Fit, Gap, and Clash – Assessing Content Before Indexing

The article emphasizes that content quality sets the standard for the agent, which is why content must be assessed before beginning build and model configuration work. Salesforce presents three questions to ask of every article in the knowledge base:

1. Fit: Does the article answer a real customer question?
2. Gap: Does it cover an issue customers encounter that has not yet been addressed?
3. Clash: Does it contradict other content, such as two articles outlining different steps for the same task?

To assess existing content, the company uses four measurement signals:

• Article Score: A composite health score of the article.
• Self-Help Failure Rate: How often a user read an article and still filed a support case.
• Problem Resolution Rate: A user vote confirming that the article helped.
• Sessions: Pure overall traffic volume.

Remediations take place first where high traffic is recorded alongside low performance. These signals are paired with top case drivers, thereby surfacing gaps in content.

The company notes that adding content does not automatically guarantee better coverage, as a larger index can make an agent slower and more confused. As an example, after assessing content for fit and given a tight timeline, Salesforce decided to build a separate agent for the Informatica support portal instead of adding the acquired content to the existing Help Agent. This decision resulted in the separate agent now resolving 92% of questions on the site without a support case being filed.

The work of preparing content for agents is shared between two teams: the content team is responsible for authoring, tone, structure, and best practices; the AI and Data team is responsible for agentic readiness, including metadata, chunking (splitting articles into pieces for the search index), retrieval tuning, and evaluation. Agent Managers work within the AI and Data team, aligned by product line and often comprising reskilled support engineers. They analyze conversation data in production environments and work directly with content writers to close knowledge gaps.

Lesson 2: Building by Hypothesis Instead of Rigid Specifications

In traditional software development, the starting point is requirements and specifications, meaning the outcome of the build is roughly known in advance. In contrast, agent performance depends on many levers, including the content included and the configuration of embedding models and search indexes. It is impossible to know the effect beforehand without building and testing. The process operates in a consistent loop:

1. Hypothesize: Define how the change is expected to affect the user experience.
2. Build: Create the new configuration in Data 360 (termed a "recipe").
3. Evaluate: Run quantitative and qualitative testing.
4. Reject or deploy: Roll out what passes testing, often via A/B testing, and continue monitoring.

The article shares an example of a rejected hypothesis: because articles do not always specify which product they belong to, it was hypothesized that adding product names as metadata would improve answers. After building the recipe, quantitative metrics before and after the change remained flat, and qualitative checks confirmed this. The decision was made not to deploy the change. The experiment demonstrated that metadata only helps a RAG system when the agent is also instructed on how to use it. Early tests combining metadata with prompt instructions look much better qualitatively, and the company is building a new recipe incorporating both that will be evaluated in the same way before decisions are made.

Lesson 3: Defining Desired Outcomes Before Testing Begins

To quantitatively evaluate a recipe, a golden data set is required. Salesforce's data set contains approximately 200 user utterances, with each utterance paired with an expected validated answer and links to articles containing the information. AI can assist in drafting these initially, but a human must validate each case against the source articles to verify accuracy and factual grounding.

In qualitative evaluations, acceptance criteria are established around three attributes: correct, complete, and relevant. Subject-matter experts previously spent hundreds of hours manually writing these criteria for every utterance. Today, they supply the best content source for each utterance and review criteria generated via an AI prompt.

Lesson 4: Combining Quantitative and Human Testing

Quantitative testing at the company is divided into three tiers:

• Deterministic: Whether the retrieval mechanism found the correct article, and what its rank was.
• Semantic quality: Whether the retrieved information was relevant and summarized correctly.
• Business outcomes: Whether the agent is achieving defined business goals.

Alongside mathematical metrics, human reviewers and LLM judges assess responses across three dimensions: answer quality (correct, complete, and relevant), service quality (tone, structure, and disambiguation), and AI quality (retrieval metrics and latency). An answer can be factually accurate yet unhelpful if it points to 12 documentation links.

The combination of testing methods was demonstrated in an evaluation of article chunking: Salesforce had previously split articles by fixed token size. It was hypothesized that section-aware chunking in Data 360 would improve answers by keeping related information together across natural section breaks. Both methods were tested against a golden data set of 195 query-document pairs:

• In quantitative testing: Most metrics improved, and those that dipped did so slightly and not significantly.
• In qualitative testing: Metrics flagged 8 of the 195 queries as failures (about 4%). Upon human review, most proved to be false failures, as the retrieval mechanism had pulled a valid document, just not the exact document bookmarked in advance. The true failure rate was close to 1%.

Approval to move to production was granted only following this human review.

Recommendations for Getting Started with an Agent

The article notes that the first step depends on the agent’s current status:

• For an agent not yet live: Evaluate Fit, Gap, and Clash across the top 100 articles in the knowledge base, avoiding model changes before completing this step.
• For an agent already operating in production: Identify the most frequent errors (missing documents, inappropriate tone, or correct documents with an incorrect answer), and match each failure to the appropriate tool in Data 360.