Back to blog

AI chatbots are confident even when they're wrong. Ask ChatGPT a question about your company's vacation policy and it'll give you a plausible-sounding answer, even though it has never seen your employee handbook. There's no way for the reader to tell whether the answer is grounded in real data or fabricated.

This is the hallucination problem, and it's the single biggest barrier to deploying AI in professional settings.

The problem with AI confidence

Language models don't know what they don't know. They generate the most probable next token based on patterns, not facts. A model will state "Employees receive 15 days of PTO" with the same fluency whether it read that in your handbook or invented it.

For internal tools, customer support, and knowledge bases, this is unacceptable. Someone asking about their benefits, your product specs, or a compliance policy needs to know: is this answer real, or is the AI guessing?

How confidence scoring works

Ask360 uses a four-tier confidence system that combines multiple independent signals to assess how well the answer is supported by your documents.

The four tiers

Verified (green): The answer is directly supported by your documents. The AI found passages that closely match the specific question asked. Trust this answer.

Grounded (blue): Good document support. Relevant passages substantiate the response, though the match may be topical rather than exact. Reliable for most purposes.

Mixed (amber): Partial support. Some parts of the answer come from your documents, but the AI may have filled in gaps. Visitors should verify important details.

AI Generated (gray): Limited document support. The topic may not be covered in your knowledge base. The AI is mostly relying on its general knowledge, not your documents.

Why one signal isn't enough

Early versions used a single score: cross-encoder relevance. A neural model reads the question and passage together and scores how directly the passage answers the question. This works brilliantly for specific factoid questions:

That last query is about vacation policy. The right documents were retrieved (the PTO handbook section), but the cross-encoder scored them near zero because the model was trained on factoid Q&A pairs. It learned "Does this passage ANSWER this question?" not "Is this passage ABOUT this topic?"

Topical questions like "What is the vacation policy?" don't have a single-sentence answer, so the cross-encoder rejects them.

Multi-signal scoring

The fix: combine two complementary signals.

Cross-encoder relevance: Excellent for factoid questions where a specific passage directly answers the question. Poor for broad topical queries.

Vector similarity: Measures how semantically close the question is to the best matching documents. Excellent for topical alignment ("Is this question within the knowledge base's coverage area?"). Less precise for specific factoid matching.

By allowing either signal to promote the confidence tier, both types of questions are handled correctly:

• "How many PTO days?" has high cross-encoder (0.9994), so it's Verified regardless of vector score
• "What is the vacation policy?" has low cross-encoder (0.0002) but decent vector similarity (0.38), so it's Grounded via the vector signal
• "How do quantum computers work?" has both signals low (CE: 0.00001, vector: 0.14), so it correctly shows as AI Generated

What the visitor sees

Every response in the chat includes:

1. Confidence badge at the top of the response, color-coded with a percentage
2. Source citations at the bottom, showing which documents were used
3. The answer itself, which the AI generates from the retrieved context

The badge is not decoration. It's a signal to the visitor: "You can trust this" or "Verify this elsewhere." In professional settings (HR, legal, compliance, support), this is the difference between a useful tool and a liability.

Why this matters for your business

Customer support: Visitors trust answers more when they can see the confidence level and source documents. Support tickets decrease when people can self-verify.

Internal knowledge bases: Employees know when to trust the AI and when to escalate to a human. No more acting on hallucinated policy interpretations.

Compliance and legal: Auditors can see which documents informed each answer. The confidence badge creates an accountability trail.

Onboarding and training: New hires can learn from the AI but know when they're getting verified information versus general guidance.

Try it

Ask360 includes confidence badges on every response, with source citations. Try the live demos to see how different questions produce different confidence levels, or sign up free to test with your own documents.