Hallucinations (Ungrounded Additions)
In the context of AI development, hallucinations refer to ungrounded additions or erroneous information generated by the system without a factual basis. These hallucinations can greatly impact the reliability and accuracy of AI applications.
Whether it's for medical diagnostics, legal research, or any critical application, ensuring the AI remains grounded in reality is paramount. This is especially vital to prevent the dissemination of false or misleading information, guaranteeing the users trust the system's outputs.
Example Scenario: Enterprise Search for Financial Institutions
In the complex landscape of financial institutions, accurate and efficient information retrieval is crucial for decision-making processes, compliance adherence, and customer service. Financial institutions rely on enterprise search solutions powered by AI to swiftly access vast amounts of data scattered across various databases, documents, and systems.
However, the presence of hallucinations in the search results could lead to critical errors. Imagine a scenario where a banking executive uses an enterprise search system to find specific financial regulations. If the AI algorithms produce hallucinated search results, displaying outdated or non-existent regulations, it could lead to incorrect interpretations.
Compliance decisions based on these erroneous results might violate regulations, leading to legal consequences, financial penalties, and potential reputational damages.
Guardrail's Firewall Factuality Detectors
To prevent such risks, it is essential to mitigate hallucinations risks from enterprise search algorithms used by financial institutions.
Our Firewall provides several out-of-the-box detectors to detect and mitigate ungrounded additions. Our Factuality Detector detects factual errors in knowledge-based QA. Does so by:
1. Factual Consistency
2. Relevance
3. Leveraging Tools for Verification and Correction
1. Extracting claims for LLM response
2. Generate queries to validate claims
3. Uses tools (Google Search, Internal KB)
4. Collect evidences from tools
5. Verify claims
By providing reliable and precise information, these optimized enterprise search systems empower financial institutions to make informed decisions, maintain compliance, and enhance overall operational efficiency.
from guardrail.firewall.output_detectors import FactualConsistency, FactualityTool, Relevance
firewall = Firewall(no_defaults=True)
output_detectors = [FactualConsistency(threshold=0.7),
Relevance(),
FactualityTool(),]
sanitized_prompt = "Context: Ruth Porat, President and Chief Investment Officer; CFO said: \n"\
"The fundamental strength of our business was apparent again in Q3, with $77 billion in revenue, up 11% year over year, \n"\
"driven by meaningful growth in Search and YouTube, and momentum in Cloud. We continue to focus on judicious capital allocation \n"\
"to deliver sustainable financial value.\n\n"\
"User input: How did Google do in Q3?"
response_text = "Google revenue is up 15% year over year from 2023 Q3 earnings report"
sanitized_response, valid_results, risk_score = firewall.scan_output(sanitized_prompt, response_text, output_detectors)
Example output of Factuality Detector breaking down evidences, claim_level_factuality, response_level_factuality, sources used and more with an online tool (Google) .
{
"average_claim_level_factuality": 0.0,
"average_response_level_factuality": 0.0,
"detailed_information": [
{
"prompt": "Introduce Percy Liang",
"response": "Percy Liang is a professor at Harvard",
"category": "kbqa",
"search_type": "online",
"claims": [
{
"claim": "Percy Liang is a professor at Harvard"
}
],
"queries": [
[
"Is Percy Liang a professor at Harvard?",
"Percy Liang Harvard"
]
],
"evidences": [
{
"evidence": [
"Percy Liang is an Associate Professor of Computer Science at Stanford University (B.S. from MIT, 2004; Ph. D. from UC Berkeley, 2011) and the director of the Center for Research on Foundation Models.",
"I publish mostly in machine learning (ICML, NeurIPS) and natural language processing (ACL, NAACL, EMNLP) conferences. I am drawn to simple things, want to ...",
"Percy Liang is an Associate Professor of Computer Science at Stanford University (B.S. from MIT, 2004; Ph.D. from UC Berkeley, 2011). His two research goals are ...",
"Missing: Harvard | Show results with:Harvard",
"Our aim is to produce methods that are provably robust to such deviations. In this talk, I will provide an overview of the work my group has done on this topic ...",
"AI to Accelerate the Life Sciences/Drug Discovery \u00b7 Alternative Models to Trials Using Real-World Data \u00b7 Driving Gene-by-Environment Discovery in Biobanked Data."
],
"source": [
"None",
"https://cs.stanford.edu/~pliang/",
"https://profiles.stanford.edu/percy-liang",
"https://profiles.stanford.edu/percy-liang",
"https://mlfoundations.org/talk/percy/",
"https://dbmi.hms.harvard.edu/research-areas/artificial-intelligence"
]
}
],
"claim_level_factuality": [
{
"reasoning": "The given text states that Percy Liang is a professor at Harvard. However, the provided evidences clearly state that Percy Liang is an Associate Professor of Computer Science at Stanford University. The evidences consistently mention Stanford University as his affiliation, while there is no mention of Harvard University. Therefore, it can be concluded that the given text is non-factual.",
"error": "The given text is non-factual.",
"correction": "Percy Liang is an Associate Professor of Computer Science at Stanford University.",
"factuality": false,
"claim": "Percy Liang is a professor at Harvard"
}
],
"response_level_factuality": false
}
]
}