RAG evaluation is hard: answers can sound fluent but hallucinate (not grounded in context). LangSmith provides automated metrics: faithfulness (uses context), relevancy (answers the question), and context precision (retrieved documents are relevant). This post covers setup and interpretation.
Setting Up LangSmith
Code Snippet
from langsmith import Client
client = Client()
# Log runs to LangSmith
@client.traced()
def rag_chain(question):
context = retriever.get_relevant_documents(question)
answer = llm.generate(question, context)
return {"answer": answer, "context": context}
result = rag_chain("What is transformers?")
Evaluating Faithfulness
Code Snippet
from langsmith.evaluation import evaluate_string
# Evaluate if answer is grounded in context
def faithfulness_evaluator(run_output, context):
answer = run_output["answer"]
context_text = "\n".join([doc.page_content for doc in context])
# Use LLM to check if answer is supported by context
return {"score": check_if_grounded(answer, context_text)}
Conclusion
LangSmith provides visibility into RAG quality. Faithfulness, relevancy, and context precision metrics help identify failure modes. Automated evaluation enables rapid iteration and production monitoring. Next: ReAct agents—how to give LLMs tools and reasoning loops.