RAG evaluation is hard: answers can sound fluent but hallucinate (not grounded in context). LangSmith provides automated metrics: faithfulness (uses context), relevancy (answers the question), and context precision (retrieved documents are relevant). This post covers setup and interpretation.
Setting Up LangSmith
from langsmith import Client
client = Client()
# Log runs to LangSmith @client.traced() def rag_chain(question): context = retriever.get_relevant_documents(question) answer = llm.generate(question, context) return {"answer": answer, "context": context}
result = rag_chain("What is transformers?")
Evaluating Faithfulness
from langsmith.evaluation import evaluate_string
# Evaluate if answer is grounded in context def faithfulness_evaluator(run_output, context): answer = run_output["answer"] context_text = "\n".join([doc.page_content for doc in context]) # Use LLM to check if answer is supported by context return {"score": check_if_grounded(answer, context_text)}
Conclusion
LangSmith provides visibility into RAG quality. Faithfulness, relevancy, and context precision metrics help identify failure modes. Automated evaluation enables rapid iteration and production monitoring. Next: ReAct agents—how to give LLMs tools and reasoning loops.