deepeval
The LLM Evaluation Framework
Written mainly in Python. Released under the Apache-2.0 licence.
Free plan
github.com/confident-ai/deepeval
What each tool is actually for, described plainly. No ratings, no scores, no invented benchmarks.
Evaluation and observability: Knowing whether the thing works and what it did: test harnesses, scoring, tracing and cost tracking.
The LLM Evaluation Framework
Written mainly in Python. Released under the Apache-2.0 licence.
Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.
Written mainly in Python.
Open source LLM observability platform. One line of code to monitor, evaluate, and experiment. YC W23
Written mainly in TypeScript. Released under the Apache-2.0 licence.
Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.
Written mainly in TypeScript.
A framework for few-shot evaluation of language models.
Written mainly in Python. Released under the MIT licence.
AI Observability & Evaluation
Written mainly in Python.
Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic.
Written mainly in TypeScript. Released under the MIT licence.
Supercharge Your LLM Application Evaluations
Written mainly in Python. Released under the Apache-2.0 licence.