Run LangSmith Evals with Pytest and Vitest
💡Bring LLM evaluations into the Pytest or Vitest/Jest workflows your team already uses.
⚡ 30-Second TL;DR
What Changed
Adds a Pytest integration for LangSmith evaluations
Why It Matters
Using standard test runners can make LLM evaluation easier to adopt in existing engineering pipelines. Teams may also find it simpler to run evaluations alongside automated application tests.
What To Do Next
Add one LangSmith evaluation to your existing Pytest or Vitest/Jest suite and run it in CI with your application tests.
Key Points
- •Adds a Pytest integration for LangSmith evaluations
- •Adds Vitest/Jest integrations for JavaScript and TypeScript testing workflows
- •Provides a new way to run evals through established test frameworks
🧠 Deep Insight
Background and context from public sources — not the original article. 11 sources cited.
🔑 Enhanced Key Takeaways
- •The integrations were launched in beta within version 0.3.0 of the LangSmith Python and TypeScript SDKs.
- •These tools enable CI/CD pipeline integration, allowing developers to trigger build failures based on specific LLM performance thresholds.
- •The framework supports heterogeneous evaluation logic, enabling developers to apply distinct test suites to different subsets of a dataset.
- •Integration allows for the evaluation of complex 'Deep Agent' trajectories, including state artifacts and multi-turn interaction analysis.
- •Test results, including full stack traces and input-output pairs, are automatically synced to the LangSmith platform for centralized debugging.
📊 Competitor Analysis▸ Show
| Feature | LangSmith (Pytest/Vitest) | Future AGI (FAGI) |
|---|---|---|
| Primary Focus | LangChain ecosystem integration | Multi-stack dashboard (OpenAI/CrewAI) |
| Testing Workflow | Native unit/integration test framework | Unified cross-platform dashboard |
| Pricing | Usage-based (LangSmith platform) | Enterprise/SaaS tiers |
| Benchmarks | Integrated CI/CD performance tracking | Cross-framework comparative analytics |
🛠️ Technical Deep Dive
- SDK Versioning: Requires LangSmith Python/TypeScript SDK version 0.3.0 or higher.
- Execution Flow: Leverages native test runners (Pytest/Vitest) to execute evaluation logic, with results serialized and pushed to the LangSmith API via SDK hooks.
- Assertion Handling: Supports standard language-specific assertion libraries (e.g., Python assert, Jest expect) to trigger failure states in CI/CD environments.
- Data Handling: Enables mapping of dataset examples to specific test cases, allowing for dynamic injection of inputs and expected outputs into the test runner context.
- Observability: Automatically captures trace metadata, including latency, token usage, and custom evaluation metrics, linking them to the specific test execution ID.
🔮 Future ImplicationsAI analysis grounded in cited sources
⏳ Timeline
📎 Sources (11)
Factual claims are grounded in the sources below. Forward-looking analysis is AI-generated interpretation.
Weekly AI Recap
Read this week's curated digest of top AI events →
👉Related Updates
AI-curated news aggregator. All content rights belong to original publishers.
Original source: LangChain Blog ↗
This is a summary, not the original. Read the source, or get the weekly briefing.
Weekly AI briefing
One email a week. Unsubscribe anytime.