promptry eval
Version updated for https://github.com/bihanikeshav/promptry to version v0.5.0.
- This action is used across all versions by 0 repositories.
Action Type
This is a Composite action.
Go to the GitHub Marketplace to find the latest changes.
Action Summary
Summary:
The promptry GitHub Action automates regression testing and evaluation of large language model (LLM) pipelines by tracking prompt versions, running evaluation suites, and detecting regressions or quality drifts. It helps developers identify changes in prompt performance, analyze root causes, and compare models against historical baselines, ensuring consistent and high-quality outputs. Key features include prompt versioning, statistical model comparison, cost tracking, and support for a web dashboard.
What’s Changed
What’s new in v0.5.0
New features
promptry doctor— Health check command that verifies Python version, storage, optional dependencies, embedding model, and LLM judge configuration- GitHub Action — One-line CI eval runs with
suite,module, and optionalcompareinputs - Schema migration system — Safe future database upgrades without losing history
- Pagination — All list endpoints now support
offset/limitparameters - Richer
inittemplates —promptry initnow scaffolds 3 example suites: smoke-test, rag-qa, classification
Performance
- Batch queries — Dashboard and model comparison no longer fire N+1 queries
- Missing index — Added index on
eval_runs.model_version
Bug fixes
- Storage abstraction leaks fixed — Deleted ~150 lines of duplicated raw SQL from CLI and MCP server
- Consistent storage access — CLI uses
get_storage()singleton everywhere
Documentation & website
- Full docs page with sidebar navigation
- Dashboard screenshot gallery in hero
- Competitive comparison table (vs Promptfoo, DeepEval, LangSmith)
- Multi-column footer, consistent nav across all pages
- README trimmed from 1031 to 110 lines
Testing
- 42 new tests (252 to 294) covering cost-report, config, custom templates, compare CLI, doctor, pagination, migrations
Install
pip install promptry # core
pip install promptry[semantic] # + semantic assertions
pip install promptry[dashboard] # + web dashboard
pip install promptry[semantic,dashboard] # everything