Automated Testing
for AI Prompts
for
AI Prompts
Now integrating with:
How it works
1.
Write your testsUse Flowprompter Studio to define your project's requirements as a suite of evaluations.
2.
Run your test suiteUse the Flowprompter CLI to trigger a test run.
3.
Evaluate completionsCompletions are sent to Flowprompter and evaluated swiftly in parallel.
4.
Analyze realtime resultsSee the results appear in the Test Explorer in realtime. Dig into failed and successful tests and track regressions.
Prompt fast and don't break things
Test Driven Development enhances developer productivity by ensuring functionality doesn't break. However, its application for Large Language Models is challenging due to their non-deterministic nature, necessitating trial-and-error processes.
Flowprompter solves this problem by utilizing a combination ML, LLM and statistical data analysis techniques to evaluate completions based on your given evaluation criteria.
Prompt Evaluation
Flowprompter evaluates your completions according to your evaluations, taking fuzzy output and making it testable.
Iterate Faster
Evaluations act as "fitting functions" enabling the engineer to quickly iterate towards prompt chains that work more effectively
Fearless Migration
Flowprompter enables engineers to save money and avoid vendor lock in by migrating models without affecting your user experience.
Test Studio
Author your evaluations using Flowprompter Studio or utilize our Flowprompter DSL and test in your language
Github Integration
Run your tests on every prompt change, or model change along side your other continuous integration tools.
Track Performance
Flowprompter Explorer allows engineers to drill down into exactly what is going on during their LLM calls and track their evaluations historically.
Secure and Private
Your code never leaves your machine. Flowprompter only send through the data it needs to function and all streams are encrypted.
Monitor Production
With production and feedback integration, Flowprompter allows you to test evaluations against real user data and stay on top of your user experience.
What people are saying
Daniel Mechea
Founder of getmagnet.ai
Pricing for any team with Flowprompter Cloud
Hobbyist
FREE!
Startup (most popular)
$149.00
$99.95
/ month
* VAT may be applicable depending on your location.
Frequently asked questions
- What is Flowprompter?
- Flowprompter is a set of tools that enables automated prompt testing of large language models. When using Flowprompter you can interact with the Evaluation Studio to author prompt evaluations and utilize the Test Explorer to analyse historical prompts versus current prompts to connect LLM performance and quality to your prompt architecture. You can also seamlessly integrate your code repository to enable automatic test execution.
- Why should I use Flowprompter?
- Developing prompts for LLMs is just difficult because of the fuzzy nature of the output. Discovering the optimal chain of prompts to have the desired outcome is often based on trial and error. Whats more you it can be difficult to know how your prompt architecture affects your token spend with LLM providers. FLowPrompter helps keeps tabs of your spending and makes sure your prompts changes are leading to better outcomes.
- How do I write my tests?
- You can write Evaluations as code using Flowprompter's Evaluation Specification Language (ESL)or choose to use Flowprompter Studio to define tests using an AI assisted interface.
- What is the Evaluation Specification Language?
- Evaluations can be specified using Flowprompters Evaluation Specification Language (ESL). It is a JSON based schema driven way to define how prompt completions should be evaluated