Skip to content
Flowprompter
FeaturesPricingFAQSign Up

Automated Testing
for

AI Prompts

Take the trial and error out of developing Machine Learning and Large Language Model applications.

Now integrating with:

Screenshot of a ListPage in Saas UI Pro

How it works

1.

Write your tests

Use Flowprompter Studio to define your project's requirements as a suite of evaluations.

2.

Run your test suite

Use the Flowprompter CLI to trigger a test run.

3.

Evaluate completions

Completions are sent to Flowprompter and evaluated swiftly in parallel.

4.

Analyze realtime results

See the results appear in the Test Explorer in realtime. Dig into failed and successful tests and track regressions.

Prompt fast and don't break things

Test Driven Development enhances developer productivity by ensuring functionality doesn't break. However, its application for Large Language Models is challenging due to their non-deterministic nature, necessitating trial-and-error processes.

Flowprompter solves this problem by utilizing a combination ML, LLM and statistical data analysis techniques to evaluate completions based on your given evaluation criteria.

Prompt Evaluation

Flowprompter evaluates your completions according to your evaluations, taking fuzzy output and making it testable.

Iterate Faster

Evaluations act as "fitting functions" enabling the engineer to quickly iterate towards prompt chains that work more effectively

Fearless Migration

Flowprompter enables engineers to save money and avoid vendor lock in by migrating models without affecting your user experience.

Test Studio

Author your evaluations using Flowprompter Studio or utilize our Flowprompter DSL and test in your language

Github Integration

Run your tests on every prompt change, or model change along side your other continuous integration tools.

Track Performance

Flowprompter Explorer allows engineers to drill down into exactly what is going on during their LLM calls and track their evaluations historically.

Secure and Private

Your code never leaves your machine. Flowprompter only send through the data it needs to function and all streams are encrypted.

Monitor Production

With production and feedback integration, Flowprompter allows you to test evaluations against real user data and stay on top of your user experience.

What people are saying

Daniel Mechea

Founder of getmagnet.ai

"We probably cut our prompt engineering time in half just by using Flowprompter"

Pricing for any team with Flowprompter Cloud

A suite of tools for testing and evaluating large language models.

Hobbyist

Perfect for trying out Flowprompter

FREE!

1 Developer

300 Free Evaluations per Month

Evaluation Specification Language

Evaluation Studio

Local CLI

Basic Analytics

Startup (most popular)

Complete suite for small production teams

$149.00

$99.95

/ month

Everything in the Hobbyist plan, plus:

Upto 5 Developers

2,000 Free Evaluations per Month

Github Actions Integration

Evaluation Studio

Test Explorer

Paralelization

Comprehensive Analytics

Enterprise

Production focussed enterprise features for large organizations.

$699.00

$499.00

/ month

Everything in the Startup plan, plus:

Unlimited Developers

Unlimited Evaluations

Production Data Integration

Active Telemetry & Feedback

24h Active Support

* VAT may be applicable depending on your location.

Frequently asked questions

What is Flowprompter?
Flowprompter is a set of tools that enables automated prompt testing of large language models. When using Flowprompter you can interact with the Evaluation Studio to author prompt evaluations and utilize the Test Explorer to analyse historical prompts versus current prompts to connect LLM performance and quality to your prompt architecture. You can also seamlessly integrate your code repository to enable automatic test execution.
Why should I use Flowprompter?
Developing prompts for LLMs is just difficult because of the fuzzy nature of the output. Discovering the optimal chain of prompts to have the desired outcome is often based on trial and error. Whats more you it can be difficult to know how your prompt architecture affects your token spend with LLM providers. FLowPrompter helps keeps tabs of your spending and makes sure your prompts changes are leading to better outcomes.
How do I write my tests?
You can write Evaluations as code using Flowprompter's Evaluation Specification Language (ESL)or choose to use Flowprompter Studio to define tests using an AI assisted interface.
What is the Evaluation Specification Language?
Evaluations can be specified using Flowprompters Evaluation Specification Language (ESL). It is a JSON based schema driven way to define how prompt completions should be evaluated

Automated Testing for Large Language Models

Copyright © 2023 Flowprompter