What evaluation means on Finn
Finn does not have an automated evaluation suite. There is no scored test-set runner, no regression harness, and no API endpoint that grades an agent. Evaluation today means placing test calls yourself, reading the transcript and the extracted fields, and fixing what you find before the agent takes a real caller. A programmatic evaluation surface is not yet settled.
That makes the loop manual, so the discipline matters more. The sections below are the order to work in.
Before the first test call
Check these in the dashboard. Each one fails in a way that looks like a model problem but is not.
| Check | Where | Failure if skipped |
|---|---|---|
| Welcome message has text, or is set to dynamic | Edit Finn → Welcome Message | Agent picks up and says nothing |
| Voice assigned, region matches | Edit Finn → Voice | Silence or wrong-sounding output |
| Knowledge base shows Indexed | Knowledge Base tab | Agent answers "I don't know" to documented facts |
| Post-call analysis fields defined | Post-call analysis config | Fields stay blank for every call in this test round |
| Trial credit remaining | Settings → Plan and billing → Usage | Test call never dials |
Post-call analysis is the one that bites hardest. Fields only populate on calls that complete after the field was added. If you add a field mid-test, every call you already placed stays blank forever, and no re-run backfills them. Define your fields first, then test. See post-call-analysis.
Placing the test call
Enter the number with a country code. US is +15551234567, India is +919876543210. The platform shows a "tested format: …" preview before it triggers, so read that preview rather than trusting the number you typed.
A test call that never arrives is usually not the agent:
| Cause | Check |
|---|---|
| Number on the global suppression list | Settings → Compliance → Do Not Call List |
| Carrier rejection | Some international carriers block new numbers. Try a different from-number. |
| Spam filter on your handset | Apple and Google filters drop calls silently. Try another from-number. |
| From-number caller ID not registered (US) | Settings → Phone numbers → [Number] → Registration Status |
Full symptom list in troubleshooting.
Reading the result
Listen to the recording. Do not reason from the transcript alone, and do not guess from the summary. Find the first turn that went wrong, not the worst one. Everything after the first bad turn is downstream noise.
Then open the Data Extractor Sidebar for that call:
| Tab | What it tells you |
|---|---|
| Knowledge | Which document snippets were retrieved. Wrong snippets mean a knowledge base problem, not a prompt problem. |
| Extraction Status | Whether post-call fields ran and whether extraction errored. |
If the retrieved snippets were correct and the agent still answered wrong, the bug is in the prompt. If the snippets were wrong or empty, fix the knowledge base first. See agents-knowledge-base.
Fixing and re-testing
The loop from the prompting guide applies directly:
- Find the first weird turn.
- Add a rule covering that exact scenario, plus one example turn.
- Re-test on the same number.
- If it is still wrong, the cause is upstream in the workflow, the audience, or the knowledge base.
Most "the model is dumb" reports are missing-rule bugs. Details in agents-prompting.
What a test call cannot tell you
Be honest about the limits, because a clean test call is weak evidence.
| Signal | Why it misleads |
|---|---|
| Sentiment | An indicator, not a verdict. It misclassifies. If it is consistently wrong for your use case, add a custom post-call field with a specific question and use that instead. |
| Very short calls | Calls under five seconds are treated as no-answer by some metrics and will not appear where you expect. Check the full call log. |
| One clean pass | You tested the happy path. Test the angry caller, the ambiguous request, and the caller who says "remove me" and expects the call to end. |
| Your own reactions | You know the script. A real caller does not, and will interrupt, mumble, and change topic. |
Test the negative rules explicitly. If the prompt says never quote a price, ask for a price. If it says never book a Sunday, ask for Sunday. A rule that has never been provoked has never been verified.
After you go live
Testing does not stop at launch. Watch the first real calls in call-logs and the deployment health panel in monitoring. The failure modes that only appear at volume, such as hangups from an unregistered caller ID or an exhausted audience, will not show up in a single test call.