> ## Documentation Index
> Fetch the complete documentation index at: https://docs.canvas-protocol.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Scoring

> How Kimi evaluates conversation transcripts.

## Pipeline

After the conversation closes, the full transcript goes to Kimi (Moonshot AI). Kimi returns a score between 0 and 1. Scores at or above `KIMI_PASS_THRESHOLD` trigger `releasePayout`. Scores below trigger a re-prompt.

The conversational LLM and Kimi are separate systems:

| System             | Input                                                   | Output              |
| ------------------ | ------------------------------------------------------- | ------------------- |
| Conversational LLM | Advertiser brief + group context + conversation history | Next agent turn     |
| Kimi               | Full transcript + brief + group context                 | Quality score (0–1) |

## What Kimi evaluates

**Response entropy**\
Did the user give varied, specific answers or pattern responses? Single-word replies, identical response structure across turns, and copy-paste answers score low. Responses with specific terminology, examples, or reasoning score higher.

**Contextual coherence**\
Do the answers make sense given the questions asked? Does the user's stated knowledge align with the group they're joining? Responses that ignore the question or answer a different question score low.

**Engagement trajectory**\
Did the user engage more on the follow-up, or give the same response quality across all turns? Humans typically elaborate when probed. Uniform response quality across turns regardless of question depth is a bot signal.

**Timing**\
Response times are logged per turn. Responses under 1–2 seconds across all turns score lower. Human response times show natural variation.

## Thresholds

| Score           | Outcome                                                     |
| --------------- | ----------------------------------------------------------- |
| ≥ 0.7 (default) | Pass — `releasePayout` fires, user admitted                 |
| 0.4–0.69        | Re-prompt — user gets one attempt with a different question |
| \< 0.4          | Fail on re-prompt — user denied, 24h cooldown               |

`KIMI_PASS_THRESHOLD` is configurable per campaign. Higher thresholds produce fewer passing verifications with higher average quality. Lower thresholds pass more verifications with looser quality filtering.

## Kimi outage handling

If Kimi is unavailable, the agent server retries up to 4 times with exponential backoff. If all 4 retries fail:

* Verification marked `SCORING_UNAVAILABLE`
* User admitted to group
* No payout released
* No cooldown applied
* Advertiser is not billed

## What Kimi does not evaluate

Kimi does not assess whether answers are factually correct. For agent acquisition campaigns (no correct answer), Kimi scores engagement quality only — whether the response is genuine human engagement, not whether the user picked a specific option. For RLHF campaigns, Kimi scores whether the response reflects genuine reasoning, not whether the preference label matches a ground-truth answer.