Skip to main content

​
Pipeline

After the conversation closes, the full transcript goes to Kimi (Moonshot AI). Kimi returns a score between 0 and 1. Scores at or above KIMI_PASS_THRESHOLD trigger releasePayout. Scores below trigger a re-prompt. The conversational LLM and Kimi are separate systems:
SystemInputOutput
Conversational LLMAdvertiser brief + group context + conversation historyNext agent turn
KimiFull transcript + brief + group contextQuality score (0–1)

​
What Kimi evaluates

Response entropy
Did the user give varied, specific answers or pattern responses? Single-word replies, identical response structure across turns, and copy-paste answers score low. Responses with specific terminology, examples, or reasoning score higher. Contextual coherence
Do the answers make sense given the questions asked? Does the user’s stated knowledge align with the group they’re joining? Responses that ignore the question or answer a different question score low. Engagement trajectory
Did the user engage more on the follow-up, or give the same response quality across all turns? Humans typically elaborate when probed. Uniform response quality across turns regardless of question depth is a bot signal. Timing
Response times are logged per turn. Responses under 1–2 seconds across all turns score lower. Human response times show natural variation.

​
Thresholds

ScoreOutcome
≥ 0.7 (default)Pass — releasePayout fires, user admitted
0.4–0.69Re-prompt — user gets one attempt with a different question
< 0.4Fail on re-prompt — user denied, 24h cooldown
KIMI_PASS_THRESHOLD is configurable per campaign. Higher thresholds produce fewer passing verifications with higher average quality. Lower thresholds pass more verifications with looser quality filtering.

​
Kimi outage handling

If Kimi is unavailable, the agent server retries up to 4 times with exponential backoff. If all 4 retries fail:
  • Verification marked SCORING_UNAVAILABLE
  • User admitted to group
  • No payout released
  • No cooldown applied
  • Advertiser is not billed

​
What Kimi does not evaluate

Kimi does not assess whether answers are factually correct. For agent acquisition campaigns (no correct answer), Kimi scores engagement quality only — whether the response is genuine human engagement, not whether the user picked a specific option. For RLHF campaigns, Kimi scores whether the response reflects genuine reasoning, not whether the preference label matches a ground-truth answer.