AI Interviews with Thousands of Participants: Scaling and Observability

An AI interviewer conducting a single conversation is quickly built. A system conducting hundreds or thousands of conversations at the same time – each with fluid responses, follow-up questions, and clean analysis – is a different task. And it rarely fails where you would expect.

The most common assumption goes: “The AI model does the work, the rest is a normal web application.” In practice, the challenge lies elsewhere: in load spikes, in the capacity limits of model providers – and above all in the question of whether you can even see what the system is doing. This post covers what matters when scaling, and which metrics you should measure from day one.

Why Interviews Scale Differently Than Websites

A normal web application answers short requests: request in, response out milliseconds later. An AI interview behaves differently:

• Long sessions: A conversation lasts 5–15 minutes and keeps an open connection to the server the whole time, streaming responses in real time.
• Everything arrives at once: After an email invitation to 10,000 recipients, a substantial share clicks within the first hour. The system has to go from zero to hundreds of parallel conversations within minutes.
• The bottleneck is the model, not the server: Every participant answer triggers a call to the AI model – carrying the entire conversation so far as context. The scarce resource is therefore not computing power, but the token quota the model provider processes per minute.

Expert Insight: Capacity here is not planned in “requests per second” but in tokens per minute. With a few hundred parallel conversations, you quickly exceed the standard limits of common API tiers – something to know before the first big wave goes out.

The Core Principles of Scaling

Three principles have proven themselves for interview systems:

Conversations must not be tied to a single server. If conversation state lives outside the individual server, any instance can handle any turn of any conversation. Only that makes automatic scaling under load possible – and ensures that a dropped connection or a software update never ends a running interview.

Use caching. Not everything has to be processed anew on every conversation turn. Well-designed caching lowers the cost per conversation and shortens response times – an effect that multiplies across thousands of interviews.

Decouple analysis from the conversation. A thousand interviews produce millions of words of transcript. Quality checks, bot detection, theme extraction, and sentiment analysis therefore run as a separate background process – they must never slow down the live conversations. We described how automated analysis works here.

Observability: You Can Only Scale What You Can See

The underestimated part of scaling is not the infrastructure – it is visibility. A classic server reports CPU and memory. An AI system needs its own metrics, because its bottlenecks lie elsewhere. These four have proven to be the most important:

1. Time-to-First-Token (TTFT)

The time from the participant submitting their answer to the first word of the AI’s reply. This is the number participants feel directly: if the screen stays blank for two or three seconds, the drop-off rate rises measurably. What matters:

• Measure per conversation turn, not per interview – TTFT grows with the conversation context, and that growth is exactly what you want to see.
• Percentiles, not averages: The median may sit at 800 milliseconds while every twentieth conversation waits five seconds. The average hides precisely the participants who abandon.
• A creeping TTFT increase over days is an early warning – it often signals a provider quota filling up, long before actual errors appear.

2. Token Consumption per Turn and per Interview

Tokens are to AI systems what CPU time is to classic applications: the real cost unit. Every model call should therefore be logged with input tokens, output tokens, and cache usage. Three analyses follow from this:

• Cost per interview: Only with this number can you seriously calculate what a study with 10,000 participants costs – and whether shortening the guide pays off.
• Cache hit rate: If it suddenly drops, an unintentional configuration change is usually behind it – and every turn costs a multiple. Without monitoring, this only shows up on the monthly invoice.
• Outlier detection: An interview consuming five times the usual tokens is a warning sign – an endless probing loop, a participant deliberately dragging the system into long discussions, or a prompt-injection attempt.

3. Drop-Off Points in the Conversation

Where exactly do participants leave? This metric connects technology and content: if drop-offs are spread evenly and correlate with high TTFT, it is an infrastructure problem. If they cluster at one particular question, the guide is the problem – too long, too personal, confusingly worded. With 20 interviews you see this by reading transcripts; with 20,000 you only see it in a dashboard.

4. Structured Logs per Conversation Turn

Every turn should be logged as one searchable entry: session ID, timestamp, TTFT, token counts, tools called, error code. It sounds mundane, but it decides how debugging goes: when a participant reports “the survey suddenly froze,” you must be able to reconstruct that specific conversation in seconds – with hundreds of parallel sessions, that only works with structured logs filterable by session.

Expert Insight: Build the monitoring before the first big wave, not after. Metrics like TTFT and token consumption cost a few lines of code if you record them from the start – but reconstructing them retroactively for past interviews is impossible.

The Path to Scale: Test First, Then Stagger

A three-stage approach has proven itself:

1. Pilot with 20–50 participants: Validate the guide and actually read the transcripts in full. Does the AI probe in the right places? Every weakness in the guide later multiplies with the number of participants. The pilot also provides the baseline for monitoring: typical TTFT, typical token consumption per interview.
2. Load test before the big wave: Run simulated participants against the real system – including real model calls. Only then do quota limits and connection limits surface that stay invisible in normal operation.
3. Staggered send-out: Send invitations in cohorts rather than one blast, for instance four batches of 2,500 an hour apart. That flattens the load peak – and between cohorts, the dashboard shows whether TTFT and drop-off rate stay stable before the next wave starts.

Expert Insight: Never scale your first draft. The 50 pilot interviews cost a day – a weak guide at 10,000 participants costs the entire study.

Conclusion

Scaling AI interviews is decided not in the AI model, but in the architecture around it: conversation state held independently of any single server; well-designed caching; analysis running decoupled in the background. And above all: observability from day one – time-to-first-token in percentiles, token consumption per interview, drop-off points in the conversation, and structured logs per turn.

Because with a thousand parallel conversations, nobody is reading along anymore. What you don’t measure, you don’t see – and what you don’t see, doesn’t scale.

Related Articles

• AI Interview Analysis: Automated from Transcript to Insight
• Agentic AI in Practice: How We Run AI Agents in Production at Feedbk
• Bot Protection in AI Surveys: Keeping Your Data Clean

Try It Yourself

Create an AI-moderated survey and experience the interview yourself.

Try the Demo