Dev blog · September 2026

Building IR Simulator: a live video call with a company's investor relations team, from transcripts to talking avatar

IR Simulator lets you pick a public company, choose five topics, and have a real-time voice and video conversation with a simulated Investor Relations representative who knows that company's earnings calls and filings. As she talks, slides appear beside her. This post is the engineering story: how the transcript data is queried, how the voice agent is built and hosted, how the avatar is wired in, and what it took to make the rep answer quickly. Code samples are trimmed from the real repo.

What we built

The product has three phases. Setup: the user names a company (with autocomplete over every ticker we have transcripts for) and an “assistant” character reaches out to IR while we prepare. Agenda: the IR rep “emails back” with fifteen company-specific topics and asks the user to pick five. Call: a LiveKit room opens, a Lemon Slice avatar joins, and the rep greets the user by name and starts taking questions. A slide panel updates as she speaks, and a transcript scrolls under the video. Calls are capped at ten minutes.

Everything the rep says is grounded: a per-company briefing built from the company's recent earnings calls, its latest annual and quarterly reports, and reported financials. When she is asked something outside the briefing, a lookup runs in the background and she folds the answer in a few seconds later.

The stack in one diagram

browser ──WebRTC──▶ LiveKit Cloud room ◀──── Python agent (LiveKit Agents, hosted on LiveKit Cloud)
   │                     ▲     ▲                 │  STT: Deepgram  ·  LLM: Claude via OpenRouter  ·  TTS: Deepgram Aura-2
   │                     │     └── Lemon Slice avatar participant (video + lip-synced audio)
   │                     └──── "ir.slides" text stream: {type:"show_slide", slide_id}
   │
   └──HTTPS──▶ Next.js on Vercel ──▶ Postgres (Neon): companies, topics, packs, calls, credits
                       │
                       ├──▶ transcript database (SQL over HTTPS)      ├──▶ filings API (10-K / 10-Q sections)
                       ├──▶ financials API (statements, prices)       ├──▶ web answers API (news, analysts)
                       └──▶ earnings-calendar + short-interest API    └──▶ news feed API (daily trivia)

Two deployables: a Next.js app on Vercel (UI, auth, preparation, all the caching) and a small Python agent on LiveKit Cloud's agent hosting. No servers of our own.

Integrating a transcript database

The foundation is a database of earnings-call transcripts: roughly a quarter of a million calls across six thousand tickers going back twenty years. It exposes two views that matter. One row per transcript with the full text as JSON, and one row per paragraph with the speaker and paragraph number. That second view is what makes the product possible: you can regex-search fifteen million paragraphs in about a second and get back exactly the exchange you need, with a speaker name attached.

Talk to it over HTTPS with a key pair

We never ship a database driver to Vercel. The warehouse has a SQL-over-REST endpoint that accepts a short-lived JWT signed with an RSA key, so the client is a hundred lines of TypeScript: sign a token with the public-key fingerprint as issuer, POST the statement with bind parameters, poll if it comes back as still running, and page through partitions for large results.

const token = await new SignJWT({})
  .setProtectedHeader({ alg: "RS256" })
  .setIssuer(`${ACCOUNT}.${USER}.SHA256:${publicKeyFingerprint}`)
  .setSubject(`${ACCOUNT}.${USER}`)
  .setIssuedAt().setExpirationTime(now + 3600).sign(privateKey);

const res = await fetch(`${BASE}/api/v2/statements`, {
  method: "POST",
  headers: { Authorization: `Bearer ${token}`, "X-Auth-Token-Type": "KEYPAIR_JWT" },
  body: JSON.stringify({ statement: sql, bindings, timeout: 90, parameters: { DATE_OUTPUT_FORMAT: "YYYY-MM-DD" } }),
});

Two gotchas cost us an afternoon. Dates come back as days-since-epoch unless you ask for a format, and the warehouse's RLIKE is anchored to the whole string, so “contains” searches need REGEXP_INSTR(content, pattern, 1, 1, 0, 'i') > 0.

The search that everything else builds on

One query does the heavy lifting: given tickers, a handful of regex synonyms, and optional date bounds, return matching paragraphs with one neighbor on each side for context, at most five hits per call, newest first, plus a footer with the total hit count and hits per fiscal year. That footer matters because it lets the model say “66 mentions, none after 2021” instead of guessing.

WITH hits AS (
  SELECT transcript_key, symbol, fiscal_year, fiscal_quarter, report_date, paragraph_number,
         ROW_NUMBER() OVER (PARTITION BY transcript_key ORDER BY paragraph_number) rn
  FROM paragraphs
  WHERE symbol IN (?) AND REGEXP_INSTR(content, ?, 1, 1, 0, 'i') > 0
),
sel AS (SELECT * FROM hits WHERE rn <= 5 ORDER BY report_date DESC LIMIT 25)
SELECT p.*, (p.paragraph_number = s.paragraph_number) AS is_hit, (SELECT COUNT(*) FROM hits) AS total_hits
FROM sel s JOIN paragraphs p
  ON p.transcript_key = s.transcript_key AND p.paragraph_number BETWEEN s.paragraph_number - 1 AND s.paragraph_number + 1
ORDER BY p.report_date DESC, p.paragraph_number;

Results are formatted as compact text lines, not JSON, before they reach a model: [SIG FY2027 Q1 | 2026-06-02 | ¶23 | Joan Hilson] …. It halves the tokens and the model cites it correctly.

Cache everything by query hash

Every search result, section fetch, and per-transcript summary is cached in Postgres keyed by a hash of the SQL and its bindings. Preparation for a company that someone already looked at is a few hundred milliseconds instead of a minute, and the warehouse bill stays flat.

Preparing a company before the call

A conversation only feels natural if the rep never has to go read something mid-sentence. So we do the reading up front and cache it per company:

Financials: eight quarters and five fiscal years of income statement, key metrics, and a year of prices from a financials API. Deterministic slides (KPI tiles, revenue bars, margin lines) come straight from this, no model involved.
Call summaries: the four most recent transcripts are summarized by a cheap model into a fixed structure (numbers and guidance, initiatives, what management is excited about, analyst skepticism, tone), each point tagged with speaker and paragraph number.
Fact sheet: the Business, Properties, and MD&A sections of the latest annual report, plus the newest quarterly MD&A, are pulled from a filings API and distilled into 20 to 40 operating facts an IR person is expected to know cold: store counts by country, headcount, segment mix, square footage, comps, e-commerce share, cash and debt. This was the single biggest quality jump. Before it, the rep would answer “how many stores do you have” with “I don't have that in front of me”.
Topics: fifteen company-specific topics generated from the summaries, always including results, guidance, margins, and capital allocation, then the company's own themes (for a furniture retailer: tariffs, design-center footprint, wholesale backlog). Cached for months, so each company builds up its own list.
Topic packs: for each chosen topic, a transcript search plus the fact sheet become five to eight talking points, verbatim quotes with speaker and date, key figures, and two or three slide specs.
Briefing: all of the above assembled into a system prompt with a persona, ground rules, the deck listing, and the fact sheet. About 8,000 tokens.

Structured output from the cheap model is validated with zod. We learned to keep the schemas lenient (accept numbers where strings are expected, tolerate renamed keys, unwrap a single-key wrapper object) and to give the model a large output budget, because “thinking” tokens count against it and a truncated JSON array fails silently otherwise.

The voice agent on LiveKit Cloud

We considered running a browser-orchestrated pipeline to avoid any long-running process, and the avatar vendor's hosted pipeline, which bundles speech and a model but only accepts a prompt configured in a dashboard. Neither gives you a per-company system prompt from code with tight control over turn-taking. LiveKit Agents does, and LiveKit Cloud will host the agent for you, so there is still no server to run.

The agent

The whole worker is one Python file. A job is dispatched per call; it reads the call id from the job metadata, fetches the briefing from our API with a shared secret, and starts a session.

@server.rtc_session(agent_name="ir-rep")
async def ir_agent(ctx: JobContext):
    call_id = json.loads(ctx.job.metadata)["callId"]
    briefing = await fetch_briefing(call_id)          # persona, instructions, deck, greeting, mode

    session = AgentSession(
        stt=inference.STT(model="deepgram/nova-3", language="en"),
        llm=openai.LLM.with_openrouter(model=model, reasoning_effort="minimal"),
        tts="deepgram/aura-2:athena",
        turn_handling=TurnHandlingOptions(
            turn_detection=inference.TurnDetector(),
            endpointing={"mode": "dynamic", "min_delay": 0.25, "max_delay": 1.5},
            interruption={"mode": "adaptive", "min_duration": 0.4, "resume_false_interruption": True},
            preemptive_generation={"preemptive_tts": True},
        ),
    )
    await ctx.connect()
    ...
    await session.start(room=ctx.room, agent=IrRep(briefing, ctx.room, call_id),
                        room_options=room_io.RoomOptions(audio_output=not avatar_ok))
    await session.say(briefing["greeting"], allow_interruptions=False)

Speech-to-text and text-to-speech run through LiveKit's inference gateway, so the only third-party key the agent holds is for the language model. Tools are plain methods with docstrings:

class IrRep(Agent):
    @function_tool()
    async def show_slide(self, context: RunContext, slide_id: str) -> str:
        """Show a slide to the investor. Call this right before you start talking about a point that has a slide."""
        await self._room.local_participant.send_text(
            json.dumps({"type": "show_slide", "slide_id": slide_id}), topic="ir.slides")
        return "shown"

Dispatch from the web app

The Next.js route that starts a call mints the participant token and attaches an agent dispatch to the room configuration, so the agent joins the moment the room is created. Metadata carries the call id. One rule to know: token-based dispatch only fires when the room is created, so every call gets a unique room name.

const at = new AccessToken(key, secret, { identity: `user-${userId}`, ttl: maxMinutes * 60 + 180 });
at.addGrant({ roomJoin: true, room: `ir-${callId}`, canPublish: true, canSubscribe: true });
at.roomConfig = new RoomConfiguration({
  agents: [new RoomAgentDispatch({ agentName: "ir-rep", metadata: JSON.stringify({ callId, symbol }) })],
});

Deploying

The LiveKit CLI generates a Dockerfile, builds in their cloud, and rolls out: lk agent create --region us-east --secrets-file secrets.env, then lk agent deploy for updates and lk agent logs to tail. Secrets are set through the CLI and never live in the image; the LiveKit credentials are injected automatically. The free tier allows five concurrent sessions and scales to zero, which adds ten to twenty seconds on the first call after a quiet period.

Adding a Lemon Slice avatar

Lemon Slice turns a single portrait into a real-time talking character. With LiveKit there is an official plugin, and integration is four lines plus two rules.

avatar = lemonslice.AvatarSession(
    agent_image_url=AVATAR_URL,                      # public HTTPS, 368x560, image/*
    agent_prompt="a friendly, composed corporate spokesperson",
    agent_idle_prompt="a calm professional listening attentively",
    idle_timeout=180,
)
await avatar.start(session, room=ctx.room)          # avatar joins as participant "lemonslice-avatar-agent"
await session.start(..., room_options=room_io.RoomOptions(audio_output=False))
await utils.wait_for_agent(ctx.room)                # don't greet until the avatar is in the room

Rule one: turn the agent's own audio output off. The avatar receives the synthesized speech, renders lip-synced video, and republishes the audio in sync. If the agent also publishes audio you get a doubled, out-of-sync voice. Rule two: the portrait must be a public URL the vendor's servers can fetch, so localhost and site-relative paths fail silently. We host it on the Vercel deployment.

Two useful signals arrive on a data topic named lemonslice: bot_ready with a session id (we record it immediately so we can look up the session's cost later even if the agent process dies) and metric with time-to-first-push. In our logs the avatar adds a steady one second of playback latency on top of text-to-speech.

On the browser side, LiveKit's React components find the avatar's camera track by participant identity, and the vendor's LiveKitAvatarReadyWatcher fires on the first decoded frame, which is the right moment to unmute the microphone and hide the “connecting” overlay. Waiting for bot_ready alone gives you a black flash.

One more thing we learned the expensive way: if the avatar account runs out of credit, the plugin raises during start(). We wrap that in a try, fall back to audio_output=True, and send an avatar_unavailable event so the page shows the still portrait with a “voice only” note instead of crashing the call.

The portrait and the assistant clips

The rep's headshot and the assistant character who appears during setup were generated with an image model and animated with image-to-video models into five-second loops: typing, on the phone, taking notes, nodding, a thumbs up, checking a watch. The setup screen picks a clip whose tags match the current narrative line and rotates otherwise, with a crossfade. Thirteen clips came to about two megabytes.

Slides that follow the conversation

Slides are JSON specs rendered in React, not images: title, bullets, KPI tiles, bar and line charts drawn with inline SVG, and quotes. Each carries a topic id and keywords. The deck for a call is a title slide, the deterministic financial slides, two “business at a glance” slides from the fact sheet, and two or three slides per chosen topic from the packs. The system prompt lists every slide as id | title | topic | one-line content.

Sync is exact because the model decides: it calls show_slide right before making a point, the agent publishes the id on a text stream, and the browser flips before the sentence is spoken. As a safety net the page also matches keywords in the live transcript. On phones the slide takes the full width, with the avatar as a tap-to-swap tile beside a scrolling transcript, because a picture-in-picture avatar kept covering the numbers.

Getting the rep to answer fast

The first version felt slow and we measured before guessing. LiveKit attaches per-turn metrics to each assistant message (time to first token, time to first sentence, text-to-speech first byte, playback latency), and we logged them. Then we timed the model directly with the real 8,000-token briefing:

Model, with the briefing as system prompt	Time to first token
Claude Sonnet 5, default	3.2 to 5.0 s
Claude Sonnet 5, reasoning off	1.1 to 1.3 s
Claude Haiku 4.5, reasoning off	1.0 to 1.2 s

The model was thinking for seconds before speaking. Setting reasoning_effort="minimal" through the OpenRouter helper cut three seconds off every reply with no visible loss in a conversational answer. We then added a fast mode (Haiku, tighter endpointing, shorter interruption threshold) and made it the default, shortened the greeting from twenty seconds to ten, and told the rep to answer in one to three sentences and lead with the answer. Measured end to end with the avatar, a question the briefing covers now gets first words in about four seconds, of which one is the avatar render.

Background lookups instead of “let me check”

Questions outside the briefing used to trigger a tool call, a five-to-twelve-second lookup, and a second model pass before the rep said anything. Now the tool starts the lookup as a background task and returns None, which in LiveKit Agents means “no second model call”. If the model said nothing alongside the tool call, a short in-context sentence is generated within half a second so there is no dead air. When the lookup completes, its text is written in a spoken style already, so the rep reads it with session.say() as soon as her sentence ends, with a varied opener rather than a fixed one.

def _start_background(self, question, mode):
    asyncio.create_task(self._deliver(question, mode))   # fetch, then session.say(answer)
    asyncio.create_task(self._bridge_if_silent(question)) # after 0.4s, if not speaking: one natural sentence
    return None                                            # no automatic tool reply

async def _deliver(self, question, mode):
    ans, source = await self._fetch(question, mode)
    text = ans if source != "web" else f"From what has been reported publicly, {ans}"
    await self.session.say(text, allow_interruptions=True)

The lookup itself is two-stage: try the compact fact sheet first (three to four seconds), and only read the full filing sections plus a transcript search when that returns nothing. Anything the filings can't answer, such as analyst reaction or a competitor's move, falls through to a web-answers API and is attributed as public coverage. We also stopped the rep from reciting sources; she gives the figure and the period, and names the document only if asked.

The scheduling narrative

Preparing a company for the first time takes a minute or two. Rather than a spinner, the wait is staged as an assistant reaching out on your behalf: typewriter messages map to real preparation steps but stay strictly in character (“Request sent, waiting to hear back from their IR desk”, “Their team is putting together a proposed agenda”), with filler lines if nothing new happens within eight seconds. The topics then arrive as an email from the rep, addressed from the company's real domain, asking where to focus. Meanwhile a quiz plays: general earnings-call trivia, then questions about the specific company, and a daily set built from that week's business news via a news feed and a web-answers API. Correct answers earn credits, tracked server-side and paid once per question, which is the hook for a paid model later.

Operations, limits, cost

Phone-OTP login with admin approval; per-user limits of ten minutes per call and three calls a day; a global daily minute cap.
A daily cron re-pulls the earnings calendar (companies over a billion in market cap reporting within two weeks, ranked by short interest), pre-warms the top five companies including their default topic packs, rebuilds the ticker list for autocomplete, and regenerates the news quiz.
Preparing a company fully costs about 25 cents in model calls; a lookup a few cents; the avatar is billed by the minute by its vendor.
Every generated artifact (summaries, fact sheets, topics, packs, quizzes) is cached with a TTL, so repeat visits are near free.

Lessons

Paragraph-level transcript search with speaker attribution is the whole product. Everything else is presentation.
Do the reading before the call. A fact sheet from the annual report made the rep sound like she works there.
Measure latency per stage before choosing a model. Reasoning tokens were the problem, not the model's speed.
Never let a tool block speech. Start it, say something true, and fold the result in.
Turn the agent's own audio off when an avatar is in the room, and never unmute the user until the first video frame arrives.
Write for a human to say aloud: short sentences, no citations, varied openers. The transcript panel makes every robotic phrase visible.

Try it at irsimulator.com. The rep is a simulation built from public transcripts and filings, not affiliated with any company, and not investment advice.