Functions

Functions are how the autopilot reads from or acts on the client. You register a function; the server can invoke it during a conversation.

registerFunction(definition)

client.registerFunction({
  name: 'get_selection',
  description: 'Returns the text the user has currently selected on the page.',
  parameters: {
    type: 'object',
    properties: {},
  },
  handler: () => window.getSelection()?.toString() ?? '',
});

Definition shape

interface AutopilotFunctionDefinition<TArgs = unknown, TResult = unknown> {
  name: string;
  description?: string;
  parameters?: Record<string, unknown>; // JSON Schema
  handler: (
    args: TArgs,
    context: { id: string; name: string; timestamp?: number },
  ) => TResult | Promise<TResult>;
  sensitive?: boolean; // gate execution behind a user-confirmation handler
}

The parameters object is JSON Schema — the same vocabulary the server's tool calls use. Mark required fields:

parameters: {
  type: 'object',
  properties: {
    city: { type: 'string' },
    units: { type: 'string', enum: ['c', 'f'], default: 'c' },
  },
  required: ['city'],
},

Async handlers

client.registerFunction({
  name: 'fetch_user',
  parameters: {
    type: 'object',
    properties: { id: { type: 'string' } },
    required: ['id'],
  },
  handler: async ({ id }: { id: string }) => {
    const res = await fetch(`/api/users/${id}`);
    return res.json();
  },
});

Anything you return is JSON-serialized and sent back as the function result.

State sync before the response

Handlers often mutate client state — setContext(), updateContext(), or even registerFunction(). Those changes are normally debounced (100 ms for context, 25 ms for the function catalog), so without coordination the function response could reach the backend before the state it depends on, and the LLM would compose its next completion from stale context.

By default the client prevents that: when a handler settles and dirty state exists, the client flushes it immediately and withholds the function response until the relay acknowledges the patches (meaning the state is committed). If nothing is dirty, the response goes out synchronously — the common case adds zero latency. A fallback timeout (default 2000 ms) guarantees a lost acknowledgement can never stall the call; the backend's own response timeout is far higher.

Local functionCall / functionSuccess / functionError events are not delayed — they fire at handler settlement as usual. Only the wire response waits.

Both knobs live on connect() options:

await client.connect({
  conversationUrl,
  syncStateBeforeFunctionResponse: false, // opt out — respond immediately
  syncStateBeforeFunctionResponseTimeoutMs: 500, // or just tighten the ceiling
});

Deferring the response for async context

The sync gate above only sees state that is dirty when the handler settles. Sometimes the context a function produces doesn't exist yet at that point: a navigate handler returns as soon as routing starts, while the destination page still has to fetch its data and setContext() it. Without help, the response reaches the LLM before that context exists.

Context resolutions let the app say "I have something to resolve into context — hold function responses until it's done":

// shared module — bridges the handler and the destination page
export let pageReady: (() => void) | null = null;

// navigate handler (old page) — acquire BEFORE starting the async work:
client.registerFunction({
  name: 'navigate',
  handler: ({ path }: { path: string }) => {
    pageReady = client.beginContextResolution('user-list');
    router.push(path);
    return `navigating to ${path}`;
  },
});

// destination page (e.g. UserList.vue) — land the data, then release:
onMounted(async () => {
  const users = await fetchUsers();
  client.updateContext({ users });
  pageReady?.(); // now the context flushes, the relay acknowledges, THEN the response goes out
});

When you already hold a promise for the work, use the sugar — it acquires and auto-releases when the promise settles (resolve or reject), passing the value/error through:

client.holdContextUntil(
  fetchUsers().then((users) => client.updateContext({ users })),
  'user-list',
);

Semantics:

• While any resolution is outstanding, every function response is withheld — including chained acquisitions (a page may acquire a new resolution before releasing the previous one).
• The release function is idempotent; calling it twice (or after a disconnect) is harmless.
• A forgotten release can't stall the conversation: after contextResolutionTimeoutMs (default 10 s) the response is sent anyway. Keep the combined budget in mind — the backend times a function call out after 30 s.
• Once resolutions clear, the state-sync gate runs as usual, so context written during the resolution is committed before the response. Resolutions are honored even when syncStateBeforeFunctionResponse is disabled.
• The contextResolutionChanged event fires with { pending } on every acquire/release — handy for a "working…" indicator.

Timing

Every call is measured with two durations:

• handlerMs — request received → handler settled (your code's time, including any confirmation prompt).
• totalMs — request received → response emitted on the wire. The difference to handlerMs is withheld time: state sync and context resolutions.

They surface in three places:

• Event payloads: functionSuccess/functionError carry the handler durationMs; functionResponseSent carries the total (see Events).
• client.getFunctionCallHistory() — the last N calls (newest first, functionCallHistoryLimit option, default 50) as FunctionCallRecords: { id, name, status, startedAt, settledAt, respondedAt, handlerMs, totalMs }. respondedAt/totalMs are null when the response was never sent (e.g. disconnect). The history is kept across disconnects.
• The Vue layer grafts these timings onto the conversation's function_calls[] (as duration_ms/total_ms) so the bar displays them on the call pills, and shows them in the activity log.

Errors

handler: () => {
  throw new Error('not allowed in incognito mode');
};

The error is caught, reported as a functionError event, and sent to the server as a failed call — it does not disconnect the client.

unregisterFunction(name)

client.unregisterFunction('fetch_user'); // returns true if it existed

clearFunctions()

Removes every registered function in one shot.

hasFunction(name)

if (!client.hasFunction('navigate')) {
  client.registerFunction({ name: 'navigate', /* … */ });
}

getRegisteredFunctions()

client.getRegisteredFunctions();
// → [{ name, description, parameters, source: 'client' }, …]

Returns the descriptors as seen by the server. The original handler is not included.

syncRegisteredFunctions()

Force an immediate sync. Normally registrations are batched into a single JSON Patch with a 25 ms debounce; call this if you must sync before the next tick (e.g. you're about to disconnect).

Like context, function sync is subject to the relay's 64 KB stored-state cap. If the projected state would exceed it, the sync is skipped and a stateError event fires instead.

Sensitive functions

Mark a function sensitive: true to require user confirmation before each invocation. The autopilot can still request the call at any time, but the client holds execution until your confirmation handler resolves.

client.registerFunction({
  name: 'delete_account',
  description: 'Permanently deletes the user account.',
  sensitive: true,
  handler: async () => {
    await fetch('/api/account', { method: 'DELETE' });
  },
});

setConfirmationHandler(handler)

Register one handler per client. It receives a ConfirmationRequest and must return — or resolve to — 'allow' or 'deny'.

client.setConfirmationHandler(async (req) => {
  const ok = window.confirm(
    `Autopilot wants to run "${req.name}" with:\n${JSON.stringify(req.arguments, null, 2)}`,
  );
  return ok ? 'allow' : 'deny';
});

Pass null to clear it.

Fail-closed behavior

If a sensitive call arrives and no confirmation handler is set — or the handler throws — the call is denied automatically. The server receives an error response ("User denied execution of sensitive function.") and the functionError event fires. Sensitive functions never run silently.

Non-sensitive functions bypass the confirmation handler entirely.

Confirmation events

These fire only for sensitive calls — see Events:

client.on('functionConfirmationRequired', ({ request }) => {
  console.log('waiting on user:', request.name);
});

client.on('functionConfirmationResolved', ({ request, decision }) => {
  console.log(request.name, '→', decision); // 'allow' | 'deny'
});

The Vue layer ships a ready-made UI on top of this seam — see useAutopilotStore and the ConfirmationPrompt component.

Listening to invocations

client.on('functionCall', ({ request }) => {
  console.log('→', request.name, request.arguments);
});

client.on('functionSuccess', ({ request, result }) => {
  console.log('✓', request.name, result);
});

client.on('functionError', ({ request, error }) => {
  console.error('✗', request.name, error);
});

Ask user

Sometimes the autopilot needs structured input from the user, not from your code — pick one option, choose several, or type an answer. This is now a backend-native capability: the LLM pauses mid-answer, the conversation suspends server-side (WAITING_FOR_ASK_USER), and resumes once the client posts the answers. There is nothing to register on the client — the feature is enabled per-app via the server's Ask User config preset (default off).

Flow

1. Prompts arrive on any conversation response under pending_ask_user, and (for streaming clients) via the ask_user.request relay message. The client surfaces them as getPendingAskUser() and the askUserRequired event.
2. Render the prompts, collect answers, and post them back with respondToAskUser(...).
3. The returned response may carry a fresh pending_ask_user (the LLM asked again); repeat until it's absent (askUserCleared).

Reload-safe: a fresh GET of the conversation also returns pending_ask_user, so connect() re-surfaces any in-flight prompts automatically.

client.on('askUserRequired', ({ prompts }) => { /* render UI */ });
client.on('askUserCleared', () => { /* hide UI */ });

await client.respondToAskUser([
  { id: 'call_abc', result: 'blue' },     // radiolist → option value
  { id: 'call_def', result: true },       // bool      → boolean
  { id: 'call_ghi', result: ['a', 'c'] }, // checklist → value[]
  { id: 'call_jkl', result: 'free text' },// text      → string
]);

A single { id, result } object is also accepted. Answer all pending prompts in one request; the conversation only resumes once every prompt is answered.

Prompt shape

interface AskUserPrompt {
  id: string;
  type: 'text' | 'bool' | 'radiolist' | 'checklist';
  question: string;
  arguments?: {
    yes_label?: string; no_label?: string;   // bool
    placeholder?: string; multiline?: boolean; // text
    options?: { value: string; label: string }[]; // radiolist / checklist
  };
}

Answer value by type

Type	Result value (AskUserResult)
text	string
bool	boolean
radiolist	option value (string)
checklist	array of option values (string[])

The Vue layer ships an opinionated tabbed UI on top — see AskUserPrompt and the pendingAskUser / submitAskUser members on the store.