Anam Assist: a talking avatar for Home Assistant

To test Anam Assist on something physical, I used a light I could see. It was on at full brightness. I typed a request to the avatar to turn it off and watched the light go out, asked for it back on and watched it come back, then asked for its state. Home Assistant reported it on at 100%, where it had started.

I started out wanting a face for Home Assistant's Assist, with good tool calling and low latency as the goals. The more useful work turned out to be the boundary between a cloud service that runs the conversation and the system that's allowed to act on the house.

What Anam Assist is#

Anam Assist is an open-source custom integration that adds a panel to Home Assistant's sidebar. In it, you talk or type to an Anam avatar that can read the devices you've exposed to Assist. Before a session starts, you can also let it control exposed lights or use Home Assistant's full set of Assist device controls.

I work at Anam, as I wrote when I joined the company, so I know the SDK well and I'm not a neutral judge of it. This is an unofficial, MIT-licensed project in my own GitHub account, and an experimental preview. It isn't an Anam, Home Assistant or HACS release. I built it with help from coding agents, had agents review it in several focused passes, then exported it to a clean public repository without my home's configuration, credentials, transcripts or infrastructure history.

Why a separate browser panel#

Assist handles voice as a pipeline of speech-to-text, a conversation agent and text-to-speech. An Anam session covers the same ground, plus the avatar's video. Home Assistant's LLM API describes Assist's tools to a language model and runs them under Assist's exposure rules. Anam Assist joins the two at the tool layer. I liked that split because the voice and face come from one Anam session, and Home Assistant only has to decide whether a tool can run. Your existing Assist pipelines are untouched, and the panel doesn't add a face to voice satellites or conversations elsewhere in Home Assistant.

A signed-in Home Assistant page already has an authenticated connection to the server, microphone access over HTTPS and somewhere to play video, so the browser came first. A wall screen or kiosk can open the same page, but there's no RTSP stream, casting or unattended recovery yet. Portrait is the default because phones and small screens are likely places to use it. The integration requests native 768 × 1152 video, which needs a portrait-capable Cara 4 avatar; older avatars can use landscape.

How a request reaches the house#

Anam's client tools are functions the model can call during a session that run in the user's browser: the SDK passes each call to a handler on the page and returns the result to the model. Home Assistant's frontend keeps a WebSocket open to the server, a long-lived two-way connection that's already authenticated as you. Anam Assist adds its commands to that connection.

When you start a session, Home Assistant uses the stored Anam API key to request a short-lived session token, along with the integration's prompt and the tools allowed this time. The browser opens the conversation with that token. When the model needs something from the house, it calls a client tool, the browser forwards the call over the WebSocket, and the integration checks it and runs the matching Assist tool. The result goes back to Anam before the avatar answers.

Starting a session

1. Your browser

   Anam Assist panel

2. ↓→asks to start

   Home Assistant

   Anam Assist integration

   Holds the API key and picks the allowed tools

3. ↓→uses the API key

   Anam API

   Issues a short-lived token

   Passed back to the browser

Conversation and avatar

1. Your browser

   Your voice or text in, avatar video out

2. ↓→Anam SDK, both ways

   Anam cloud

   Speech recognition, model, voice and avatar

Tool calls

1. Anam cloud

   Model calls a client tool

2. ↓→client tool call

   Your browser

   Forwards the call

3. ↓→authenticated WebSocket

   Home Assistant

   Checks the call and runs the Assist tool

   The result goes back the same way

The Anam API key stays in Home Assistant's config entry, and so in your normal backups; the browser only gets a token that expires. There's no extra long-lived Home Assistant token in page code and no public webhook, so tool calls can only arrive over a signed-in administrator's connection. The conversation itself runs in Anam's cloud, so this isn't a local voice assistant.

The integration takes the avatar, voice and model from the Anam persona you choose, or from the public Cara example, and adds its own prompt and tools for each session. You don't configure Home Assistant tools in the persona, and the saved persona isn't changed.

I haven't compared latency with other voice stacks. The activity log shows each tool call's round trip to Home Assistant, which is one hop, not the time from finishing a sentence to hearing the answer.

Bugs that changed the design#

Double underscores in tool names#

Home Assistant gives its Assist tools prefixed names with a double underscore, such as intent__HassGetState. An early version used those as client tool names, and a temperature query failed. Names with double underscores were normalised differently between the session's tool list and the call the browser received, and the SDK only routes a call to a handler whose name matches exactly.

Each tool now gets an opaque, provider-safe name, ha_ plus 32 hex characters of a SHA-256 hash of the original, and the server keeps the mapping. It translates each incoming call back to the Home Assistant name and rejects any name it didn't issue. A tool name looks like a plain string, but every system it passes through has to agree on it exactly, so it's part of the protocol.

Real packages caught a missing name#

Testing against Home Assistant's real packages caught another naming problem. Current Home Assistant registers the brightness tool as light__HassLightSet, which the allowlist didn't include, so brightness control would never have been offered. A test now loads the real light tools and checks them against the allowlist. Mocks with invented names couldn't have caught it, because they shared the code's assumption about the name.

Late arrivals in the browser#

In a browser voice session, plenty can finish late, from a microphone permission prompt to a tool call still running when you press End. Reviews found ways this could leave the microphone live after you'd ended a conversation, show the last session's tool result in the current activity log, or leave the buttons stuck after a failed cleanup. Each conversation now gets a number that every callback checks before touching the page. Late media is stopped as soon as it arrives, the abandoned session is revoked on the server, and cleanup gives up after three seconds so the controls come back. The browser tests cover each case.

Where the boundary sits#

The integration's prompt tells the model whether the session is read-only and to treat device names and states as data. That's guidance, and a model can be confused or manipulated, or read an instruction hidden in a device name. So the server enforces what can run, whatever the model asks for.

Only Home Assistant administrators can open the panel or use its server commands. Before starting a session, you choose one of three scopes: Read-only, the default; Lights only; or Assist controls. The scope can't change mid-conversation. Each session is tied to the WebSocket connection that started it and expires after ten minutes. There can be four open across the installation, with 100 distinct tool calls each.

Every call is checked against the tools issued for that session, with limits on the size of its arguments. Assist's exposure rules are checked again when each tool runs, so an entity you stop exposing mid-session can't be controlled. In Lights only, Home Assistant's generic turn-on and turn-off intents are pinned to the light domain, so a turn-on aimed at a lock is refused before Assist sees it.

Assist controls passes on every tool from Home Assistant's built-in Assist device-control API. Depending on your integrations and exposed entities, that can include switches, fans, covers, locks, heating, media and exposed scripts or scenes, with no separate confirmation before an action runs. Home Assistant's administrative API is never offered. Neither scope prevents side effects: if an automation fires when a light turns off, Lights only can trigger it, and in Assist controls an exposed script or scene can change devices you haven't exposed.

The server records each call ID as "outcome unknown" before the action runs. If a response is lost and the same call arrives again, it gets that recorded result instead of a second run; reusing an ID with different arguments is rejected. Nothing retries automatically, and after a timeout the model is told the action may have happened. Ending a session stops queued calls but can't undo one already sent to a device.

Read-only sessions still send data out of the house. Anam receives your audio or typed messages, Assist's prompt context and every tool result, and Home Assistant's live-context tool can return every exposed entity's state when you ask about one sensor. Your Assist exposure settings decide what can go. Session video recording and stored transcripts are switched off; that's a session setting, not a promise that Anam retains nothing. The panel's transcript stays in page memory.

The integration can't protect itself from a compromised Home Assistant administrator, installation or browser origin, and browser extensions with access to the page can reach the session. It hasn't had an independent security audit.

What the testing showed#

On Home Assistant 2026.10.3, a native portrait stream, a temperature query and the light test from the start, with control limited to lights, all worked. Afterwards I ended the session and switched light control off. Every request was typed, with the microphone off, so the test covers the tool path against a real device but not spoken commands.

For the current version, 57 backend tests and 40 browser tests across Chromium and WebKit passed, along with linting, a reproducible build and a dependency audit. The backend tests ran against Home Assistant's real packages in a network-disabled container, covering admin checks, exposure filtering, input limits, replays, configuration and cancellation. The browser tests use stand-ins for Home Assistant and the Anam SDK, so they don't prove real audio works. Assist controls hasn't been tried on physical devices yet; its backend tests run a real climate tool against a synthetic service, including exposure being revoked mid-session. None of that is a lasting security guarantee.

Try it#

You'll need Home Assistant (tested on 2026.10.3), an administrator account, HTTPS for the microphone and an Anam API key with streaming access. Anam usage is billed separately.

1. Back up Home Assistant.
2. Download the repository and copy the whole custom_components/anam_assist folder into the custom_components directory of your Home Assistant configuration. The browser bundle is included, so the host doesn't need Node.js.
3. Restart, then go to Settings → Devices & services → Add integration → Anam Assist.
4. Enter your Anam API key. Optionally add a published persona ID, or leave it blank for the public Cara example.
5. Open Anam Assist in the sidebar and ask about an exposed sensor. Untick "Use microphone" to type instead.
6. To test a light, end the session, choose Lights only under Home controls, start again and ask for one light you can see. Leave Assist controls until you've reviewed everything you expose to Assist, including scripts and scenes.

Reconfigure changes the persona, orientation or key, and a blank key keeps the saved one. To update, replace the whole folder so the frontend and backend stay in step. It's a custom integration with a sidebar panel, not a Supervisor add-on or a Lovelace card. HACS metadata is included, but the project isn't in the default catalog and the HACS lifecycle hasn't been validated, so manual installation is the verified route. The setup guide has current instructions and troubleshooting.

What still needs testing#

• Assist controls on physical devices.
• A clean-install HACS lifecycle on a real instance.
• Home Assistant versions other than 2026.10.3.
• Microphone and audio on physical mobile browsers.
• Unattended kiosk recovery.
• An independent security review.

Reports on any of these would help most. Keep keys, transcripts and home state out of public issues, and send security problems through private vulnerability reporting.

To try it safely, review what you've exposed to Assist, then start read-only with one sensor. If you'd rather start with the code, the tool checks are in __init__.py.