Sightline: page context for AI assistants without screenshots

When someone asks an assistant "why can't I click this?", a screenshot can help. A vision model can inspect the image and answer the question. But a web app already exposes information about its controls: their labels, whether they're disabled and which dialog is open. For questions like this, I want to give the assistant that information directly.

That's the idea behind Sightline, an open-source JavaScript package I've been working on. It runs inside a web page and produces a compact text outline for an LLM. There's no model call in the page-reading step, and no runtime dependencies. It's available under the MIT licence.

Sightline was built for Sophie, Anam's in-app assistant. The problem applies to other products too: documentation can explain how a feature works, but helping someone use it also means understanding the state of the page they're looking at.

What the model gets#

Sightline walks the DOM and reads HTML and accessibility semantics, including roles, accessible names and control states. It builds its own outline from that information; it doesn't retrieve the browser's complete accessibility tree or analyse the pixels on screen.

An illustrative outline for a form might look like this:

dialog "Create project"
  textbox "Name" [empty] [ref=e1]
  button "Create project" [disabled] [ref=e2]
  button "Cancel" [ref=e3]

The assistant can see that the name is empty and the button is disabled. If the product requires a name, it can suggest filling that in. The outline alone doesn't prove why the button is disabled: permissions or another requirement could be involved. Product knowledge and tools still matter.

Sending the whole DOM would include far too much irrelevant detail. Sightline condenses repeated items, summarises tables and caps the outline at 6,000 characters by default. It skips content marked as hidden from accessibility or inert, and lets you exclude the assistant's own panel. With viewport reporting enabled, it marks offscreen controls, so the outline isn't limited to what's visible in a screenshot.

Where it saves time and money#

The extraction runs locally in the browser. The published benchmark reports median capture times of 0.8 to 1.7 milliseconds on generated pages in Chromium 143, with control references and viewport reporting enabled. Repeated content is condensed and output is capped. These are extraction timings, not assistant response times or a comparison against a vision model.

If your application sends screenshots to a separate model for descriptions, replacing that stage with Sightline removes an inference call and its network round trip. You still pay for the conversational model to process the outline and answer the question.

If the conversational model accepts images directly, you're comparing text and image input within one call. Savings depend on the model, image settings and outline size; a text outline isn't automatically cheaper or faster in every case. The appeal for me is being able to supply control names and state with a small local operation when that's all the task needs.

Keeping the context useful#

A snapshot goes stale when someone opens a menu or navigates. Sightline can watch for changes and report them after the page settles. It can also report named interactions, such as a button click. Control references let an assistant highlight the element it's discussing, which is useful when several fields have similar names.

Typed field values are excluded, but page text can still contain sensitive information. The application needs to decide what context it sends and configure exclusions and redaction accordingly. Page content also remains untrusted input to the model; turning it into an outline doesn't make instructions found on the page safe to follow.

Where screenshots still help#

An outline won't tell you whether a photograph looks right, a chart's visual pattern is meaningful or a layout is broken. Those tasks need visual information. Sightline is for questions about page content and controls, within the page's access boundaries. It can't read your desktop or the contents of cross-origin frames.

It also depends on the quality of the page. A labelled button gives it a clear name; an anonymous clickable div gives it much less. Optional inference uses clues such as icon names and styling to fill some gaps, with guesses marked where used. On a product I control, I'd fix the labels first. That helps people using assistive technology too.

The broader approach of giving agents structured page snapshots already exists in tools such as Playwright. Sightline focuses on an assistant embedded in the product, with a bounded outline, exclusions and updates as the user interacts.

Try it in your own app#

You can run Sightline on this article first. The button below captures the real page, including its navigation and links, and shows the text an assistant could receive.

Install the package:

npm install @smidge/sightline

Capture an outline in browser code when the user asks a question:

import { createSightline } from '@smidge/sightline';
 
const sightline = createSightline({
  skip: '[data-agent-panel]',
  viewport: true,
});
 
const pageOutline = sightline.capture()?.text;

Pass pageOutline as context through your existing chat integration. The documentation covers watching for changes, tracking interactions and highlighting controls.

You can also try the demo and bookmarklet before integrating anything. Start with a form or an open dialog on a page you're comfortable inspecting. The output will show which labels and states your assistant could use, and where the page leaves it guessing.