browserTools()

browserTools(options): ToolSet

The browser tools: browser_open, browser_click, browser_fill, browser_select, browser_press, browser_scroll and browser_screenshot, over Telnyx Agent Browser. Pass them to createHarness({ tools }), alone or beside your own. Each durable run holds one browser session, opened on the site of the first URL the model opens (that host and its subdomains), replaced when the model opens another site, and closed when the run ends: the operation that ended the run waits for the close for at most HARNESS_RUN_END_TIMEOUT_MS (ten seconds). If the API answers neither the create nor the DELETE within that, the session may still be live when that operation returns, and the API ends it by its idle timeout (idleTimeoutS) or lifetime. Under a plain turn() (no run) the tools share one session that only the API’s idle timeout closes. The model acts on the page by the element refs in the page tree it gets back; a stale observation is refreshed once, a ref that is gone is answered with the page read again, and the tree is cut at BROWSER_TREE_MAX_CHARS characters and BROWSER_MAX_REFS refs. A failure comes back to the model as text it can act on, not as a failed run. Parameters

Parameter	Type
`options`	`BrowserToolsOptions`

Returns ToolSet

Over Agent Browser

browserTools() is exported from @telnyx/agent-harness/tools, from 0.1.0-beta.3. The tools call Agent Browser through transport, the Telnyx API client; in a deployed function that is the TELNYX binding.

import { createHarness } from "@telnyx/agent-harness";
import { browserTools } from "@telnyx/agent-harness/tools";

const harness = createHarness({
  ports,
  model,
  limits: { steps: 8 },
  tools: {
    ...browserTools({ transport: env.TELNYX }),
  },
});

Tool	Effect
`browser_open`	Opens a URL and returns the page. Opening the URL the session is on reads the page again without reloading it.
`browser_click`	Clicks an element by its ref and returns the resulting page.
`browser_fill`	Types a value into an input by its ref, replacing its content. The value is never logged.
`browser_select`	Chooses an option of a dropdown by its ref, by the option’s value or label.
`browser_press`	Presses one of `BROWSER_KEYS`. `Enter` submits the focused form.
`browser_scroll`	Scrolls by `dy` pixels, from `-5000` to `5000`, and returns what is in view.
`browser_screenshot`	Takes a screenshot and returns its `artifactId`, `downloadUrl`, `contentType` and `bytes`. The model does not see the image.

A page is a BrowserPage: the URL, the title, the element refs (e13: link "Learn more") and the text tree, cut at 20,000 characters and 300 refs. A stale observation is refreshed once. A ref that is gone is answered with the page read again. A failure, such as a link to another site or a site that refuses to load, comes back to the model as text, not as a failed run.

Sessions

Per run. Each durable run holds at most one session, keyed by the run’s id. The tools learn the run from the harness; the host wires nothing. A run paused for approval keeps its session until it ends.
Per site. The session’s domain policy is the host of the first URL the model opens and its subdomains, plus the bare host when the URL starts with www. (see siteDomains()). Opening another site closes the session and opens one for that site. browser_open waits up to 45 seconds for a new session to be ready.
Cleanup. The session is closed when the run ends: completed, failed or canceled. The harness operation that ended the run waits for the close for at most 10 seconds, the harness’s run-end timeout. If the API does not answer within it, the operation returns with the session possibly live, and the session’s idle timeout (idleTimeoutS) or lifetime (lifetimeS) ends it.
Reopen. A session the platform ended is reopened once per run.
Without a run. Under a plain turn() the tools share one session, which only its idle timeout closes.
Budget. A run takes at most actions browser actions, 40 by default.

BrowserToolsOptions

Properties actions?

optional actions?: number

How many actions a run may take. Default DEFAULT_BROWSER_ACTIONS. idleTimeoutS?

optional idleTimeoutS?: number

How long a session lives without an action, in seconds. Default BROWSER_IDLE_TIMEOUT_DEFAULT_S. lifetimeS?

optional lifetimeS?: number

How long a session may live, in seconds. Default BROWSER_LIFETIME_DEFAULT_S. log?

optional log?: (line) => void

Where the tools’ log lines go: one per call with the tool, its URL or ref and the time it took, and one per session opened or closed. Default: each line as JSON on standard output. Never the value of a fill. Parameters

Parameter	Type
`line`	`Record`<`string`, `unknown`>

Returns void now?

optional now?: () => number

The clock, for tests. Returns number sleep?

optional sleep?: (ms) => Promise<void>

The pause between readiness polls, for tests. Parameters

Parameter	Type
`ms`	`number`

Returns Promise<void> transport

transport: BrowserTransport | (() => BrowserTransport)

The Telnyx API client the tools call, or a function returning it. The function form is called on every call, so a host can resolve the client lazily and throw its own message when there is none; the message reaches the model as the tool’s answer.

BrowserTransport

The slice of a Telnyx API client the browser tools call. Paths are relative to the API’s /v2; the Telnyx client bound to a deployed function has all three methods, and a test’s fake declares them. Methods delete()

delete<T>(path): PromiseLike<T>

Type Parameters

Type Parameter
`T`

Parameters

Parameter	Type
`path`	`string`

Returns PromiseLike<T> get()

get<T>(path, opts?): PromiseLike<T>

Type Parameters

Type Parameter
`T`

Parameters

Parameter	Type
`path`	`string`
`opts?`	{ `query?`: `Record`<`string`, `unknown`>; }
`opts.query?`	`Record`<`string`, `unknown`>

Returns PromiseLike<T> post()

post<T>(path, opts?): PromiseLike<T>

Type Parameters

Type Parameter
`T`

Parameters

Parameter	Type
`path`	`string`
`opts?`	{ `body?`: `unknown`; `query?`: `Record`<`string`, `unknown`>; }
`opts.body?`	`unknown`
`opts.query?`	`Record`<`string`, `unknown`>

Returns PromiseLike<T>

BrowserPage

The page as the model sees it after an action. Properties refs

refs: Record<string, string>

The element refs the model may act on: e13: link "Learn more". title

title: string

tree

tree: string

truncated

truncated: boolean

url

url: string

BrowserObservation

A page as the API observes it. Properties observation_id?

optional observation_id?: string

refs?

optional refs?: Record<string, { name?: string; role?: string; }>

title?

optional title?: string

tree?

optional tree?: string

truncated?

optional truncated?: boolean

url?

optional url?: string

browserToolNames()

browserToolNames(): string[]

Returns string[]

BROWSER_TOOL_NAMES

const BROWSER_TOOL_NAMES: readonly ["browser_open", "browser_click", "browser_fill", "browser_select", "browser_press", "browser_scroll", "browser_screenshot"]

The names browserTools returns, so a host can list them or name them in an approval policy.

BrowserToolName

BrowserToolName = typeof BROWSER_TOOL_NAMES[number]

BROWSER_KEYS

const BROWSER_KEYS: readonly ["Enter", "Tab", "Escape", "Backspace", "ArrowUp", "ArrowDown", "ArrowLeft", "ArrowRight", "PageUp", "PageDown", "Home", "End"]

The keys browser_press takes.

siteDomains()

siteDomains(hostname): string[]

The session policy for a site: the host and its subdomains, and the bare host with its subdomains when the model opened www., since a site’s links go between the two. No guessed apex: without the public suffix list a guess is either too wide (*.github.io for one site) or wrong (co.uk), so a link to a sibling host is a browser_open away. Parameters

Parameter	Type
`hostname`	`string`

Returns string[]

trimPage()

trimPage(obs): BrowserPage

The observation within the model’s budget: the tree cut at BROWSER_TREE_MAX_CHARS, refs as one line each, at most BROWSER_MAX_REFS. Parameters

Parameter	Type
`obs`	`BrowserObservation`

Returns BrowserPage

DEFAULT_BROWSER_ACTIONS

const DEFAULT_BROWSER_ACTIONS: 40 = 40

How many actions a run may take by default.

BROWSER_LIFETIME_DEFAULT_S

const BROWSER_LIFETIME_DEFAULT_S: 900 = 900

The API’s defaults for a session’s lifetime and idle timeout, in seconds.

BROWSER_IDLE_TIMEOUT_DEFAULT_S

const BROWSER_IDLE_TIMEOUT_DEFAULT_S: 300 = 300

BROWSER_TREE_MAX_CHARS

const BROWSER_TREE_MAX_CHARS: 20000 = 20_000

How much of a page’s tree one observation hands the model, and how many refs.

BROWSER_MAX_REFS

const BROWSER_MAX_REFS: 300 = 300

Get Started

Inside the Managed Harness

The Agent Harness SDK

The Agent Runtime SDK

The Agent Client SDK

browserTools()

Over Agent Browser

Sessions

BrowserToolsOptions

BrowserTransport

BrowserPage

BrowserObservation

browserToolNames()

BROWSER_TOOL_NAMES

BrowserToolName

BROWSER_KEYS

siteDomains()

trimPage()

DEFAULT_BROWSER_ACTIONS

BROWSER_LIFETIME_DEFAULT_S

BROWSER_IDLE_TIMEOUT_DEFAULT_S

BROWSER_TREE_MAX_CHARS

BROWSER_MAX_REFS

​Over Agent Browser

​Sessions

​BrowserToolsOptions

​BrowserTransport

​BrowserPage

​BrowserObservation

​browserToolNames()

​BROWSER_TOOL_NAMES

​BrowserToolName

​BROWSER_KEYS

​siteDomains()

​trimPage()

​DEFAULT_BROWSER_ACTIONS

​BROWSER_LIFETIME_DEFAULT_S

​BROWSER_IDLE_TIMEOUT_DEFAULT_S

​BROWSER_TREE_MAX_CHARS

​BROWSER_MAX_REFS

Over Agent Browser

Sessions

BrowserToolsOptions

BrowserTransport

BrowserPage

BrowserObservation

browserToolNames()

BROWSER_TOOL_NAMES

BrowserToolName

BROWSER_KEYS

siteDomains()

trimPage()

DEFAULT_BROWSER_ACTIONS

BROWSER_LIFETIME_DEFAULT_S

BROWSER_IDLE_TIMEOUT_DEFAULT_S

BROWSER_TREE_MAX_CHARS

BROWSER_MAX_REFS