Experiments (A/B tests)

An experiment shows different versions of a web widget to different visitors and stamps every event and conversation with the version each visitor saw. You can test anything in the widget’s settings, such as the launcher, the theme, the suggested questions and teaser, or the welcome text. You can also send each variant’s conversations to a different published version of the agent. One experiment can cover several widgets. A brand with one widget per region can test one hypothesis on all of them at once, and each widget keeps its own settings underneath the change.

Experiments are available on web widgets. Phone numbers and email addresses are not supported yet.

How it works

Variants. Every experiment has a control variant, which shows each widget exactly as it is configured, plus one or more variants of your own. A variant only lists what it changes. Everything else comes from the widget, so later edits to the widget reach every variant.
Weights. Each variant gets a percentage of visitors. The percentages add up to 100. Use an uneven split, such as 90/10, to limit how many visitors see something new.
Assignment. The embed loader assigns each visitor before the launcher appears, by hashing the visitor’s anonymous id. The same browser sees the same variant on every page and every return visit. Clearing site data, a private window or another device counts as a new visitor.
Conversations. When a visitor starts a conversation, the server works out the variant again from the same id and records it on the conversation. A visitor cannot pick a variant.
One at a time. A widget can be in only one running experiment, so every result is attributable to one change.

Creating an experiment

In the dashboard, open a widget and click New experiment in the Experiments box on the General tab, or go to Widgets → Experiments. Pick the widgets to include (they must use the same agent) and create a draft. The draft starts with control and one empty variant. On the experiment’s page, click Edit next to a variant. The editor is the widget editor, with a live preview, showing the widget with the variant’s changes on top:

Change any setting. Each change appears as a chip under Changed from the widget; click its × to put that setting back.
Pick an Agent version to send this variant’s conversations to another published version.
With several widgets, Edit on top of chooses which widget the editor shows. The editor warns when a setting you change differs between the widgets, because the variant’s value replaces each widget’s own.

Save the variant, set the weights under Variants, and click Start. You can do the same with the Experiments API, which needs the web_widgets:manage scope.

curl -X POST https://api.anyreach.ai/core/experiments \
  -H "Authorization: Bearer $TOKEN" \
  -H "X-Anyreach-Org: $ORG_ID" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Shorter suggested questions",
    "hypothesis": "Two short questions and a teaser get more visitors to start a chat.",
    "widget_ids": ["<uk widget id>", "<us widget id>"],
    "variants": [
      { "key": "control", "name": "Control", "weight": 50 },
      {
        "key": "short",
        "name": "Short questions + teaser",
        "weight": 50,
        "overlay": {
          "v1": {
            "suggested_questions": {
              "default": {
                "teaser": { "text": "Questions? I can help." },
                "questions": [{ "text": "Order status" }, { "text": "Returns" }]
              }
            }
          }
        }
      }
    ]
  }'

The experiment starts as a draft. Nothing changes for visitors until you start it:

curl -X POST https://api.anyreach.ai/core/experiments/<id>/start \
  -H "Authorization: Bearer $TOKEN" -H "X-Anyreach-Org: $ORG_ID"

Writing a variant

A variant’s overlay uses the same v1 shape as the widget’s configuration and holds only the keys that change.

In the overlay	Effect
An object	Merged key by key into the widget’s settings.
A value	Replaces the widget’s value.
`null`	Removes the widget’s value, so the built-in default applies. For example, `"teaser": null` hides a teaser.
A list	Replaces the widget’s list, except page rules for suggested questions, which are matched on their `match` pattern. A variant that changes the `/pricing*` rule keeps the widget’s other rules.
`agent_version_id`	Sends this variant’s conversations to another published version of the widget’s agent. Every widget in the experiment must use that agent.

To test a new prompt, publish the new agent version first and point a variant at it. Publishing makes that version the agent’s latest, so anything that uses the agent without naming a version will already use it. Verified identity on the page uses the identity verifiers of the widget’s own agent version. Keep the same verifiers on a variant’s agent version. Variant keys are lowercase letters, digits, - and _. They appear in events and on conversations, so keep them stable once the experiment starts.

Running and ending an experiment

Action	Endpoint	Notes
Start	`POST /core/experiments/{id}/start`	Checks each variant’s agent version is still published, and that every widget is published and collecting analytics. Fails with `409` if a widget is already in another running experiment. An experiment that only swaps the agent version needs a primary metric first.
Pause	`POST /core/experiments/{id}/pause`	Every visitor sees the widgets as configured. Assignments come back unchanged on resume.
Resume	`POST /core/experiments/{id}/resume`
Conclude	`POST /core/experiments/{id}/conclude`	Body `{"winner_key": "short"}`. Keeping `control` changes nothing. Keeping another variant writes its changes, and its agent version, into every widget in the experiment, all or nothing. Add `"dry_run": true` to see what would change first.

The experiment page has the same actions: Start, Pause, Resume and Conclude. Conclude asks which variant to keep and shows what each widget will take before you confirm. Variants and widgets can be edited while the experiment is a draft. Name, hypothesis and primary metric can be edited any time. Only drafts can be deleted, so finished experiments stay as history. From when an experiment starts until it is concluded, paused included, the settings it tests are greyed out in each widget’s editor. The lock is in the editor only: a widget preset the widget follows, or a write through the web widgets API, still changes those settings, and the change reaches every variant. Avoid editing them until the experiment ends. Changes take effect on each visitor’s next page load.

Widgets that follow a preset

If a widget follows a widget preset for a section the winning variant changes, concluding stops that widget following the section. Otherwise the preset would put its own values back. The conclude response lists the sections each widget stopped following. The widget’s other preset sections are untouched.

Measuring results

The experiment’s page shows its results per variant: the widget analytics metrics (visitors, open rate, chat rate, the funnel, suggested-question clicks, conversations, link clicks and cart adds from chat), counted per visitor over the time the experiment ran. An experiment that covers several widgets can show one of them at a time. The page warns when the variants’ visitor counts are further from their weights than chance allows, which means assignment is broken. Experiments that ran before analytics was collected show conversation outcomes instead (GET /core/experiments/{id}/results). Each conversation also carries experiment_id and experiment_variant, which the conversations list accepts as filters. Conversions on your own site (orders, sign-ups) stay in your analytics tool: every host-page event carries experiment_id and variant_key, and the loader emits an exposure event on every page load while the visitor is in an experiment. Count distinct anonymous_id per variant_key there, alongside your own track goals.

Primary metric

Choose the number you’ll judge the experiment by. The choices follow what the variants change: visitor metrics (chat rate, open rate, suggested-question clicks, link clicks, adds to cart) when a variant changes the widget, conversation metrics (messages per conversation, left while waiting, handoffs, time to first reply) when one swaps the agent version, and both when it does both. Chat rate is offered either way. Without a choice the page leads with chat rate; an experiment that only swaps the agent version must choose before it starts.

Decide what counts as success before you start. Small samples move a lot: wait until each variant has enough visitors before you compare them.

API

Method	Path	Purpose
`GET`	`/core/experiments?scope_id={widget_id}`	List experiments, optionally only those covering one widget.
`POST`	`/core/experiments`	Create a draft.
`GET`	`/core/experiments/{id}`	Read one, with its widgets.
`GET`	`/core/experiments/{id}/results`	Conversation outcomes per variant, and per widget and variant.
`GET`	`/core/widget-analytics/experiments/{id}`	Visitor analytics per variant (widget analytics).
`PATCH`	`/core/experiments/{id}`	Edit. Widgets and variants only while a draft.
`DELETE`	`/core/experiments/{id}`	Delete a draft.
`POST`	`/core/experiments/{id}/start` · `/pause` · `/resume`	Change status.
`POST`	`/core/experiments/{id}/conclude`	End and keep one variant.

Reading needs web_widgets:read or web_widgets:manage. Writing needs web_widgets:manage.

Getting Started

Agents

Knowledge Bases

Telephony

Web Widgets

Email

Campaigns

Conversations

Evaluation & Metrics

Agent Assist

Workflows

Organizing Agents & Workflows

Custom Data

Anyreach over MCP

Organizations & Billing

Experiments (A/B tests)

How it works

Creating an experiment

Writing a variant

Running and ending an experiment

Widgets that follow a preset

Measuring results

Primary metric

API

​How it works

​Creating an experiment

​Writing a variant

​Running and ending an experiment

​Widgets that follow a preset

​Measuring results

​Primary metric

​API

How it works

Creating an experiment

Writing a variant

Running and ending an experiment

Widgets that follow a preset

Measuring results

Primary metric

API