> ## Documentation Index
> Fetch the complete documentation index at: https://docs.anyreach.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Experiments (A/B tests)

> Split a widget's visitors between variants of its look, suggested questions or agent version, and measure which one does better.

An experiment shows different versions of a web widget to different visitors and stamps every event and conversation with the version each visitor saw. You can test anything in the widget's settings, such as the launcher, the theme, the [suggested questions and teaser](/web-widgets/text-voice-and-quick-actions#suggested-questions), or the welcome text. You can also send each variant's conversations to a different published version of the agent.

One experiment can cover several widgets. A brand with one widget per region can test one hypothesis on all of them at once, and each widget keeps its own settings underneath the change.

<Note>
  Experiments are available on web widgets. Phone numbers and email addresses are not supported yet.
</Note>

## How it works

* **Variants.** Every experiment has a `control` variant, which shows each widget exactly as it is configured, plus one or more variants of your own. A variant only lists what it changes. Everything else comes from the widget, so later edits to the widget reach every variant.
* **Weights.** Each variant gets a percentage of visitors. The percentages add up to 100. Use an uneven split, such as 90/10, to limit how many visitors see something new.
* **Assignment.** The embed loader assigns each visitor before the launcher appears, by hashing the visitor's anonymous id. The same browser sees the same variant on every page and every return visit. Clearing site data, a private window or another device counts as a new visitor.
* **Conversations.** When a visitor starts a conversation, the server works out the variant again from the same id and records it on the conversation. A visitor cannot pick a variant.
* **One at a time.** A widget can be in only one running experiment, so every result is attributable to one change.

## Creating an experiment

In the dashboard, open a widget and click **New experiment** in the Experiments box on the General tab, or go to **Widgets → Experiments**. Pick the widgets to include (they must use the same agent) and create a draft. The draft starts with `control` and one empty variant.

On the experiment's page, click **Edit** next to a variant. The editor is the widget editor, with a live preview, showing the widget with the variant's changes on top:

* Change any setting. Each change appears as a chip under **Changed from the widget**; click its **×** to put that setting back.
* Pick an **Agent version** to send this variant's conversations to another published version.
* With several widgets, **Edit on top of** chooses which widget the editor shows. The editor warns when a setting you change differs between the widgets, because the variant's value replaces each widget's own.

Save the variant, set the weights under **Variants**, and click **Start**.

You can do the same with the [Experiments API](#api), which needs the `web_widgets:manage` scope.

```bash theme={null}
curl -X POST https://api.anyreach.ai/core/experiments \
  -H "Authorization: Bearer $TOKEN" \
  -H "X-Anyreach-Org: $ORG_ID" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Shorter suggested questions",
    "hypothesis": "Two short questions and a teaser get more visitors to start a chat.",
    "widget_ids": ["<uk widget id>", "<us widget id>"],
    "variants": [
      { "key": "control", "name": "Control", "weight": 50 },
      {
        "key": "short",
        "name": "Short questions + teaser",
        "weight": 50,
        "overlay": {
          "v1": {
            "suggested_questions": {
              "default": {
                "teaser": { "text": "Questions? I can help." },
                "questions": [{ "text": "Order status" }, { "text": "Returns" }]
              }
            }
          }
        }
      }
    ]
  }'
```

The experiment starts as a draft. Nothing changes for visitors until you start it:

```bash theme={null}
curl -X POST https://api.anyreach.ai/core/experiments/<id>/start \
  -H "Authorization: Bearer $TOKEN" -H "X-Anyreach-Org: $ORG_ID"
```

### Writing a variant

A variant's `overlay` uses the same `v1` shape as the widget's configuration and holds only the keys that change.

| In the overlay | Effect |
| - | - |
| An object | Merged key by key into the widget's settings. |
| A value | Replaces the widget's value. |
| `null` | Removes the widget's value, so the built-in default applies. For example, `"teaser": null` hides a teaser. |
| A list | Replaces the widget's list, except page rules for suggested questions, which are matched on their `match` pattern. A variant that changes the `/pricing*` rule keeps the widget's other rules. |
| `agent_version_id` | Sends this variant's conversations to another **published** version of the widget's agent. Every widget in the experiment must use that agent. |

To test a new prompt, publish the new agent version first and point a variant at it. Publishing makes that version the agent's latest, so anything that uses the agent without naming a version will already use it.

[Verified identity](/agents/verified-identity) on the page uses the identity verifiers of the widget's own agent version. Keep the same verifiers on a variant's agent version.

Variant keys are lowercase letters, digits, `-` and `_`. They appear in events and on conversations, so keep them stable once the experiment starts.

## Running and ending an experiment

| Action | Endpoint | Notes |
| - | - | - |
| Start | `POST /core/experiments/{id}/start` | Checks each variant's agent version is still published, and that every widget is published and [collecting analytics](/web-widgets/analytics#turning-it-off). Fails with `409` if a widget is already in another running experiment. An experiment that only swaps the agent version needs a primary metric first. |
| Pause | `POST /core/experiments/{id}/pause` | Every visitor sees the widgets as configured. Assignments come back unchanged on resume. |
| Resume | `POST /core/experiments/{id}/resume` | |
| Conclude | `POST /core/experiments/{id}/conclude` | Body `{"winner_key": "short"}`. Keeping `control` changes nothing. Keeping another variant writes its changes, and its agent version, into every widget in the experiment, all or nothing. Add `"dry_run": true` to see what would change first. |

The experiment page has the same actions: **Start**, **Pause**, **Resume** and **Conclude**. Conclude asks which variant to keep and shows what each widget will take before you confirm.

Variants and widgets can be edited while the experiment is a draft. Name, hypothesis and primary metric can be edited any time. Only drafts can be deleted, so finished experiments stay as history.

From when an experiment starts until it is concluded, paused included, the settings it tests are greyed out in each widget's editor. The lock is in the editor only: a widget preset the widget follows, or a write through the web widgets API, still changes those settings, and the change reaches every variant. Avoid editing them until the experiment ends.

Changes take effect on each visitor's next page load.

### Widgets that follow a preset

If a widget follows a widget preset for a section the winning variant changes, concluding stops that widget following the section. Otherwise the preset would put its own values back. The conclude response lists the sections each widget stopped following. The widget's other preset sections are untouched.

## Measuring results

The experiment's page shows its results per variant: the [widget analytics](/web-widgets/analytics) metrics (visitors, open rate, chat rate, the funnel, suggested-question clicks, conversations, link clicks and cart adds from chat), counted per visitor over the time the experiment ran. An experiment that covers several widgets can show one of them at a time. The page warns when the variants' visitor counts are further from their weights than chance allows, which means assignment is broken. Experiments that ran before analytics was collected show conversation outcomes instead (`GET /core/experiments/{id}/results`).

Each conversation also carries `experiment_id` and `experiment_variant`, which the [conversations list](/api-reference/conversations/overview) accepts as filters.

Conversions on your own site (orders, sign-ups) stay in your analytics tool: every [host-page event](/web-widgets/host-page-events) carries `experiment_id` and `variant_key`, and the loader emits an `exposure` event on every page load while the visitor is in an experiment. Count distinct `anonymous_id` per `variant_key` there, alongside your own `track` goals.

### Primary metric

Choose the number you'll judge the experiment by. The choices follow what the variants change: visitor metrics (chat rate, open rate, suggested-question clicks, link clicks, adds to cart) when a variant changes the widget, conversation metrics (messages per conversation, left while waiting, handoffs, time to first reply) when one swaps the agent version, and both when it does both. Chat rate is offered either way. Without a choice the page leads with chat rate; an experiment that only swaps the agent version must choose before it starts.

<Tip>
  Decide what counts as success before you start. Small samples move a lot: wait until each variant has enough visitors before you compare them.
</Tip>

## API

| Method | Path | Purpose |
| - | - | - |
| `GET` | `/core/experiments?scope_id={widget_id}` | List experiments, optionally only those covering one widget. |
| `POST` | `/core/experiments` | Create a draft. |
| `GET` | `/core/experiments/{id}` | Read one, with its widgets. |
| `GET` | `/core/experiments/{id}/results` | Conversation outcomes per variant, and per widget and variant. |
| `GET` | `/core/widget-analytics/experiments/{id}` | Visitor analytics per variant ([widget analytics](/web-widgets/analytics#api)). |
| `PATCH` | `/core/experiments/{id}` | Edit. Widgets and variants only while a draft. |
| `DELETE` | `/core/experiments/{id}` | Delete a draft. |
| `POST` | `/core/experiments/{id}/start` · `/pause` · `/resume` | Change status. |
| `POST` | `/core/experiments/{id}/conclude` | End and keep one variant. |

Reading needs `web_widgets:read` or `web_widgets:manage`. Writing needs `web_widgets:manage`.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.