# Experiments

An experiment takes decisions your agent already made, runs a candidate function on the same inputs, and compares its result with what was recorded. Use it to check a new model, prompt or rule against real history before you ship it.

**We never run your AI.** The candidate runs on your machine, with your own models and keys. The service only stores and shows the results.

```python
import signafaide

def approve(claim_id, amount):            # same arguments as the recorded tool
    return {"approved": amount < 300}     # your new model or rule

report = signafaide.compare(approve, agent="claims-bot", name="approve_payout", since="2026-11-01", limit=500)
print(report)                             # n, agreed, changed, errors, then the changed decisions
report.upload()                           # optional: show it in the console
```

```ts
import { compare } from "signafaide";

const report = await compare(async (input) => ({ approved: input.amount < 300 }), { agent: "claims-bot", name: "approve_payout", since: "2026-11-01" });
console.log(String(report));
await report.upload();
```

From the command line:

```bash
signafaide compare --module candidate.py:approve --agent claims-bot --name approve_payout --from 2026-11-01 --upload          # add --values to send the values (stored content only)
npx signafaide compare --module ./candidate.mjs:approve --agent claims-bot --upload
```

## How it works

1. The agent's local chain is verified first. If it fails, nothing is compared.
2. Each decision that has a co-signed outcome is taken (filtered by `name`, dates and `limit`, which keeps the newest).
3. The candidate gets the recorded input. In Python an object input is passed as keyword arguments, so a new version of the same function fits. In TypeScript the candidate gets the input as recorded: the single argument, or the argument list.
4. Its result is compared with the outcome's `output`: exact equality of canonical JSON by default, or your own `same(recorded, candidate)`.
5. The report has `n`, `agreed`, `changed`, `errors` and one row per decision: `record_hash`, recorded value, candidate value, and whether they are the same.

A candidate that raises counts as an error, not a crash.

## Candidates must be side-effect free

A candidate gets inputs only. Compare never writes records, and while a candidate runs, any attempt to record is refused, so a recorded tool called from a candidate (sending money, an email) raises instead of running. Don't call anything that changes the world from a candidate.

## Uploading

`upload()` (or `--upload`) sends the report to the service. It is hash-only by default: record hashes, agreed, changed or error, and the hash of each candidate output. The recorded and candidate values are included only when your organization stores content with the service (`store_content=True`); the service checks the recorded value against the stored outcome and keeps candidate values encrypted like stored content.

In the console, **Review → Experiments** lists uploads with their agreement rate. An experiment's page shows the changed decisions, with a link to each record, and can compare two experiments over the decisions they share. Experiments are on the Team and Enterprise plans.