Experiments

An experiment takes decisions your agent already made, runs a candidate function on the same inputs, and compares its result with what was recorded. Use it to check a new model, prompt or rule against real history before you ship it.

We never run your AI. The candidate runs on your machine, with your own models and keys. The service only stores and shows the results.

import signafaide

def approve(claim_id, amount):            # same arguments as the recorded tool
    return {"approved": amount < 300}     # your new model or rule

report = signafaide.compare(approve, agent="claims-bot", name="approve_payout", since="2026-11-01", limit=500)
print(report)                             # n, agreed, changed, errors, then the changed decisions
report.upload()                           # optional: show it in the console

import { compare } from "signafaide";

const report = await compare(async (input) => ({ approved: input.amount < 300 }), { agent: "claims-bot", name: "approve_payout", since: "2026-11-01" });
console.log(String(report));
await report.upload();

From the command line:

signafaide compare --module candidate.py:approve --agent claims-bot --name approve_payout --from 2026-11-01 --upload          # add --values to send the values (stored content only)
npx signafaide compare --module ./candidate.mjs:approve --agent claims-bot --upload

How it works #

1. The agent's local chain is verified first. If it fails, nothing is compared.
2. Each decision that has a co-signed outcome is taken (filtered by name, dates and limit, which keeps the newest).
3. The candidate gets the recorded input. In Python an object input is passed as keyword arguments, so a new version of the same function fits. In TypeScript the candidate gets the input as recorded: the single argument, or the argument list.
4. Its result is compared with the outcome's output: exact equality of canonical JSON by default, or your own same(recorded, candidate).
5. The report has n, agreed, changed, errors and one row per decision: record_hash, recorded value, candidate value, and whether they are the same.

A candidate that raises counts as an error, not a crash.

Candidates must be side-effect free #

A candidate gets inputs only. Compare never writes records, and while a candidate runs, any attempt to record is refused, so a recorded tool called from a candidate (sending money, an email) raises instead of running. Don't call anything that changes the world from a candidate.

Uploading #

upload() (or --upload) sends the report to the service. It is hash-only by default: record hashes, agreed, changed or error, and the hash of each candidate output. The recorded and candidate values are included only when your organization stores content with the service (store_content=True); the service checks the recorded value against the stored outcome and keeps candidate values encrypted like stored content.

In the console, Review → Experiments lists uploads with their agreement rate. An experiment's page shows the changed decisions, with a link to each record, and can compare two experiments over the decisions they share. Experiments are on the Team and Enterprise plans.