Experiment Memory Guide: Shared by Your Team and AI Agents

A familiar failure

The test ran. The learning disappeared.

Imagine a social-proof test that moved pricing-page trial starts from 4.8% to 5.1%, but ended before reaching the planned sample. Six months later, someone proposes the same test because the inconclusive result, evidence, and follow-up decision live in three different places. This is an illustrative example, not a customer result.

A useful experiment record would surface the hypothesis, owner, target metric, actual result, evidence, learning, and next step before the duplicate enters the backlog.

Shared by people and agents

One memory, whoever does the work

An AI agent forgets everything between conversations, and a teammate cannot read your chat history. Experiment memory sits outside both: every person and every connected agent reads the same record before proposing a test, and writes the outcome back to it under the same rules.

Working alone with an AI agent

Your agent starts each conversation knowing what you already tested, what won, and why. You decide; the agent keeps the record current without you rewriting notes.

Working with a team

People and agents add ideas, score them, and record outcomes in the same place. Whoever asks next, a colleague or an agent, gets the same answer with its evidence.

Definition and boundaries

Four things that are easy to confuse

Chat history

Useful for: Preserves a conversation in one client.

Not sufficient because: It is not a canonical team record, and a later agent may not see it.

Free-form document

Useful for: Captures context and narrative with little setup.

Not sufficient because: Fields, states, and evidence drift, so reliable comparison and querying become difficult.

Experiment database

Useful for: Keeps structured hypotheses, scores, owners, states, outcomes, and learnings.

Not sufficient because: It still needs disciplined updates and a defined permission model.

Execution tool

Useful for: Runs or measures tests in a delivery or analytics system.

Not sufficient because: It does not automatically become the cross-tool memory of why a test existed or what was learned.

What the next person or agent needs

From a promising idea to a reusable learning

A complete record answers nine questions without making the next person reconstruct the test from chat, dashboards, and slide decks.

Experiment and proposed change

A stable name and concise description of what the team wants to test.

Hypothesis

A falsifiable prediction connecting a change, an audience, a metric, and an expected effect.

Priority

The ICE, PIE, RICE, or custom score, with the evidence behind the team's ratings.

Owner and status

Who is accountable and where the experiment sits from Idea to Learned.

Success and failure

The target metric, baseline, and thresholds that decide whether the test won, lost, or stayed neutral.

Variants

The control and each variant that ran, and which one won, so nobody reruns the arm that already lost.

Actual result

What happened, with the named metric and the observed change written in plain language.

Evidence and learning

Where the analysis lives and what the team can reuse in a future decision.

Next step

The rollout, follow-up test, or stop decision made possible by the result.

Lifecycle

The record changes as the experiment does

Frame

Write the idea, hypothesis, target segment, metric, baseline, and failure criterion before scoring.

Prioritize

Choose ICE, PIE, RICE, or a custom framework; record the score and the evidence behind estimates.

Prepare

Assign an owner, define an action plan, and move through approved lifecycle states.

Run and analyze

Keep dates and status current and log interim readings as dated comments; execution and measurement remain in the tools that perform them.

Close the loop

Record result, actual result, evidence, learning, and next step before calling the record complete.

Synthetic example

The same idea before and after a usable record

This example is illustrative; it is not a customer result.

Before

“Try social proof on pricing.”

There is no named audience, baseline, success condition, owner, evidence requirement, or place to record what happened.

Structured record

Hypothesis: If first-time pricing visitors see two role-matched customer quotes, trial starts will increase because the page answers fit concerns.
Success metric: Pricing-page visitor to trial-start conversion; compare against the recorded baseline.
Variants: A: current pricing page, the control. B: two customer quotes above the plans.
Framework and state: ICE, score recorded, owner assigned, Ready.
Closure rule: Store actual result, evidence, learning, next step and the winning variant before moving to Learned.

With a connected AI agent

Reuse the learning without handing over the decision

AI agents no longer forget: most of them ship with memory. What they still do is conclude. Ask one how the test went and it will give you an answer, measured or not. Connected through MCP, an agent can find prior experiments, add ideas, update the record, score in your framework, log interim readings, and save outcomes with the winning variant. What it cannot do is close an experiment without a recorded result and learning: the same rule your team follows. It acts only inside the projects and permissions you grant.

You keep control: the agent cannot manage members, roles, billing or admin controls, and it can delete a project or experiment only if you grant the separate delete scope at connection. It can read the latest 50 comments and add dated comments, but cannot edit or delete them or browse files. It can save evidence URLs, paths, or external IDs, but it does not run or measure an A/B test and does not synchronize Notion, Slack, or spreadsheets on its own.

Connecting an agent? Review the exact tools and limits

Operational checklist

Before a record becomes memory

The hypothesis names the change, audience, expected effect, and reason.
The target metric, success condition, and failure condition are set before execution.
Every score can be traced to assumptions or evidence, not only a number.
The owner and lifecycle state reflect reality.
The actual result links to evidence and distinguishes observation from interpretation.
The learning is reusable, and the next step is explicit.
A connected AI agent has access only to the projects and actions it needs.