How PostHog turns every release into a measurable question, and why almost no one in growth does the same

A brand changes the checkout copy, raises the free-shipping threshold from $15,000 to $20,000, and that same week the ad platform adjusts its bidding algorithm. Conversion drops three points. Which of the three moved it? At most companies, the answer gets decided in a meeting, by whoever argues most convincingly. Nobody built the mechanism that would let them answer that question with data, so they answer it with intuition and call it analysis.

That's the problem solved, or attempted, by the other half of what we already wrote about PostHog's self-driving mode. That note covered who proposes a change and who approves it: agents that watch product signals, build a pull request, and a human who reviews it before it ships. What we left out on purpose was the part that keeps that loop from being a blind bet: the mechanism that decides, after the change ships, whether it worked. Without that judge, approving an agent's PR is an act of faith with extra steps.

A flag isn't a developer trick, it's a way of deciding

PostHog's documentation is explicit about a distinction most teams don't make in practice: deploy is not the same as release. Deploy is when the new code sits in production. Release is when someone experiences it. A feature flag separates the two: the code can be live in production and visible to only 5% of traffic, with the option to ramp that percentage up gradually or switch it off entirely without touching the code again. PostHog puts it plainly: when something breaks, you turn it off; when it works, you expand it and measure the impact.

That turns every launch into something reversible by design, not an all-or-nothing event. But a flag on its own only controls exposure. What connects it to the question of whether something worked is the experiment built on top of it.

Creating an experiment in PostHog is a three-step flow, and the first step already forces something most teams skip: writing the hypothesis before launching. The experiment sits on top of a feature flag, which PostHog can generate automatically or reuse an existing one if it already has at least two variants, one called "control." The second step splits traffic across variants, with up to nine test variants in addition to control. The third is the one that matters for this note: that's where you define which event marks a user as "exposed" to the change, and add the primary metrics (the ones that decide whether the hypothesis holds) and the secondary ones (the ones that flag whether the change broke something nobody was watching). Only events that happen after exposure count toward the result, so the experiment can't be contaminated by behavior that predates the change.

The metrics themselves aren't a single type. PostHog supports funnel metrics (a sequence of steps, typical for onboarding or checkout), mean metrics (average revenue per user, for instance), ratio metrics (revenue per purchase, items per session), and retention metrics. Which metric type you pick is, in practice, the most important decision in the experiment, because it's what determines what counts as "worked."

Statistics you can read without a translator

What most sets PostHog apart from other experimentation tools is how it presents the result. By default it uses Bayesian statistics instead of frequentist: instead of a p-value you have to interpret against an arbitrary significance threshold, the system calculates the probability that one variant beats another and states it in plain language, along the lines of "there's a 96% probability that variant B improves conversion." It also offers credible intervals around the estimated effect and, optionally, CUPED to reduce variance when metrics are noisy.

It's as much a product decision as a statistical one: a "probability to win" number reads faster in a founders' meeting than a 95% confidence interval. But easier to read isn't the same as certain. An 87% probability that something works still leaves a 13% chance it doesn't, and a person still has to make that call, not the dashboard. Bayesian statistics doesn't remove human judgment from the gate, it makes that judgment more accessible.

What the gate doesn't solve on its own

PostHog also offers no-code experiments, still in beta: you edit text, buttons, or styles from a visual toolbar without touching the repository. The documentation itself is honest about the limit: it works well on server-rendered sites with simple text or formatting changes, and it isn't recommended for single-page applications with heavy re-rendering, where the visual change can get lost. It isn't a promise that anyone can build an experiment without engineering; it's a tool for a narrow subset of changes.

The other limit is pricing, and it's worth keeping in mind before assuming this is free forever: feature flags are billed per request processed (starting at $0.0001 per request, with tiers that drop as volume grows), and experiments are billed together with flags, not separately. The free plan includes one million flag requests a month, which covers a fair amount, but the model is usage-based, not flat, so it scales with traffic just like any other infrastructure cost.

The discipline paid media is missing

Almost no growth or paid media team has this reflex. The ad copy changes, the new creative goes up, the bid gets adjusted, and it all happens the same week with no flag, no control variant, no primary metric declared beforehand. When the number moves, the cause gets pinned on whatever change someone happens to remember making. That isn't measurement, it's selective memory with a chart attached.

That's why the PostHog pattern is worth studying regardless of whether you use that specific tool: the idea that every release carries a metric and an explicit way of reading it, defined before launch, is what later lets you accept a proposed change with some confidence, whether it comes from a senior analyst or from an agent that generated it from product data. That's the thread connecting this note to self-driving mode: an agent can propose with all the discipline in the world, but if nobody built the gate that measures whether the change worked, the human approval in the middle isn't evaluating anything, it's just signing off.

One question remains open that we couldn't verify against the public documentation: how PostHog's Bayesian approach behaves in experiments with small samples and seasonal traffic, the scenario typical of a paid media account in LATAM rather than a SaaS product generating millions of events a week. It's a point we recommend validating case by case before assuming the same confidence threshold applies equally in both contexts.