Prerequisites
Before running an A/B test, make sure the following are in place:
Your Tracify Meta dataset (pixel) is created and active. If you haven't set this up yet, see our Dataset Setup Guide.
Tracify is sending conversion data to your dataset. You can verify this in Events Manager by checking that purchase events are arriving.
Your dataset has received data for at least one week. This gives Meta's algorithm a baseline of conversion data to work with before you start testing.
How the Test Works
You run two campaigns side by side.
One optimizes on your existing dataset (whether that's a direct Meta dataset or another provider's).
The other optimizes on the Tracify dataset.
After enough conversions have come in, you compare cost per acquisition across the two to see which dataset is giving Meta better optimization signal.
The key variable being tested is which conversion data source Meta's algorithm uses to optimize delivery. Everything else - creative, audiences, budgets - should stay as similar as possible. Both campaigns should optimize on the same event (all-purchases); the dataset is the only thing that changes between them.
Important: We recommend comparing conversions directly in Tracify for most accurate reporting (on both sides of the test). If you choose to also review Meta conversions use Meta's 7d click attribution. Avoid view attribution which models and inflates conversions.
Choosing Your Test Method
There are four ways to run this test. Each has different tradeoffs in terms of rigor, effort, and budget requirements.
A note on cost:
Options 1 and 2 both require running two campaigns at the same time, which means your total budget for the test is roughly double what you'd normally spend on a single campaign. For example, if you'd normally run one campaign at $50/day, you'll need $100/day during the test to keep both campaigns funded equally.
Options 3 and 4 have no additional cost - you're simply redirecting your existing budget.
Option 1: Meta's Experiments Tool
Meta has a built-in A/B testing feature in Ads Manager under Experiments. You select two campaigns or ad sets and Meta handles the rest.
How it works:
Create two campaigns that are identical in every way - same creative, same audience targeting, same daily budget.
Set Campaign A to optimize conversions on your existing dataset.
Set Campaign B to optimize conversions on the Tracify dataset.
In Ads Manager, go to Experiments > A/B Test and select both campaigns.
Set the test duration based on the tier you've chosen (see below).
Meta handles the audience split and reports a winner with a confidence percentage.
You can also use existing campaigns rather than creating new ones - Meta will automatically shift delivery to prevent audience overlap during the test period.
Strengths:
Cleanest methodology - Meta guarantees no audience overlap between campaigns, so each user only sees ads from one side of the test.
Built-in reporting with a confidence percentage, so you don't need to do your own analysis.
Can use existing campaigns instead of building new ones from scratch.
Weaknesses:
Requires enough budget per campaign for Meta to optimize effectively (see Budget Considerations below).
Less flexibility - you can't easily adjust the test once it's running.
Note: Meta's declared winner is computed on Meta's own data. To judge on new-customer performance, cross-reference your NC numbers in Tracify against the campaign Meta picks (see What to Measure).
Option 2: Manual Campaign Split
Create two separate campaigns yourself and point each at a different dataset. This gives you more control but requires more care in setup.
Strengths:
Full control over campaign structure, budgets, and timing.
Easier to start small and scale up.
Works well if you want to test within specific audience segments or geos.
Weaknesses:
Both campaigns may bid on the same users, which can muddy results.
Requires manual analysis to compare performance.
You need to actively manage audience separation (e.g. geographic splits or exclusion audiences).
If using this approach, keep budgets equal between campaigns.
Option 3: Before/After Comparison
Switch your existing campaigns to optimize on the Tracify dataset and compare performance to the prior period. No additional budget required.
Strengths:
Zero incremental spend - you're just redirecting your existing budget.
Simple to execute and requires no campaign duplication.
Good option for smaller accounts where splitting budget isn't practical.
Weaknesses:
Not a fully controlled test. Seasonality and creative fatigue or other market factors can affect results.
Works best as a directional signal.
If using this approach, compare at least 4 weeks before vs. 4 weeks after, and avoid making major changes to creative or audiences during the evaluation window.
Option 4: Campaign Budget Optimization (CBO) with Two Ad Sets
Run one Campaign Budget Optimization (CBO) campaign containing two ad sets:
one optimizing on your existing Meta dataset
the other on the Tracify dataset.
Instead of funding each side equally, you let Meta's algorithm decide how to distribute spend across the two ad sets, then use Tracify data to verify whether the algorithm's allocation decisions were correct.
Unlike Option 3, this runs both datasets at the same time, so it avoids the seasonality and creative-fatigue confounds of a before/after comparison. The tradeoff is that spend is unequal by design.
How it works:
Create one CBO campaign with a single overall budget.
Add two ad sets that are identical in creative and audience targeting. Set one to optimize on your existing dataset and the other on the Tracify dataset.
Set a minimum spend per ad set so neither side gets starved entirely, but otherwise let the algorithm allocate budget freely.
Let it run, then use Tracify data to check whether Meta directed more spend toward the ad set that was actually performing better.
Strengths:
No incremental budget required - this works well for brands with budget constraints.
The algorithm's allocation itself becomes a signal: if Meta consistently shifts spend toward the Tracify-optimized ad set, that's evidence the Tracify dataset is giving it a better signal.
Simple to set up.
Weaknesses:
Not a balanced test. Because spend is unequal by design, you can't make a straightforward CPA comparison the way you would with equally funded campaigns.
The two ad sets compete for the same budget and may bid on overlapping users, so results are directional rather than conclusive.
Minimum spend floors mean accumulated spend per ad set can be lower than the total CBO budget, with the remainder distributed wherever Meta finds it most efficient - so the split won't be an exact 50/50.
Best for: brands with budget constraints that want a low-effort, directional read rather than a rigorously controlled test.
How Many Conversions Do You Need?
The reliability of your test depends on how many conversions each campaign generates.
For this it is important to count conversions using Tracify's incremental attribution numbers, not Meta's reported data.
Meta's in-platform numbers include modeled and view-through conversions that inflate the count, and each dataset will report differently in Events Manager - which makes Meta's own figures an unreliable basis for comparison.
Tracify's incremental attribution gives you a consistent, like-for-like measure across both campaigns.
We recommend thinking in terms of two levels:
Level 1: Directional Read - 50 conversions per campaign
This is the minimum needed for Meta's algorithm to actually optimize on each dataset. At this level, you can observe whether the Tracify campaign is directionally outperforming and whether Meta is able to use the Tracify signal effectively.
What it tells you: Whether Meta can optimize on Tracify's signal, and whether there's a large performance difference.
What it doesn't tell you: Small or moderate improvements won't be visible at this volume. Only large differences will stand out clearly from the noise.
Total conversions needed: ~100 across both campaigns (50 for A / 50 for B).
Typical duration: 2-4 weeks depending on your conversion volume.
Best for: A quick directional check before committing to a longer evaluation.
Level 2: Business Decision - 100 conversions per campaign
This is the level we recommend for making a confident decision about switching. With 100 conversions per campaign, you have enough data to detect a meaningful CPA improvement (~20%) with reasonable confidence. This aligns well with Meta's Experiments tool, which reports results at 90% confidence by default.
What it tells you: Whether Tracify delivers a meaningful, repeatable CPA improvement.
Total conversions needed: ~200 across both campaigns.
Typical duration: 2-3 weeks depending on your conversion volume.
Best for: Making a real decision about which dataset to use going forward.
Estimating Your Test Duration
To figure out how long your test will take:
Look at your average weekly purchases (conversions) across the campaigns you plan to test.
Divide that number by 2 (since conversions will be split across two campaigns).
Divide your target conversions per campaign by that weekly number.
Example: If you typically get 100 purchases per week and you're aiming for Level 2 (100 per campaign), each campaign will generate roughly 50 conversions per week. That means about 2 weeks to reach 100 per campaign.
If the math says your test would take longer than 8 weeks, consider using Level 1 or the before/after approach instead. Extended tests are more vulnerable to external factors changing over time.
What to Measure
Primary metric: New Customer CPA (NC CPA)
An attribution dataset's real value shows up in how efficiently it helps Meta acquire new customers, not in re-counting purchases from existing ones. Blended CPA can mask the real difference by lumping returning buyers in with new ones, so we recommend judging the test on new-customer metrics.
NC CPA directly answers the question the test exists to answer: "Which dataset helps Meta find new customers more efficiently?"
Tracify exposes new-customer numbers directly, so you can read NC CPA, NC ROAS, and NC share without deriving them yourself.
Secondary metrics: NC ROAS and NC share
NC ROAS tells you the revenue efficiency of new-customer acquisition; it's useful but noisier because it's affected by order value variation. NC share - the proportion of conversions that are new rather than returning customers - tells you whether a dataset is genuinely shifting delivery toward new customers. If a dataset wins on NC share as well as NC CPA, that's a strong signal.
Also worth watching: blended CPA and ROAS
Keep an eye on blended figures for context, but don't make the switch decision on them - they can look favorable while hiding weak new-customer performance.
Best Practices
Define Success Criteria Before You Start
Decide upfront what would constitute a meaningful result, and define it against a specific KPI. We recommend NC CPA. For example: "If the Tracify campaign achieves an NC CPA that's 10% or more lower than the existing dataset, we'll switch."
Having this defined in advance prevents second-guessing after the fact.
Keep Everything Else Constant
The test is only valid if the dataset is the only variable. During the test period:
Don't change creative on one campaign but not the other.
Don't adjust budgets unevenly.
Don't modify audience targeting mid-test.
Avoid launching major promotions or sales that might skew one campaign's results.
(Note: in Option 4, unequal spend is expected by design - the algorithm sets the split, so this doesn't apply to that method.)
Give Meta Time to Learn
When a new campaign starts optimizing on an unfamiliar dataset, Meta's algorithm needs time to calibrate. The first few days of data may look erratic - that's normal. Plan for at least a week of warm-up before drawing conclusions.
Budget Considerations
Each campaign in the test needs enough daily budget to generate conversions consistently. A rough rule of thumb: your daily budget per campaign should be at least 2-3x your expected CPA. If your typical CPA is $30, aim for at least $60-90/day per campaign. Below this, Meta's algorithm may stay stuck in learning mode, which would unfairly disadvantage whichever dataset it's less familiar with.
A Note on Event Differences
Depending on your setup, the Tracify dataset and your existing dataset may not receive identical events - for example, there may be differences in event volume, parameters, or timing. This is expected and is part of what the test evaluates: which data source, as configured, gives Meta the best optimization signal for your business.
After the Test
If the Tracify campaign wins: You can confidently migrate your campaigns to optimize on the Tracify dataset. We recommend transitioning campaign by campaign rather than all at once, so you can monitor stability.
If results are close or inconclusive: Consider extending the test to accumulate more conversions, or evaluate secondary metrics like NC ROAS and NC share for tiebreakers.
If the existing dataset wins: Reach out to our team. This may indicate a configuration issue or an event mapping gap where further tuning could improve results.