Experiments
Buzz supports the concept of Experiments, which allow you to split the exposed audience into groups in order to test various iterations of targeting and creatives. This technique is also sometimes called "A/B Testing" or "Test/Control Methodology". This article explains the concepts of how Experiments are implemented in the Buzz API.
Experiments Overview
It is a well-established technique in advertising to split the exposed audience into multiple distinct groups in order to measure effectiveness. In digital advertising, these groups are determined by cookies or mobile identifiers. The placement of a user into a group is deterministic and never changes, so you don't accidentally expose a user in one group to a test from another group. However, the group placement is arbitrary such that the membership of any user into any group is unbiased.
Beeswax has implemented this concept is by assigning every inbound auction request with an identifier ranging from 1-1000, with the special value of zero indicating the user had no usable identifier.
Buzz Implementation
In Buzz, Experiments are implemented through a two-level hierarchy:
- A
Test Planis associated with a Campaign and represents a static way of segmenting users for experiments. - Every Line Item m](doc:line-items) within that Campaign is assigned a
Test Group, which represents the specific ranges of user identifiers such that the line serves only to a segregated group. For example, a Test Group might include all users from 1-50, representing 5% of the available users. There is a new API endpoint, campaign_experiment_assignment, that allows you to associate all Line Items within a Campaign to their respective Test Groups in one POST.
The available Test Plans are described below:
| test_plan_id | Test Plan | Test Groups (with IDs) |
|---|---|---|
| 1 | Five Percent Hold-out Plan | Control Group (5%) (ID: 1) Test Group (95%) (ID: 2) |
| 2 | Ten Percent Hold-out Plan | Control Group (10%) (ID: 3) Test Group (90%) (ID: 4) |
| 3 | Twenty Percent Hold-out Plan | Control Group (20%) (ID: 5) Test Group (80%) (ID: 6) |
| 4 | Two-Way Split Plan | Group A (50%) (ID: 7) Group B (50%) (ID: 8) |
| 5 | Three-Way Split Plan | Group A (33.3%) (ID: 9) Group B (33.3%) (ID: 10) Group C (33.4%) (ID: 11) |
| 6 | Ten-Way Split Plan | Group A (10%) (ID: 12) Group B (10%) (ID: 13) Group C (10%) (ID: 14) Group D (10%) (ID: 15) Group E (10%) (ID: 16) Group F (10%) (ID: 17) Group G (10%) (ID: 18) Group H (10%) (ID: 19) Group I (10%) (ID: 20) Group J (10%) (ID: 21) |
| 7 | Three-Way Split with Fifty Percent Hold-Out Plan | Control (50%) (ID: 22) Group A (25%) (ID: 23) Group B (25%) (ID: 24) |
| 8 | Twenty Five Percent Hold-out Plan | Control Group (25%) (ID: 25) Test Group (75%) (ID: 26) |
A list of active Test Plans may be retrieved at any time using the View method:
curl -X GET "https://[host].api.beeswax.com/rest/test_plan" -b cookies.txt
And Test Groups within a plan may similarly be retrieved:
curl -X GET "https://canary.api.beeswax.com/rest/test_group?test_plan_id=1" -b cookies.txt
Logging and Reporting
Within logging and reporting new fields are available for:
test_plan_idtest_plantest_group_idtest group
Within logs, the experiment_user_index field is provided, which indicates the persistent index of the user, from 0-1000.
Bidding Agents and Augmentors
For Bidding Agents and Augmentators the user's persistent experiment_user_index is provided.
Updated over 4 years ago