Experiments
Buzz supports Experiments, which allow you to split the exposed audience into groups in order to test various iterations of targeting and creatives. This technique is also called "A/B Testing" or "Test/Control Methodology". This article explains the concepts of how Experiments are implemented in the Buzz API.
Experiments Overview
It is a well-established technique in advertising to split the exposed audience into multiple distinct groups in order to measure effectiveness. In digital advertising, these groups are determined by cookies or mobile identifiers. The placement of a user into a group is deterministic and never changes, so you don't accidentally expose a user in one group to a test from another group. However, the group placement is arbitrary such that the membership of any user into any group is unbiased.
Beeswax has implemented this concept is by assigning every inbound auction request with an identifier ranging from 1-1000, with the special value of zero indicating the user had no usable identifier.
Buzz Implementation
In Buzz, Experiments are implemented through a two-level hierarchy:
- A
Test Planis associated with a Campaign and represents a static way of segmenting users for experiments. - Every Line Item within that Campaign is assigned a
Test Group, which represents the specific ranges of user identifiers such that the line serves only to a segregated group. For example, a Test Group might include all users from 1-50, representing 5% of the available users.
The available Test Plans are described below:
| test_plan_id | Test Plan | Test Groups (with IDs) |
|---|---|---|
| 1 | Five Percent Hold-out Plan | Control Group (5%) (ID: 1) Test Group (95%) (ID: 2) |
| 2 | Ten Percent Hold-out Plan | Control Group (10%) (ID: 3) Test Group (90%) (ID: 4) |
| 3 | Twenty Percent Hold-out Plan | Control Group (20%) (ID: 5) Test Group (80%) (ID: 6) |
| 4 | Two-Way Split Plan | Group A (50%) (ID: 7) Group B (50%) (ID: 8) |
| 5 | Three-Way Split Plan | Group A (33.3%) (ID: 9) Group B (33.3%) (ID: 10) Group C (33.4%) (ID: 11) |
| 6 | Ten-Way Split Plan | Group A (10%) (ID: 12) Group B (10%) (ID: 13) Group C (10%) (ID: 14) Group D (10%) (ID: 15) Group E (10%) (ID: 16) Group F (10%) (ID: 17) Group G (10%) (ID: 18) Group H (10%) (ID: 19) Group I (10%) (ID: 20) Group J (10%) (ID: 21) |
| 7 | Three-Way Split with Fifty Percent Hold-Out Plan | Control (50%) (ID: 22) Group A (25%) (ID: 23) Group B (25%) (ID: 24) |
| 8 | Twenty Five Percent Hold-out Plan | Control Group (25%) (ID: 25) Test Group (75%) (ID: 26) |
| 9 | One Percent Hold-out Plan | Control Group (1%) (ID: 27) Test Group (99%) (ID: 28) |
| 10 | Three-Way Split with Ten Percent Hold-Out Plan | Control Group (10%) (ID: 29) Group A (45%) (ID: 30) Group B (45%) (ID: 31) |
| 11 | Three-Way Split with Eighty Percent Hold-Out Plan | Control Group (80%) (ID: 32) Group A (10%) (ID: 33) Group B (10%) (ID: 34) |
| 12 | Three-Way Split with Ninety Percent Hold-Out Plan | Control Group (90%) (ID: 35) Group A (5%) (ID: 36) Group B (5%) (ID: 37) |
| 13 | Four-Way Split Plan | Group A (25%) (ID: 38) Group B (25%) (ID: 39) Group C (25%) (ID: 40) Group D (25%) (ID: 41) |
| 14 | Four-Way Split with Ten Percent Hold-Out Plan | Control Group (10%) (ID: 42) Group A (30%) (ID: 43) Group B (30%) (ID: 44) Group C (30%) (ID: 45) |
| 15 | Thirty Percent Hold-out Plan | Control Group (30%) (ID: 46) Test Group (70%) (ID: 47) |
A list of active Test Plans may be retrieved at any time using the following method:
curl -X GET "https://{buzz_key}.api.beeswax.com/rest/test_plan" -b cookies.txtAnd Test Groups within a plan may similarly be retrieved:
curl -X GET "https://{buzz_key}.api.beeswax.com/rest/test_group?test_plan_id=1" -b cookies.txtLogging and Reporting
Within logging and reporting fields are available for:
test_plan_idtest_plantest_group_idtest group
Within logs, the experiment_user_index field is provided, which indicates the persistent index of the user, from 0-1000.
Bidding Agents and Augmentors
For Bidding Agents and Augmentors the user's persistent experiment_user_index is provided.
Updated 13 days ago