Experiments

Buzz supports the concept of Experiments, which allow you to split the exposed audience into groups in order to test various iterations of targeting and creatives. This technique is also sometimes called "A/B Testing" or "Test/Control Methodology". This article explains the concepts of how Experiments are implemented in the Buzz API.

Experiments Overview

It is a well-established technique in advertising to split the exposed audience into multiple distinct groups in order to measure effectiveness. In digital advertising, these groups are determined by cookies or mobile identifiers. The placement of a user into a group is deterministic and never changes, so you don't accidentally expose a user in one group to a test from another group. However, the group placement is arbitrary such that the membership of any user into any group is unbiased.

Beeswax has implemented this concept is by assigning every inbound auction request with an identifier ranging from 1-1000, with the special value of zero indicating the user had no usable identifier.

Buzz Implementation

In Buzz, Experiments are implemented through a two-level hierarchy:

  • A Test Plan is associated with a Campaign and represents a static way of segmenting users for experiments.
  • Every Line Item m](doc:line-items) within that Campaign is assigned a Test Group, which represents the specific ranges of user identifiers such that the line serves only to a segregated group. For example, a Test Group might include all users from 1-50, representing 5% of the available users. There is a new API endpoint, campaign_experiment_assignment, that allows you to associate all Line Items within a Campaign to their respective Test Groups in one POST.

The available Test Plans are described below:

test_plan_idTest PlanTest Groups (with IDs)
1Five Percent Hold-out PlanControl Group (5%) (ID: 1)
Test Group (95%) (ID: 2)
2Ten Percent Hold-out PlanControl Group (10%) (ID: 3)
Test Group (90%) (ID: 4)
3Twenty Percent Hold-out PlanControl Group (20%) (ID: 5)
Test Group (80%) (ID: 6)
4Two-Way Split PlanGroup A (50%) (ID: 7)
Group B (50%) (ID: 8)
5Three-Way Split PlanGroup A (33.3%) (ID: 9)
Group B (33.3%) (ID: 10)
Group C (33.4%) (ID: 11)
6Ten-Way Split PlanGroup A (10%) (ID: 12)
Group B (10%) (ID: 13)
Group C (10%) (ID: 14)
Group D (10%) (ID: 15)
Group E (10%) (ID: 16)
Group F (10%) (ID: 17)
Group G (10%) (ID: 18)
Group H (10%) (ID: 19)
Group I (10%) (ID: 20)
Group J (10%) (ID: 21)
7Three-Way Split with Fifty Percent Hold-Out PlanControl (50%) (ID: 22)
Group A (25%) (ID: 23)
Group B (25%) (ID: 24)
8Twenty Five Percent Hold-out PlanControl Group (25%) (ID: 25)
Test Group (75%) (ID: 26)

A list of active Test Plans may be retrieved at any time using the View method:

curl -X GET "https://[host].api.beeswax.com/rest/test_plan" -b cookies.txt

And Test Groups within a plan may similarly be retrieved:

curl -X GET "https://canary.api.beeswax.com/rest/test_group?test_plan_id=1" -b cookies.txt

Logging and Reporting

Within logging and reporting new fields are available for:

  • test_plan_id
  • test_plan
  • test_group_id
  • test group

Within logs, the experiment_user_index field is provided, which indicates the persistent index of the user, from 0-1000.

Bidding Agents and Augmentors

For Bidding Agents and Augmentators the user's persistent experiment_user_index is provided.