Data Feeds

One of the most important considerations in working with an RTB bidder is designing the data pipeline by which auction, bid, and win events are processed and analyzed.

Unlike DSPs, with the Beeswax Bidder-as-a-Service technology you can access the full, unfiltered stream of RTB events, in much the same way you would if you were building a bidder from scratch.

Streaming vs Batch

The first consideration for designing your data pipeline is whether you prefer to get the data in batch form, or as a continuous stream. For the very high-volume data, like auction logs, Beeswax only supports batch data. For win logs (impressions) Beeswax supports both methods, and there are pros and cons to each:

Pipeline Method	Description	Pros	Cons
Batch	Hourly or daily files of data placed in an S3 bucket.	Fairly easy to ingest, fault tolerant	Delay in utilizing data. Also may include many files written per hour.
Stream	Near real-time data in JSON or protobuf format sent over http or to AWS Kinesis	Use data as fast as you can process it	Higher cost and complexity to support data ingestion

Data Definitions

Column definitions, protobuf mapping, field lists and a data dictionary can be found in the is publicly-accessible Microsoft Excel file on Github: Beeswax Log File Header Definitions and Proto Mapping.

Beeswax makes multiple types of data available from Stinger as described in the chart below. Based on your use case you may need some, or all of this data. Because some of this data can be quite large, additional fees may apply (contact your Account Manager for more information).

A more comprehensive description of these log types and implementation details can be found in this publicly-accessible Microsoft Excel file on Github: Beeswax Log Summary.

Data Type	Description	Batch Field Manifest	Column Definitions and Protobuf Mapping
Auctions	The auction request from the exchange, normalized to OpenRTB fields.	auction_log_headers.csv
Bids	The bids returned from the Bidding Agent to the exchange, whether the auction was won or not.	bid_log_headers.csv
Conversions	The conversions recorded by Beeswax	conversion_log_headers.csv
Attributed Conversions	The conversions recorded by Beeswax, attributed back to an auction	attributed_conversion_log_headers.csv
Losses	Loss logs provided by a limited number of exchanges (Google)	bid_response_feedback_logs.csv
Wins	The winning auctions (impressions), clicks, and events (video plays, etc)	win_log_headers.csv	ad_log.proto
Segments	All 1st party and 3rd party segments available on the auction	segment_log_headers.csv