Skip to main content

Monitoring Ad Campaign Performance

Ad campaigns are very crucial to the business and success of a product or the company. The difference in the customer impact created by the advertisements can decide whether a product skyrockets or fails in the market. These high stakes makes it critical to measure the performance of an Ad campaign and take instant measures to improve the performance and make the campaign successful.

One of the important factor here is identifying the issues as early as possible as every aspect is time sensitive and losses are exponential with respect to time. Minimizing the damage by identifying and investigating the issue in realtime is where ThirdEye comes into picture. In this recipe we will try to monitor an Ad campaign data and investigate the anomalies generated by ThirdEye for the anomalous behaviours.

Architecture diagram​

image

Here are the steps to quickly try this out​

  • Data Preparation and Availability
  • Data Ingestion
  • Alert creation and notifications
  • Anomaly reporting
  • Root-cause analysis: anomaly investigation

Data Preparation and Availability​

  • Identify key metrics to monitor. One of the core metrics for learning about ad campaign is:

    • Number of clicks

  • Identify the dimensions that are applicable for a given metric:

    • Country
    • Exchange

  • Decide granularity for your detection:

    • Granularity enables you for real-time or near real-time monitoring (hourly/daily/weekly time series).
AdCampaignSizeAdCampaignTypeCampaignDeliveryTimestampClicksCountryExchangeFailedBidsImpressionsLostBids
SmallDisplay151286760022037.0945USADoubleClick2455.16686784.69042973.6278
SmallDisplay151287120022505.3753USADoubleClick2473.19846839.19052940.1981
SmallDisplay151287480022758.7071USADoubleClick2485.50296909.33463003.7753
SmallDisplay151287840023112.5034USADoubleClick2590.75557158.97252966.8914
SmallDisplay151288200023830.7055USADoubleClick2639.08027235.75742950.0345
SmallDisplay151288560023979.5515USADoubleClick2666.92617176.23413086.7601
SmallDisplay151288920024021.3573USADoubleClick2683.29937427.77442969.4343
SmallDisplay151289280024228.0634USADoubleClick2792.05657568.59633047.8218
SmallDisplay151289640024914.9754USADoubleClick2896.01427854.94663034.1817
SmallDisplay151290000025691.9473USADoubleClick2804.62767685.08933032.0785

Data Ingestion​

  • You can download this CSV file (link).
  • Ingest this CSV data to Pinot using Data Manager (link) or Pinot API.
  • The Pinot schema should look like this:
{
  "schemaName": "AdCampaignData",
  "dimensionFieldSpecs": [
    {
      "name": "AdCampaignSize",
      "dataType": "STRING"
    },
    {
      "name": "Country",
      "dataType": "STRING"
    },
    {
      "name": "AdCampaignType",
      "dataType": "STRING"
    },
    {
      "name": "Exchange",
      "dataType": "STRING"
    }
  ],
  "metricFieldSpecs": [
    {
      "name": "Impressions",
      "dataType": "DOUBLE"
    },
    {
      "name": "Clicks",
      "dataType": "DOUBLE"
    },
    {
      "name": "LostBids",
      "dataType": "DOUBLE"
    },
    {
      "name": "FailedBids",
      "dataType": "DOUBLE"
    }
  ],
  "dateTimeFieldSpecs": [
    {
      "name": "CampaignDeliveryTimestamp",
      "dataType": "LONG",
      "format": "1:SECONDS:EPOCH",
      "granularity": "1:SECONDS"
    }
  ]
}

Note: The “dateTimeFieldSpecs” will be used to set the granularity and understand the seasonality for accurate predictions by anomaly detectors.

Alert creation and notifications​

  • Refer to how to guide to create an alert (link) and use the following alert configurations to create alerts
  • Subscribe to notifications (link)

Use the following alert configuration for “Simple percentage rule based alert creation”.​

{
  "name": "AdCampaignData-percentage-sample-alert-test",
  "description": "Monitor number of clicks",
  "template": {
    "name": "startree-percentage-rule"
  },
  "templateProperties": {
    "dataSource": "pinot",
    "dataset": "AdCampaignData",
    "aggregationFunction": "sum",
    "seasonalityPeriod": "P7D",
    "monitoringGranularity": "P1D",
    "aggregationColumn": "Clicks",
    "baselineOffset": "P7D",
    "percentageChange": "0.01"
  },
  "cron": "0 0 5 ? * * *"
}

Use the following alert configuration for “Startree-ets alert creation”​

Advanced detection model based on metrics pattern and seasonality

{
  "name": "AdCampaignData-seasonal-sample-alert",
  "description": "Monitor number of clicks",
  "template": {
    "name": "startree-ets"
  },
  "templateProperties": {
    "dataSource": "pinot",
    "dataset": "AdCampaignData",
    "aggregationFunction": "sum",
    "seasonalityPeriod": "P7D",
    "lookback": "P20D",
    "monitoringGranularity": "P1D",
    "sensitivity": "1",
    "aggregationColumn": "Clicks"
  },
  "cron": "0 0 5 ? * * *"
}

Use the following alert configuration for “Startree-ets multidimensional alert creation”​

Advanced detection model based on metrics pattern and seasonality + dimension exploration ( multiple-timeseries) (Learn more about the concepts by clicking this link)

{
  "name": "AdCampaignData-seasonal-dx-sample-alert",
  "description": "Monitor number of clicks across multiple dimensions such as Country and Ad Campaign Size",
  "template": {
    "name": "startree-ets-dx"
  },
  "templateProperties": {
    "dataSource": "pinot",
    "dataset": "AdCampaignData",
    "aggregationFunction": "sum",
    "seasonalityPeriod": "P7D",
    "lookback": "P20D",
    "monitoringGranularity": "P1D",
    "sensitivity": "1",
    "aggregationColumn": "Clicks",
    "queryFilters": "${queryFilters}",
    "enumerationItems": [
      {
        "name": "Overall",
        "params": {
          "queryFilters": ""
        }
      },
      {
        "name": "DoubleClick-Canada",
        "params": {
          "queryFilters": " AND Country='Canada'"
        }
      },
      {
        "name": "DoubleClick-US",
        "params": {
          "queryFilters": " AND Country='USA'"
        }
      }
    ]
  },
  "cron": "0 0 5 ? * * *"
}

Anomaly reporting​

The anomalies can be reported in multiple ways using ThirdEye.

Different ways of reporting anomalies are:

  • Email
  • Slack
  • Webhook
  • Using APIs
  • Directly going to ThirdEye and viewing the anomalies

Here is the link to “how to guide” talking about how the different types of notifications (Email, Slack, Webhook) can be used to report anomalies.

One can also pull these anomalies using ThridEye API (link to guide) and report those anomalies in custom applications.

Access anomalies directly going to ThirdEye UI. (link)

Root-cause analysis: anomaly investigation​

Follow this guide to perform root cause analysis with heatmaps, custom events, and other signals.

In this case if you create one of the above alerts, out of the few anomalies that are detected by ThirdEye, the one detected from March 28, 2018 to March 29, 2018 shows a dip of around 120k clicks than predicted.

If you investigate the anomaly the heatmap shows no prominent dimension that is contributing to the dip.

But if you observe the list of events around that time then that gives a better picture about why the dip happened. Around the same time 2 events took place - ProductCatalogChange - API Failures

Once we have this insight we can narrow down on these two events and validate our findings. If the API failures were related to the Ad campaign section then that explains the dip, and we know there is nothing wrong with the campaign strategy.

But it's also interesting to see if the change in the catalog was not well received at first by the users, and it eventually worked (as the following trend looks normal). This can be an input to the catalog designers that users experienced some initial friction upon the catalog changes.

Want to learn more?​

Note: If interested to learn more about it join StarTree slack community (link).

StarTree ThirdEye comes with a lot of APIs (Application Programming Interfaces) (link) – One can build their own custom portals/web wrapper over StarTree ThirdEye. As a user, you have more control over using and customizing StarTree ThirdEye in creating great user experiences.

Interested in trying out StarTree ThirdEye?​

Follow this link to find different deployment options for trying out StarTree ThirdEye.