Skip to end of metadata
Go to start of metadata

The license could not be verified: License Certificate has expired!

Do not manually create any manifest- or feed-related paths. Doing so will cause your export to fail to verify due to object-level ACLs. The necessary paths will be automatically created by AppNexus.

AppNexus provides bulk exports of Log Level Data (LLD) using client-specific keys (as set up using the procedures on this page). Data is exported via SSL file transfers to the client's cloud storage location (see below for options). The security of those client buckets/containers is the sole responsibility of the client.

Log Level Data - Cloud Export

Cloud Export is a feature used to automatically transfer Log-Level Data (LLD) to cloud storage. AppNexus supports data transfer to Amazon S3 buckets, Google Cloud Storage buckets and Microsoft Azure Blob Storage containers. Data begins transferring as soon as batch data becomes available. This setup provides faster enablement and a simpler workflow for AppNexus clients (e.g., no need to poll the siphon service as with the Log-Level Data service).

Cloud Export Setup

Prerequisites

Please confirm the following prior to proceeding to setup.

  1. The LLD feeds you plan to consume should already be enabled for your AppNexus seat (member ID). If that is not the case, contact your Account Representative to subscribe to Log-level data.
  2. Only the protobufprotobuf-delimited and avro formats are supported by AppNexus Cloud Export at this time.
  3. You must have Network Admin privileges to perform the steps below.

Setup

Please follow the procedures in the sections below for the cloud vendor you'd like to export to. AppNexus currently supports the following through the Console UI:

  • Amazon S3
  • Microsoft Azure Blob Storage

  • Google Cloud Storage

If you use a firewall or other IP-restricting security features, add all of these IPs to your whitelist: 68.67.155.230, 68.67.155.231, 68.67.135.70, 68.67.135.71.

Amazon S3

  1. Create an Amazon S3 bucket which will serve as a drop box for log-level data from the AppNexus Cloud Export system.
  2. Select Log-Level Data from the Network (or Reporting) menu in AppNexus Console.
  3. Click Manage in the Cloud Export column of the feed for which you want to create an S3 export. The Amazon S3 section will be displayed.
  4. Click New to set up a new export or Configure to edit an existing export. The Cloud Export to: Amazon S3 page will be displayed. 
  5. Setup. Enter the information in the table below and click Save:

    Bucket

    Name of the Amazon S3 bucket created in step 1.

    Manifest path

    Root for manifest files (e.g., /manifests). This file path does not support macros.

    Feed Path

    Feed path with macros (e.g., /feeds/%FEED_NAME%/%YEAR%/%MONTH%/%DAY%/%HOUR%/%TIMESTAMP%)

    Notification Email

    Comma-separated list of emails. If Cloud Export can no longer access your bucket, your export will be deactivated and you will be notified by email.

    FormatData format for log-level data files (e.g., protobuf )
    Server-side EncryptionThe server-side encryption applied to uploaded LLD files:
    • None - do not apply server-side encryption
    • SSE-S3 - apply server-side encryption with Amazon S3-managed encryption keys
    • SSE-KMS - apply server-side encryption with AWS KMS-managed encryption keys. You must grant the Xandr AWS user access to your key.
    Selecting SSE-S3 or SSE-KMS will override any default bucket encryption settings for the uploaded files.

    Do not manually create any manifest- or feed-related paths. Doing so will cause your export to fail to verify due to object-level ACLs. The necessary paths will be automatically created by AppNexus.

  6. Authorization. For each AppNexus member seat, AppNexus creates a unique AWS user to upload data to S3 buckets. AppNexus generates a suggested bucket policy to apply to your bucket that allows our AWS user access to your bucket. This policy can be used as-is, but if you already have a policy applied to your bucket, you will have to merge our policy statements with yours.
  7. Verification. Once you apply a policy to your bucket, you must verify your permissions before your S3 Cloud Export can be activated. Click Verify (on the Cloud Export to: Amazon S3 page) to run a series of tests that mimic actions that occur during an actual export. Once your export is verified, it is active. Data from the feed will automatically be exported to your S3 bucket (starting in the next hour).

Microsoft Azure Blob Storage 

  1. Create an Azure storage account which will host one or more storage containers.

  2. Create an Azure Blob storage container inside your storage account which will serve as a drop box for log-level data from the AppNexus Cloud Export system.
  3. Select  Log-Level Data from the Network (or Reporting) menu in AppNexus Console.
  4. Click Manage in the Cloud Export column of the feed for which you want to create an Azure Blob storage export. The Microsoft Azure Blob Storage section will be displayed.
  5. Click New to set up a new export or Configure to edit an existing export. The Cloud Export to: Microsoft Azure Blob Storage page will be displayed. 
  6. Setup. Enter the information in the table below and click Save:

    Storage Account

    Name of the Azure storage account created in step 1.

    Container

    Name of the Azure Blob storage container created in step 2.

    Manifest Path

    Root for manifest files (e.g., /manifests). This file path does not support macros.

    Feed Path

    Feed path with macros (e.g., /feeds/%FEED_NAME%/%YEAR%/%MONTH%/%DAY%/%HOUR%/%TIMESTAMP%)

    Notification EmailComma-separated list of emails. If Cloud Export can no longer access your container, your export will be deactivated and you will be notified by email.
    Format

    Data format for log-level data files (e.g., protobuf)

  7. Authorization. Generate a shared access signature (SAS) for the storage account with the following permissions:
    • Allowed services - Blob
    • Allowed resource types - Container, Object
    • Allowed Permissions - Read, Write, Delete, List, Add, Create
    • Start and expiry date/time - Select expiry date in the distant future
    • Allowed protocols - HTTPS only
    After you create the SAS token, paste it in the SAS Token field in AppNexus Console and click Save.
  8. Verification. Once you give us your SAS token, you must verify your SAS permissions before your Azure Blob Storage Cloud Export can be activated. Click Verify (on the Cloud Export to: Microsoft Azure Blob Storage page) to run a series of tests that mimic actions that occur during an actual export. Once your export is verified, it is active. Data from the feed will automatically be exported to your Azure container (starting in the next hour).

Google Cloud Storage

  1. Create a Google Storage Bucket which will serve as a drop box for log-level data from the AppNexus Cloud Export system.

  2. Select Log-Level Data from the Network (or Reporting) menu in AppNexus Console.
  3. Click Manage in the Cloud Export column of the feed for which you want to create an Google Cloud Storage export. The Google Cloud Storage section will be displayed.
  4. Click New to set up a new export or Configure to edit an existing export. The Cloud Export to: Google Cloud Storage page will be displayed. 
  5. Setup. Enter the information in the table below and click Save:

    Bucket

    Name of Google Storage Bucket created in step 1.

    Manifest Path

    Root for manifest files (e.g., /manifests). This file path does not support macros.

    Feed Path

    Feed path with macros (e.g., /feeds/%FEED_NAME%/%YEAR%/%MONTH%/%DAY%/%HOUR%/%TIMESTAMP%)

    Notification EmailComma-separated list of emails. If Cloud Export can no longer access your bucket, your export will be deactivated and you will be notified by email.
    Format

    Data format for log-level data files (e.g., protobuf)

    Do not manually create any manifest- or feed-related paths. Doing so will cause your export to fail to verify due to object-level ACLs. The necessary paths will be automatically created by AppNexus.

  6. Authorization. Nagivate to the Cloud Storage console, select your bucket, and click SHOW INFO PANEL. Add our Google member prod-lld-gcs-{APPNEXUS MEMBER ID}@appnexus-cloud-export.iam.gserviceaccount.com  to your bucket permissions and assign it the Storage Legacy Bucket Writer Cloud Storage IAM Role. An example of Google member for Appnexus member ID 123456 is : prod-lld-gcs-123456@appnexus-cloud-export.iam.gserviceaccount.com

  7. Verification. Once you give our Google member the Storage Legacy Bucket Writer role in your bucket, you must verify your permissions before your Google Storage Cloud Export can be activated. Click Verify (on the Cloud Export to: Google Cloud Storage page) to run a series of tests that mimic actions that occur during an actual export. Once your export is verified, it is active. Data from the feed will automatically be exported to your Google Cloud Storage bucket (starting in the next hour).

Failures and Export Deactivation

If the export fails, AppNexus automatically runs verification tests to ensure that the failure was not caused by a permissions issue.

  • If the failure was caused by a permissions issue, your export will be deactivated and you will be notified via email. You will then need to re-verify your export. A successful verification will reactivate your export.
  • If the failure was not caused by permissions issues, the export will automatically be re-tried.

If AppNexus deactivated your cloud export due to a permissions issue, once the export has been re-verified and re-enabled, we will only backfill up to a maximum of 3 days of feed data.

Feed Path Macros

Macros can be used within feed paths to customize the feed output location by data hour, feed timestamp, and other metadata. Supported macros are listed below:

Macro

Description

%YEAR%

Data year (yyyy)

%MONTH%

Data month (MM)

%DAY%

Data day (dd)

%HOUR%

Data hour (HH)

%TIMESTAMP% 1

Feed processing timestamp (yyyyMMddHHmmss)

%FEED_NAME%

Name of the feed



1  Since directories are overwritten when data is copied and AppNexus can sometimes reprocess data, the %TIMESTAMP% macro is required for feed paths.

Manifest Files

Manifest files help keep track of the files uploaded by the Cloud Export service. Since cloud object stores (e.g., Amazon S3, Azure Blob Storage, Google Cloud Storage) are eventually consistent, a source of truth (i.e., which files should exist) is required for downstream consumers. The manifest file is a JSON object written to a pre-configured location in your bucket. It is specified in the JSONSchema format. We recommend that all accessing of paths (e.g., for data processing, purging, etc.) be done by first referencing the manifest files. The manifest files contain important information about data age, status, and format.

File Naming and Paths

Manifest files are created when the feed begins processing for a given hour. The files will be created in the path location you specified. Manifest file names contain the name of the feed, a timestamp denoting the hour of the data (the range of time it covers), and a processing timestamp. A manifest file is created for each feed, member (seat) ID, hour of data, and feed run. AppNexus automatically reprocesses data as needed. The files are also given suffixes indicating the status of the associated feed job. 

  • If a manifest file has a suffix of "-failed" or "-processing", it indicates that the feed was not successfully processed.
  • If a manifest file has no suffix, it indicates that the feed's data is ready for ingestion.

General format

Format

{feed_name}-{seat_id}-{YYYYMMddHH}-{yyyyMMddHHmmss}.json[-{suffix}] 

Status: Processing

Suffix

-processing

Example

standard_feed-998-2017013109-20170131132522.json-processing

Status "Failed"

Since failures can occur from unrecoverable situations that make it impossible to change the manifest file's suffix, it is possible that a manifest file suffix is changed from “processing” to “failed” sometime later (e.g., during the next successful run). For example, if a Cloud Export process fails because of a change to the bucket's access policy, it will no longer be able to write to that bucket (and the export will be deactivated). The manifest's file suffix will remain "-processing" until the export is reactivated and runs successfully.

Suffix

-failed

Example

standard_feed-998-2017013109-20170131132522.json-failed

Status "Successful"

Suffix

None

Example

standard_feed-998-2017013109-20170131132522.json

Feed Processing Workflow

It is useful to understand how Feed jobs interpret manifest files in order to build your downstream logic. The process is as follows:

  1. The manifest path is scanned for entries with the “-processing” suffix. Entries with this suffix indicate a previous run that experienced a fatal error. These entries are renamed with a “-failed” suffix (e.g., json-processing > .json-failed).  
  2. A manifest file is generated and written to the bucket with a “-processing” suffix.
  3. The feed data is written to the bucket.
  4. If the write process fails, the manifest file will be renamed to include a “-failed” suffix (e.g., .json-failed). If the write process is completed successfully, the “-processing” suffix is simply removed (e.g., .json-processing > .json), indicating a completed run.

Manifest File Schema 

The schema for the manifest file is described in the well-known JSONSchema format. AppNexus recommends code-generating classes from the JSONSchema we provide. A project that achieves this for Java is jsonschema2pojo.  

Manifest File Example

Failures/Partial Loads

Feed job failures can cause partial data loading. To avoid consuming partially loaded data, use the manifest file names to determine what to consume. 

  • Do not consume files which will have a suffix of “-processing” or “-failed”. These are incomplete data sets.
  • Only consume data from paths corresponding to completed loads (i.e., manifest files with a .json extension but no additional suffix). 

Purging

When purging old data from your bucket, the manifest files should be used for determining when data is eligible for purging. Only purge those files that are no longer relevant (e.g., older than a week).

To improve directory performance, manifest files older than 30 days will be moved to an /archive folder located within the manifest file path. As a precaution, if you have any procedures/scripts that refer to the manifest file path directly, you should amend them to also look in the /archive folder for manifest files older than 30 days.

Reprocessing

Feeds re-processed for a given hour will never overwrite old data. AppNexus will reprocess data when inconsistencies are detected. When a feed job is reprocessed, there will be two (or more) manifest files corresponding to the same data hour. However, the processing timestamps and paths will differ between runs. The file with the latest timestamp will be the corrected one.

 

Client Data Loss

AppNexus does not re-export data that has already been successfully exported. To mitigate situations in which you may have accidentally deleted successfully exported data, we recommend you leverage your cloud provider’s backup or versioning features.

Troubleshooting

Uniqueness Criteria for Configurations

 Your Cloud Export configuration must be unique, which means:

  • For Manifest paths:
    • For Microsoft Azure, if the Destination (Storage Account and Container) combined and Path are the same, the Member ID and Feed must be different.
    • For all other configurations, if the Bucket and Path are the same, the Member ID and Feed must be different.
  • For Feed paths, if the Destination and Resolved Feed path are the same, the Member ID must be different and the Feed and Format must be the same.

Verification Issues

The table below describes a few common issues that you may encounter when getting verification set up. 

Failed Verify ActionPotential CauseApplies toSolution
Checking for manifest files in GLACIERArchiving of manifest files is not properly set upAmazon S3

Please ensure that all lifecycle rules are set up to move manifest objects to Glacier only if they are not in the manifests path (manifests/archive is fine).

Checking meta-information access on manifest filesYou are revoking object-level access for non-archived manifestsAmazon S3

For each object, aws s3api get-object-acl --bucket <bucket> --key <object> should yield the following as one of the grants for each non-archived manifest object:

Google Cloud Storage

For each object , gsutil acl get <object> should yield the following as one of the grants for each non-archived manifest object:

Creating prefix at <path>You have already applied authorization to your bucket or container but have manually created any manifest or feed "folders".All Cloud Export configurationsManually created objects have different ACLs that may deny access to the Cloud Export system. Please delete any manually created manifest or feed objects. The Cloud Export system will create necessary objects on its own.
  • No labels