The SafeGraph Developer Hub

Welcome to the SafeGraph developer hub. You'll find comprehensive guides and documentation to help you start working with SafeGraph as quickly as possible, as well as support if you get stuck. Let's jump right in!

Get Started    

Delivery for Enterprise

This document outlines the configuration process for receiving SafeGraph places data on a monthly basis as part of an enterprise license. If you need assistance accessing a data sample, please refer to Places Data Sample Delivery.

If you are not an enterprise license customer, please get your data from the SafeGraph Data Bar!

S3 Bucket Configuration

The first step of the delivery process is to configure an S3 bucket for us to deliver the data to. You'll need to have access to your organization's Amazon Web Services (AWS) console with privileges to create an S3 bucket. If your organization doesn't use AWS, let us know and we'll help configure the data delivery process on your behalf. To begin, please follow the instructions below which should only take a few minutes to complete.

  1. Sign into the AWS S3 console
  2. Create an S3 bucket in which you want to receive SafeGraph data (e.g. s3://my-company-sg-data)
  3. Create a policy for SafeGraph to access the bucket by first selecting the Permissions tab.
  1. Click on the Bucket Policy box.
  2. Copy this bucket policy as formatted below and paste into the permission editor:
{
   "Version": "2012-10-17",
   "Statement": [
      {
         "Sid": "SafeGraph Bucket Permissions",
         "Effect": "Allow",
         "Principal": {
            "AWS": "arn:aws:iam::461305247324:role/safegraph-egress"
         },
         "Action": [
            "s3:*"
         ],
         "Resource": [
            "arn:aws:s3:::{{your bucket name}}"
         ]
      },
      {
         "Sid": "SafeGraph Object Permissions",
         "Effect": "Allow",
         "Principal": {
            "AWS": "arn:aws:iam::461305247324:role/safegraph-egress"
         },
         "Action": [
            "s3:*"
         ],
         "Resource": [
            "arn:aws:s3:::{{your bucket name}}/*"
         ]
      }
   ]
}

Include your bucket name in resources

When pasting the policy above, please make sure to substitute your bucket name and path for {{your bucket name}} when specifying the resource.

  1. Click Save
  2. That's it! Please send your customer success manager the name of your bucket.

Do not set up KMS encryption

Please do not enable KMS encryption when configuring the bucket. We will not be able to write to your S3 bucket if you enable this setting.

Places File Delivery

Once we receive the name of your configured S3 bucket, SafeGraph will begin to deliver data once per month formatted per the Places Schema. Please note we may push updates more frequently on occasion in the case an urgent bug fix.

  • If you are an enterprise customer, SafeGraph products are delivered (together) on ~ 7th of the month.
    • Up to 5 file paths will be delivered with the following structure: s3://customer-bucket/customer-prefix/{{sg-file-name}}/yyyy/mm/dd/hh/*.csv.gz. {{sg-file-name}} is one of the following:
      • core_poi, geometry, patterns or some combination like core_poi-geometry or core_poi-patterns or core_poi-geometry-patterns (depending on your subscription) includes all of the following for which you are subscribed: Core, Geometry, Patterns.
      • Weekly Patterns deliveries have a separate partition to distinguish from monthly deliveries and have the following structure: s3://customer-bucket/customer-prefix/weekly/{{sg-file-name}}/yyyy/mm/dd/hh/*.csv.gz.
      • brand_info (if subscribed to Core)
      • home_panel_summary (if subscribed to Patterns)
      • visits_panel_summary (if subscribed to Patterns)
      • normalization_stats (if subscribed to Patterns)

The files are sent as gzipped CSVs. Given that our data deliveries contain a lot of data we break up the delivery into multiple small files. Let us know if this doesn't make the process easier on your side!

See also Column Orderings

Delivery Frequency

  • Customers who license Weekly Patterns (+core/geo optionally) get weekly deliveries and no monthly deliveries
  • Customers who license Monthly Patterns (+core/geo optionally) get monthly deliveries and no weekly deliveries
  • Customers who license both Weekly and Monthly Patterns (+core/geo optionally) get monthly and weekly deliveries

Historical Patterns Delivery

From time to time SafeGraph will provide you with historical Patterns data. You may be a new customer receiving historical data, or an existing customer receiving historical backfill of the latest version. Historical/Backfill deliveries are handled specially by the SafeGraph team and delivered in a specific format as follows:

  1. s3://customer-bucket/customer-prefix/{{products_ordered}}_backfill/del_yyyy/del_mm/del_dd/del_hh/data_yyyy/data_mm/*
  2. s3://customer-bucket/customer-prefix/home_panel_summary_backfill/del_yyyy/del_mm/del_dd/del_hh/data_yyyy/data_mm/*
  3. s3://customer-bucket/customer-prefix/vists_panel_summary_backfill/del_yyyy/del_mm/del_dd/del_hh/data_yyyy/data_mm/*
  4. s3://customer-bucket/customer-prefix/normalization_stats_backfill//del_yyyy/del_mm/del_dd/del_hh/data_yyyy/data_mm/*

Where

  • del_yyyy/del_mm/del_dd/del_hh is the datetime the backfill was delivered (hour is start hour)
  • data_yyyy/data_mm represents year and month being summarized for this set of patterns data.
  • {{products_ordered}} will be patterns or core_poi-patterns or core_poi-geometry-patterns, depending on which data products are being delivered.
  • Note that the data_* fields will also align with the columns date_range_start and date_range_end contained in each file of the Patterns data itself.
  • This delivery format was first introduce as part of the October 2019 release.

Updated 2 months ago


Delivery for Enterprise


Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.