Integrating Open Data

Mike Juniper :: Washington DC R&D Center

Alex Harris :: Washington DC R&D Center

mjuniper.github.io/presentations/ds2017/integrating-opendata.html

ArcGIS Open Data

Interoperability

Koop

Alex Harris

Connect APIs to ArcGIS

Koop Providers

  • Responsible for translating API into GeoJSON
  • Koop handles GeoJSON to Geoservices transformation

Koop Providers

Example: Socrata Provider

Koop Providers (for Koop >= 3.0)

Requirements

Koop Demo

  • Open Data from
  • Koop Providers for Zillow, Craigslist, and Yelp

DCAT

geohub.lacity.org/data.json
or:
<your site>-<your organization>.opendata.arcgis.com/data.json
DCAT enables integration with CKAN portals such as data.gov

Open Data API

What is the Open Data API?

It is not the Portal API
It is not the GeoServices API
It is used for finding Open Data Datasets & managing Open Data Sites

Simple API example

        		https://opendata.arcgis.com
        	
        		/api/v2
        	
		        	/datasets
	        	
			      ?q=crime&page[size]=5&page[number]=3
        	

An Aside on the V1 API

Don't Use It

What uses the OpenData API?

OpenData "Umbrella" (Ember)
OpenData Custom Sites 2.X (Ember)
OpenData Admin 2.0 (Ember)

OpenData V2 API

Versioning via URL
API returns JSON
adheres to JSONAPI standard

/api/v2/{:resource}

/api/v2/{:resource}/{:id}

/api/v2/{:resource}/{:id}/{:related_resource}

Why JSONAPI?

Standardizes JSON responses
Supports features like side-loading
Supports client-defined payloads
Has many client and server implementations
Go to http://jsonapi.org/implementations/ for client and server libraries
Go to http://jsonapi.org/format for the spec

OpenData V2 API: Query Abilities

Full text queries supported with the `q` parameter
Can 'side-load' related resources in a single request via `include` parameter
"Bring me all the datasets that match the query 'parcel' and also include the organizations that published those datasets"

OpenData V2 API: Response Format

Standardized response for a resource or a collection of resources

/api/v2/datasets -> {"data": []}
/api/v2/datasets/:id -> {"data": {}}

OpenData V2 API: Object Format

Each object has a standard format

{
  "id": "f2e1c2ef9eb44f2899f4a310a80ecec9_2",
  "type": "dataset",
  "attributes": {
    "name": ...,
    "url": ...,
    "recordCount": ...,
  }
}

OpenData V2 API: Side-Loaded Resources

How are side-loaded resources returned?

{
  "data": [...],
  "included": [
    {
      "id": "afd7refdr",
      "type": "organization",
      "attributes": {
        "name": ...,
        "homePageUrl": ...,
      }
    }
  ]
}

OpenData V2 API: Metadata

Aggregations included on each request for collections


{
  "data": [...],
  "meta": {
    "apiRoot": "https://opendata.arcgis.com/api/v2/",
    "resourceRoot": "https://opendata.arcgis.com/api/v2/datasets/",
    "queryParameters": {
      query-parameters-of-request...
    },
    "stats": {
      counts-and-other-aggregations
    }
  }
}
          

OpenData V2 API: Request Metadata Example

/api/v2/datasets?q=population&filter[content]=spatial%20dataset&page[size]=25

{
  "meta": {
    "apiRoot": "http://opendata.arcgis.com/api/v2",
    "resourceRoot": "http://opendata.arcgis.com/api/v2/datasets",
    "queryParameters": {
      "page": {
        "number": 1,
        "size": 25
      },
      "q": "population",
      "filter": {
        "content": "spatial dataset"
      }
    },
    "stats": {
      "count": 25,
      "totalCount": 186,
      "aggs": {
        "content": [{ "key": "spatial dataset", "docCount": 186 }],
        "tags": [
          { "key": "census", "docCount": 101 },
          { "key": "population", "docCount": 97 },
          { "key": "demographics", "docCount": 71 },
          { "key": "acs": "docCount": 21}
        ],
        "source": [
          { "key": "U.S. Federal Maps and Apps", "docCount": 24 },
          { "key": "Miami-Dade County, Florida", "docCount": 11 }
        ]
      }
    }
  }
}
          

OpenData V2 API: Pagination

/api/v2/datasets?q=population&filter[content]=spatial%20dataset&page[size]=25

{
  "data": [...],
  "meta": {...},
  "links": {
    "first": "https://opendata.arcgis.com/api/v2/datasets?page[number]=1&page[size]=25&q=population&filter[content]=spatial%20dataset",
    "next": "https://opendata.arcgis.com/api/v2/datasets?page[number]=2&page[size]=25&q=population&filter[content]=spatial%20dataset",
    "last": "https://opendata.arcgis.com/api/v2/datasets?page[number]=8&page[size]=25&q=population&filter[content]=spatial%20dataset"
  }
}
          

OpenData V2 API: Standardized Errors

/api/v2/datasets?foo=bar

{
  "errors": [
    {
      "title": "Unrecognized Parameter",
      "detail": "'foo' is not a recognized parameter for this request.",
      "status": 400,
      "source": { "parameter": "foo=bar" },
      "meta": { detailed-information-to-debug-the-request }
    }
  ]
}
					

OpenData V2 API: More Standardized Errors

/api/v2/datasets?foo=bar&filter[pizza]=anchovies
All errors are processed before a response is generated

{
  "errors": [
    {
      "title": "Unrecognized Parameter",
      "detail": "'foo' is not a recognized parameter for this request.",
      "status": 400,
      "source": { "parameter": "foo=bar" },
      "meta": { detailed-information-to-help-debug-the-request }
    },
    {
      "title": "Invalid filter key",
      "detail": "'filter[pizza]' is not a valid filter key",
      "status": 400,
      "source": { "parameter": "filter[pizza]" },
      "meta": { detailed-information-to-help-debug-the-request }
    }
  ]
}
					

OpenData V2 API Sandbox

Unoffical ArcGIS Open Data API Sandbox at

Embedded Open Data

What can you do with the Open Data API?

Build an app!


$.getJSON('http://my.api.url')
  .done(function (response, status, xhr) {
    /* stuff it into the DOM... */
  })
  .fail(function (xhr, status, error) {
    /* do something about the error! */
  });
          

$.getJSON('http://my.api.url')
  .done(function (response, status, xhr) {
    /* do something with the response! */
  })
  .fail(function (xhr, status, error) {
    /* do something about the error! */
  });
          

MV*

Key endpoints:

/api/v2/datasets?q=<search string>&page[number]=<page number>
and
/api/v2/datasets/<id>

And maybe:

/api/v2/datasets/<id>/related
And:
/datasets/autocomplete?query=<search string>

Also:

/datasets/<id>.csv
/datasets/<id>.geojson
/datasets/<id>.kml
/datasets/<id>.zip

Errors...


$.getJSON('http://my.api.url')
  .done(function (response, status, xhr) {
    if (response.error) {
      /* you don't have to do this! */
    }
  })
  .fail(function (xhr, status, error) {
    switch (xhr.status) {
      case 404:
        /* handle 404 */
      default:
        /* handle other errors */
    }
  });
          

Gotcha's:

  • SSL only (https://opendata.arcgis.com/api/v2)
  • Currently no JSONP support
  • Does support CORS
  • But ... IE < 10

Ok,... how?

              
$ git clone https://github.com/esridc/OpenData-Backbone.git
$ cd opendata-backbone
$ npm install
$ bower install
$ gulp serve
              
            

In backbone-land...


DatasetCollection = Backbone.Collection.extend({
  url: function () {
    //get the params (q=, page=, etc) from somewhere...
    var queryParams = '';
    return MyOD.config.api + 'datasets?' + queryParams;
  },

  parse: function (resp) {
    return resp.data;
  }
});
          

var datasetCollection = new DatasetCollection();
datasetCollection.fetch({ ... });
          

DatasetModel = Backbone.Model.extend({
  url: function () {
    return MyOD.config.api + 'datasets/' + this.get('id');
  },

  parse: function (response) {
    return response.data;
  }
});
          

var datasetModel = new DatasetModel();
datasetModel.fetch({ id: myId });
          

Fork it:
then:
              
$ npm install -g ember-cli
$ git clone https://github.com/esridc/opendata-ember.git
$ cd opendata-ember
$ npm install
$ bower install
$ ember serve
              
            

In ember-land...


import Ember from 'ember';
import DS from 'ember-data';

export default DS.JSONAPIAdapter.extend({

  host: ENV.APP.API,

  namespace: 'api/v2',

  pathForType: function(type) {
    const camelized = Ember.String.camelize(type);
    return Ember.String.pluralize(camelized);
  },

  urlForFindRecord: function (id/*, modelName, snapshot*/)  {
    const host = this.get('host');
    const namespace = this.get('namespace');
    return `${host}/${namespace}/datasets/${id}`;
  }

});
          

import Ember from 'ember';
import DS from 'ember-data';

export default DS.JSONAPISerializer.extend({
  normalizeResponse: function (store, primaryModelClass, payload/*, id, requestType*/) {
    if (Ember.isArray(payload.data)) {
      payload.data = payload.data.map(this._mapDataset);
    } else {
      payload.data = this._mapDataset(payload.data);
    }
    return payload;
  },

  _mapDataset: function (item) {
    if(!item.attributes.name){
      item.attributes.name = item.attributes.item_name;
    }
    return item;
  }
});
          

Or:

            
$ ember install ember-arcgis-opendata-services
            
          
            
// datasets route
model: function (params) {
  const qryParams = ...
  return this.store.query('dataset', qryParams);
},
            
          
            
// datasets.dataset route
model: function (params) {
  return this.store.findRecord('dataset', params.id);
},
            
          

More to come?