Integrating Open Data

Mike Juniper :: Washington DC R&D Center

Alex Harris :: Washington DC R&D Center

mjuniper.github.io/presentations/ds2017/integrating-opendata.html

github.com/mjuniper/presentations

ArcGIS Open Data

Interoperability

Koop

Alex Harris

Connect APIs to ArcGIS

Open Source (github.com/Esri/koop) R & D
node js
Converts third-party APIs into Feature Services
export as kml, csv, shp

Koop Providers

Responsible for translating API into GeoJSON
Koop handles GeoJSON to Geoservices transformation

Koop Providers

Koop Providers (for Koop >= 3.0)

Requirements

Docs are found at koopjs.github.io/docs/specs/provider/
Implement a single function
That function fetches data from remote API and translates it into GeoJSON
Koop handles the rest
Slides for Koop Presentation: www.slideshare.net/DanielFenton2/koop-using-3rd-party-services-in-arcgis

Koop Demo

Open Data from City of DC
Koop Providers for Zillow, Craigslist, and Yelp
Demo

DCAT

geohub.lacity.org/data.json

or:

<your site>-<your organization>.opendata.arcgis.com/data.json

DCAT enables integration with CKAN portals such as data.gov

Open Data API

What is the Open Data API?

It is not the Portal API

It is not the GeoServices API

It is used for finding Open Data Datasets & managing Open Data Sites

Simple API example

        		https://opendata.arcgis.com

        		/api/v2

		        	/datasets

			      ?q=crime&page[size]=5&page[number]=3

Example:

https://opendata.arcgis.com/api/v2/datasets?q=crime&page[size]=5&page[number]=3

An Aside on the V1 API

Don't Use It

What uses the OpenData API?

OpenData "Umbrella" (Ember)

OpenData Custom Sites 2.X (Ember)

OpenData Admin 2.0 (Ember)

OpenData V2 API

Versioning via URL

API returns JSON

adheres to JSONAPI standard

/api/v2/{:resource}

/api/v2/{:resource}/{:id}

/api/v2/{:resource}/{:id}/{:related_resource}

Why JSONAPI?

Standardizes JSON responses

Supports features like side-loading

Supports client-defined payloads

Has many client and server implementations

Go to http://jsonapi.org/implementations/ for client and server libraries

Go to http://jsonapi.org/format for the spec

OpenData V2 API: Query Abilities

Full text queries supported with the `q` parameter

/api/v2/datasets?q=parcels

Can 'side-load' related resources in a single request via `include` parameter

"Bring me all the datasets that match the query 'parcel' and also include the organizations that published those datasets"

/api/v2/datasets?q=parcels&include=organizations

OpenData V2 API: Response Format

Standardized response for a resource or a collection of resources

/api/v2/datasets -> {"data": []}

/api/v2/datasets/:id -> {"data": {}}

OpenData V2 API: Object Format

Each object has a standard format


{
  "id": "f2e1c2ef9eb44f2899f4a310a80ecec9_2",
  "type": "dataset",
  "attributes": {
    "name": ...,
    "url": ...,
    "recordCount": ...,
  }
}

OpenData V2 API: Side-Loaded Resources

How are side-loaded resources returned?


{
  "data": [...],
  "included": [
    {
      "id": "afd7refdr",
      "type": "organization",
      "attributes": {
        "name": ...,
        "homePageUrl": ...,
      }
    }
  ]
}

OpenData V2 API: Metadata

Aggregations included on each request for collections


{
  "data": [...],
  "meta": {
    "apiRoot": "https://opendata.arcgis.com/api/v2/",
    "resourceRoot": "https://opendata.arcgis.com/api/v2/datasets/",
    "queryParameters": {
      query-parameters-of-request...
    },
    "stats": {
      counts-and-other-aggregations
    }
  }
}

OpenData V2 API: Request Metadata Example

/api/v2/datasets?q=population&filter[content]=spatial%20dataset&page[size]=25


{
  "meta": {
    "apiRoot": "http://opendata.arcgis.com/api/v2",
    "resourceRoot": "http://opendata.arcgis.com/api/v2/datasets",
    "queryParameters": {
      "page": {
        "number": 1,
        "size": 25
      },
      "q": "population",
      "filter": {
        "content": "spatial dataset"
      }
    },
    "stats": {
      "count": 25,
      "totalCount": 186,
      "aggs": {
        "content": [{ "key": "spatial dataset", "docCount": 186 }],
        "tags": [
          { "key": "census", "docCount": 101 },
          { "key": "population", "docCount": 97 },
          { "key": "demographics", "docCount": 71 },
          { "key": "acs": "docCount": 21}
        ],
        "source": [
          { "key": "U.S. Federal Maps and Apps", "docCount": 24 },
          { "key": "Miami-Dade County, Florida", "docCount": 11 }
        ]
      }
    }
  }
}

OpenData V2 API: Pagination

/api/v2/datasets?q=population&filter[content]=spatial%20dataset&page[size]=25


{
  "data": [...],
  "meta": {...},
  "links": {
    "first": "https://opendata.arcgis.com/api/v2/datasets?page[number]=1&page[size]=25&q=population&filter[content]=spatial%20dataset",
    "next": "https://opendata.arcgis.com/api/v2/datasets?page[number]=2&page[size]=25&q=population&filter[content]=spatial%20dataset",
    "last": "https://opendata.arcgis.com/api/v2/datasets?page[number]=8&page[size]=25&q=population&filter[content]=spatial%20dataset"
  }
}

OpenData V2 API: Standardized Errors

/api/v2/datasets?foo=bar


{
  "errors": [
    {
      "title": "Unrecognized Parameter",
      "detail": "'foo' is not a recognized parameter for this request.",
      "status": 400,
      "source": { "parameter": "foo=bar" },
      "meta": { detailed-information-to-debug-the-request }
    }
  ]
}

OpenData V2 API: More Standardized Errors

/api/v2/datasets?foo=bar&filter[pizza]=anchovies

All errors are processed before a response is generated


{
  "errors": [
    {
      "title": "Unrecognized Parameter",
      "detail": "'foo' is not a recognized parameter for this request.",
      "status": 400,
      "source": { "parameter": "foo=bar" },
      "meta": { detailed-information-to-help-debug-the-request }
    },
    {
      "title": "Invalid filter key",
      "detail": "'filter[pizza]' is not a valid filter key",
      "status": 400,
      "source": { "parameter": "filter[pizza]" },
      "meta": { detailed-information-to-help-debug-the-request }
    }
  ]
}

Try it yourself

OpenData V2 API Sandbox

Unoffical ArcGIS Open Data API Sandbox at

http://akharris.github.io/ember-arc-swag/

Embedded Open Data

Open Data Search Web Component

github.com/esridc/opendata-search-component

What can you do with the Open Data API?

Build an app!

ArcGIS Open Data


$.getJSON('http://my.api.url')
  .done(function (response, status, xhr) {
    /* stuff it into the DOM... */
  })
  .fail(function (xhr, status, error) {
    /* do something about the error! */
  });


$.getJSON('http://my.api.url')
  .done(function (response, status, xhr) {
    /* do something with the response! */
  })
  .fail(function (xhr, status, error) {
    /* do something about the error! */
  });

MV*

Key endpoints:

/api/v2/datasets?q=<search string>&page[number]=<page number>

and

/api/v2/datasets/<id>

And maybe:

/api/v2/datasets/<id>/related

And:

/datasets/autocomplete?query=<search string>

Also:

/datasets/<id>.csv

/datasets/<id>.geojson

/datasets/<id>.kml

/datasets/<id>.zip

Errors...


$.getJSON('http://my.api.url')
  .done(function (response, status, xhr) {
    if (response.error) {
      /* you don't have to do this! */
    }
  })
  .fail(function (xhr, status, error) {
    switch (xhr.status) {
      case 404:
        /* handle 404 */
      default:
        /* handle other errors */
    }
  });

Gotcha's:

SSL only (https://opendata.arcgis.com/api/v2)
Currently no JSONP support
Does support CORS
But ... IE < 10

esridc.github.io/OpenData-Backbone

Ok,... how?

Fork it: github.com/esridc/OpenData-Backbone

then:

              
$ git clone https://github.com/esridc/OpenData-Backbone.git
$ cd opendata-backbone
$ npm install
$ bower install
$ gulp serve

In backbone-land...


DatasetCollection = Backbone.Collection.extend({
  url: function () {
    //get the params (q=, page=, etc) from somewhere...
    var queryParams = '';
    return MyOD.config.api + 'datasets?' + queryParams;
  },

  parse: function (resp) {
    return resp.data;
  }
});


var datasetCollection = new DatasetCollection();
datasetCollection.fetch({ ... });


DatasetModel = Backbone.Model.extend({
  url: function () {
    return MyOD.config.api + 'datasets/' + this.get('id');
  },

  parse: function (response) {
    return response.data;
  }
});


var datasetModel = new DatasetModel();
datasetModel.fetch({ id: myId });

esridc.github.io/opendata-ember

Fork it: github.com/esridc/opendata-ember

then:

              
$ npm install -g ember-cli
$ git clone https://github.com/esridc/opendata-ember.git
$ cd opendata-ember
$ npm install
$ bower install
$ ember serve

In ember-land...


import Ember from 'ember';
import DS from 'ember-data';

export default DS.JSONAPIAdapter.extend({

  host: ENV.APP.API,

  namespace: 'api/v2',

  pathForType: function(type) {
    const camelized = Ember.String.camelize(type);
    return Ember.String.pluralize(camelized);
  },

  urlForFindRecord: function (id/*, modelName, snapshot*/)  {
    const host = this.get('host');
    const namespace = this.get('namespace');
    return `${host}/${namespace}/datasets/${id}`;
  }

});


import Ember from 'ember';
import DS from 'ember-data';

export default DS.JSONAPISerializer.extend({
  normalizeResponse: function (store, primaryModelClass, payload/*, id, requestType*/) {
    if (Ember.isArray(payload.data)) {
      payload.data = payload.data.map(this._mapDataset);
    } else {
      payload.data = this._mapDataset(payload.data);
    }
    return payload;
  },

  _mapDataset: function (item) {
    if(!item.attributes.name){
      item.attributes.name = item.attributes.item_name;
    }
    return item;
  }
});

Or:

            
$ ember install ember-arcgis-opendata-services

            
// datasets route
model: function (params) {
  const qryParams = ...
  return this.store.query('dataset', qryParams);
},

            
// datasets.dataset route
model: function (params) {
  return this.store.findRecord('dataset', params.id);
},

More to come?

mjuniper.github.io/presentations/ds2017/integrating-opendata.html
koopjs.github.io
jsonapi.org
akharris.github.io/ember-arc-swag (not API feature-complete but still useful)
emberjs.com
backbonejs.org

Integrating Open Data

Mike Juniper :: Washington DC R&D Center

Alex Harris :: Washington DC R&D Center

mjuniper.github.io/presentations/ds2017/integrating-opendata.html

github.com/mjuniper/presentations

ArcGIS Open Data

Interoperability

Koop

Alex Harris

Connect APIs to ArcGIS

Koop Providers

Koop Providers

Example: Socrata Provider

Via koop:

Registered as an item in ArcGIS Online:

Shared through Open Data:

Koop Providers (for Koop >= 3.0)

Requirements

Koop Demo

DCAT

Open Data API

What is the Open Data API?

Simple API example

Example:

An Aside on the V1 API

What uses the OpenData API?

OpenData V2 API

/api/v2/{:resource}

/api/v2/{:resource}/{:id}

/api/v2/{:resource}/{:id}/{:related_resource}

Why JSONAPI?

OpenData V2 API: Query Abilities

OpenData V2 API: Response Format

Standardized response for a resource or a collection of resources

OpenData V2 API: Object Format

OpenData V2 API: Side-Loaded Resources

OpenData V2 API: Metadata

Aggregations included on each request for collections

OpenData V2 API: Request Metadata Example

OpenData V2 API: Pagination

OpenData V2 API: Standardized Errors

OpenData V2 API: More Standardized Errors

OpenData V2 API Sandbox

Embedded Open Data

What can you do with the Open Data API?

Build an app!

ArcGIS Open Data

MV*

Key endpoints:

And maybe:

Also:

Errors...

Gotcha's:

esridc.github.io/OpenData-Backbone

Ok,... how?

In backbone-land...

esridc.github.io/opendata-ember

In ember-land...

Or:

More to come?

`/api/v2/{:resource}`

`/api/v2/{:resource}/{:id}`

`/api/v2/{:resource}/{:id}/{:related_resource}`