Developing with ArcGIS Open Data

Mike Juniper :: Washington DC R&D Center

Alex Harris :: Washington DC R&D Center

Jake Sower :: Washington DC R&D Center

mjuniper.github.io/presentations/opendata-api-2016.html

github.com/mjuniper/presentations

ArcGIS Open Data

Interoperability

koop

Jake Sower

Feature Services from APIs

  • Open Source (github.com/Esri/koop) R & D
  • node js
  • geojson (& others)    ==>    feature services
  • feature services    ==>    geojson
  • export as kml, csv, shp

Socrata

Via koop:

Registered as an item in ArcGIS Online:

Shared through Open Data:

DCAT

opendata.arcgis.com/data.json
or:
<your site>.<your organization>.opendata.arcgis.com/data.json

Open Data API

Alex Harris

Jake Sower

What is an API?

A contract for accessing and modifying data from an authoritative source.

Simple API example

        		https://opendata.arcgis.com
        	
        		/api/v2
        	
		        	/datasets
	        	
			      ?q=water&page[size]=5&page[number]=3

Example:

An Aside on the V1 API

    Unsupported
    Undocumented
    Not long for this world -- will soon be deprecated

OpenData V1 API

HTTP protocol (GET, POST, PUT, DELETE)

opendata.arcgis.com

OR

imap.maryland.opendata.arcgis.com

Available resources /datasets
JSON format

Who consumes the OpenData API?

OpenData "Umbrella" (Ember)
OpenData Custom Sites 1.0 (Backbone)
OpenData Admin 1.0 (Angular)
OpenData Custom Sites 2.0 - coming soon (Ember)
OpenData Admin 2.0 - coming soon (Ember)

OpenData V2 API (beta)

Versioning via URL
API returns JSON
adheres to JSONAPI standard

/api/v2/{:resource}

/api/v2/{:resource}/{:id}

/api/v2/{:resource}/{:id}/{:related_resource}

Why choose the JSONAPI standard?

Standardizes JSON responses
Supports features like side-loading
Has many client and server implementations
Go the http://jsonapi.org/implementations/ for client and server libraries
Go the http://jsonapi.org/format for the spec

OpenData V2 API (beta): Query Abilities

Full text queries supported with the `q` parameter
/api/v2/datasets?q=parcels
Can 'side-load' related resources in a single request via `include` parameter
"Bring me all the datasets that match the query 'parcel' and also include the organizations that published those datasets"
/api/v2/datasets?q=parcels&include=organizations

OpenData V2 API (beta): Response Format

Standardized response for a resource or a collection of resources

/api/v2/datasets -> {"data": []}
/api/v2/datasets/:id -> {"data": {}}

OpenData V2 API (beta): Object Format

Each object has a standard format

{
  "id": "f2e1c2ef9eb44f2899f4a310a80ecec9_2",
  "type": "dataset",
  "attributes": {
    "name": ...,
    "url": ...,
    "recordCount": ...,
  }
}

OpenData V2 API (beta): Side-Loaded Resources

How are side-loaded resources returned?

{
  "data": [datasets...],
  "included": [
    {
      "id": "afd7refdr",
      "type": "organization",
      "attributes": {
        "name": ...,
        "homePageUrl": ...,
      }
    }
  ]
}

OpenData V2 API (beta): Metadata

Aggregations included on each request for collections


{
  "data": [collection-of-resources...],
  "meta": {
    "apiRoot": "https://opendata.arcgis.com/api/v2/",
    "resourceRoot": "https://opendata.arcgis.com/api/v2/datasets/",
    "queryParameters": {
      query-parameters-of-request...
    },
    "stats": {
      counts-and-other-aggregations
    }
  }
}

OpenData V2 API (beta): Request Metadata Example

/api/v2/datasets?q=population&filter[content]=spatial%20dataset&page[size]=25

{
  "meta": {
    "apiRoot": "http://opendata.arcgis.com/api/v2",
    "resourceRoot": "http://opendata.arcgis.com/api/v2/datasets",
    "queryParameters": {
      "page": {
        "number": 1,
        "size": 25
      },
      "q": "population",
      "filter": {
        "content": "spatial dataset"
      }
    },
    "stats": {
      "count": 10,
      "totalCount": 186,
      "aggs": {
        "content": [{ "key": "spatial dataset", "docCount": 186 }],
        "tags": [
          { "key": "census", "docCount": 162 },
          { "key": "population", "docCount": 97 },
          { "key": "demographics", "docCount": 71 },
          { "key": "acs": "docCount": 21}
        ],
        "source": [
          { "key": "U.S. Federal Maps and Apps", "docCount": 24 },
          { "key": "Miami-Dade County, Florida", "docCount": 11 }
        ]
      }
    }
  }
}

OpenData V2 API (beta): Paging

/api/v2/datasets?q=population&filter[content]=spatial%20dataset&page[size]=25

{
  "data": [...],
  "meta": {...},
  "links": {
    "first": "https://opendata.arcgis.com/api/v2/datasets?page[number]=1&page[size]=25&q=population&filter[content]=spatial%20dataset",
    "next": "https://opendata.arcgis.com/api/v2/datasets?page[number]=2&page[size]=25&q=population&filter[content]=spatial%20dataset",
    "last": "https://opendata.arcgis.com/api/v2/datasets?page[number]=8&page[size]=25&q=population&filter[content]=spatial%20dataset"
  }
}

OpenData V2 API (beta): Standardized Errors

/api/v2/datasets?foo=bar

{
  "errors": [
    {
      "title": "Unrecognized Parameter",
      "detail": "'foo' is not a recognized parameter for this request.",
      "status": 400,
      "source": { "parameter": "foo=bar" },
      "links": { "about": "coming-soon: link-to-help-docs" },
      "meta": { detailed-information-to-debug-the-request }
    }
  ]
}

OpenData V2 API (beta): More Standardized Errors

/api/v2/datasets?foo=bar&filter[pizza]=anchovies
All errors are processed before a response is generated

{
  "errors": [
    {
      "title": "Unrecognized Parameter",
      "detail": "'foo' is not a recognized parameter for this request.",
      "status": 400,
      "source": { "parameter": "foo=bar" },
      "links": { "about": "coming-soon: link-to-help-docs" },
      "meta": { detailed-information-to-help-debug-the-request }
    },
    {
      "title": "Invalid filter key",
      "detail": "'filter[pizza]' is not a valid filter key",
      "status": 400,
      "source": { "parameter": "filter[pizza]" },
      "links": { "about": "coming-soon: link-to-help-docs" },
      "meta": { detailed-information-to-help-debug-the-request }
    }
  ]
}
Try it yourself

OpenData V2 API (beta): Support

Spike on an ArcGIS Open Data API Sandbox at
http://akharris.github.io/ember-arc-swag/
API is being finalized. Expect documentation and an API console once the API is out of beta

What can you do with it?

Build an app!


$.getJSON('http://my.api.url')
  .done(function (response, status, xhr) {
    /* stuff it into the DOM... */
  })
  .fail(function (xhr, status, error) {
    /* do something about the error! */
  });
          

$.getJSON('http://my.api.url')
  .done(function (response, status, xhr) {
    /* do something with the response! */
  })
  .fail(function (xhr, status, error) {
    /* do something about the error! */
  });

MV*

Key endpoints:

/api/v2/datasets?q=<search string>&page[number]=<page number>
$.getJSON(MyApiUrl + '/api/v2/datasets?q=water&page[number]=1');
and
/api/v2/datasets/<id>
$.getJSON(MyApiUrl + '/api/v2/datasets/abc123');

And maybe:

/api/v2/datasets/<id>/related
$.getJSON(MyApiUrl + '/api/v2/datasets/abc123/related');
And:
/datasets/autocomplete?query=<search string>
$.getJSON(MyApiUrl + '/datasets/autocomplete?query=wat');

Also:

/datasets/<id>.csv
/datasets/<id>.geojson
/datasets/<id>.kml
/datasets/<id>.zip

Errors...


$.getJSON('http://my.api.url')
  .done(function (response, status, xhr) {
    if (response.error) {
      /* you don't have to do this! */
    }
  })
  .fail(function (xhr, status, error) {
    switch (xhr.status) {
      case 404:
        /* handle 404 */
      default:
        /* handle other errors */
    }
  });

Gotcha's:

  • SSL only (https://opendata.arcgis.com/api/v2)
  • Currently no JSONP support
  • Does support CORS
  • But ... IE < 10

Ok,... how?

Fork it:
then:
              
$ git clone
$ cd opendata-backbone
$ npm install
$ bower install
$ gulp serve

In backbone-land...


DatasetCollection = Backbone.Collection.extend({
  url: function () {
    //get the params (q=, page=, etc) from somewhere...
    var queryParams = '';
    return MyOD.config.api + 'datasets?' + queryParams;
  },

  parse: function (resp) {
    return resp.data;
  }
});
          

DatasetModel = Backbone.Model.extend({
  url: function () {
    return MyOD.config.api + 'datasets/' + this.get('id');
  },

  parse: function (response) {
    return response.data;
  }
});

Fork it:
then:
              
$ npm install -g ember-cli
$ git clone
$ cd opendata-ember
$ npm install
$ bower install
$ ember serve

In ember-land...


import Ember from 'ember';
import DS from 'ember-data';

export default DS.JSONAPIAdapter.extend({

  host: ENV.APP.API,

  namespace: 'api/v2',

  pathForType: function(type) {
    const camelized = Ember.String.camelize(type);
    return Ember.String.pluralize(camelized);
  },

  urlForFindRecord: function (id/*, modelName, snapshot*/)  {
    const host = this.get('host');
    const namespace = this.get('namespace');
    return `${host}/${namespace}/datasets/${id}`;
  }

});
          

import Ember from 'ember';
import DS from 'ember-data';

export default DS.JSONAPISerializer.extend({
  normalizeResponse: function (store, primaryModelClass, payload/*, id, requestType*/) {
    if (Ember.isArray(payload.data)) {
      payload.data = payload.data.map(this._mapDataset);
    } else {
      payload.data = this._mapDataset(payload.data);
    }
    return payload;
  },

  _mapDataset: function (item) {
    if(!item.attributes.name){
      item.attributes.name = item.attributes.item_name;
    }
    return item;
  }
});

More to come?