Developing with ArcGIS Open Data
Mike Juniper :: Washington DC R&D Center
Alex Harris :: Washington DC R&D Center
Jake Sower :: Washington DC R&D Center
ArcGIS Open Data
Interoperability
koop
Jake Sower
Feature Services from APIs
- Open Source (github.com/Esri/koop) R & D
- node js
- geojson (& others) ==> feature services
- feature services ==> geojson
- export as kml, csv, shp
Socrata
Registered as an item in ArcGIS Online:
dcdev.maps.arcgis.com/home/item.html?id=ebf151a0feda4c57ac7422dcd48d87dcShared through Open Data:
jupe.dc.opendatadev.arcgis.com/datasets/75892e4ebf134c8fab2d72c84f529545_0DCAT
opendata.arcgis.com/data.json
or:
<your site>.<your organization>.opendata.arcgis.com/data.json
Open Data API
Alex Harris
Jake Sower
What is an API?
A contract for accessing and modifying data from an authoritative source.
Simple API example
https://opendata.arcgis.com
/api/v2
/datasets
?q=water&page[size]=5&page[number]=3
An Aside on the V1 API
Unsupported
Undocumented
Not long for this world -- will soon be deprecated
OpenData V1 API
HTTP protocol (GET, POST, PUT, DELETE)
opendata.arcgis.com
OR
imap.maryland.opendata.arcgis.com
Available resources /datasets
JSON format
Who consumes the OpenData API?
OpenData "Umbrella" (Ember)
OpenData Custom Sites 1.0 (Backbone)
OpenData Admin 1.0 (Angular)
OpenData Custom Sites 2.0 - coming soon (Ember)
OpenData Admin 2.0 - coming soon (Ember)
OpenData V2 API (beta)
Versioning via URL
API returns JSON
adheres to JSONAPI standard
/api/v2/{:resource}
/api/v2/{:resource}/api/v2/{:resource}/{:id}
/api/v2/{:resource}/{:id}/api/v2/{:resource}/{:id}/{:related_resource}
/api/v2/{:resource}/{:id}/{:related_resource}Why choose the JSONAPI standard?
Standardizes JSON responses
Supports features like side-loading
Has many client and server implementations
Go the http://jsonapi.org/implementations/ for client and server libraries
Go the http://jsonapi.org/format for the spec
OpenData V2 API (beta): Query Abilities
Full text queries supported with the `q` parameter
Can 'side-load' related resources in a single request via `include` parameter
"Bring me all the datasets that match the query 'parcel' and also include the organizations that published those datasets"
OpenData V2 API (beta): Response Format
Standardized response for a resource or a collection of resources
/api/v2/datasets -> {"data": []}/api/v2/datasets/:id -> {"data": {}}OpenData V2 API (beta): Object Format
Each object has a standard format
{
"id": "f2e1c2ef9eb44f2899f4a310a80ecec9_2",
"type": "dataset",
"attributes": {
"name": ...,
"url": ...,
"recordCount": ...,
}
}
OpenData V2 API (beta): Side-Loaded Resources
How are side-loaded resources returned?
{
"data": [datasets...],
"included": [
{
"id": "afd7refdr",
"type": "organization",
"attributes": {
"name": ...,
"homePageUrl": ...,
}
}
]
}
OpenData V2 API (beta): Metadata
Aggregations included on each request for collections
{
"data": [collection-of-resources...],
"meta": {
"apiRoot": "https://opendata.arcgis.com/api/v2/",
"resourceRoot": "https://opendata.arcgis.com/api/v2/datasets/",
"queryParameters": {
query-parameters-of-request...
},
"stats": {
counts-and-other-aggregations
}
}
}
OpenData V2 API (beta): Request Metadata Example
/api/v2/datasets?q=population&filter[content]=spatial%20dataset&page[size]=25
{
"meta": {
"apiRoot": "http://opendata.arcgis.com/api/v2",
"resourceRoot": "http://opendata.arcgis.com/api/v2/datasets",
"queryParameters": {
"page": {
"number": 1,
"size": 25
},
"q": "population",
"filter": {
"content": "spatial dataset"
}
},
"stats": {
"count": 10,
"totalCount": 186,
"aggs": {
"content": [{ "key": "spatial dataset", "docCount": 186 }],
"tags": [
{ "key": "census", "docCount": 162 },
{ "key": "population", "docCount": 97 },
{ "key": "demographics", "docCount": 71 },
{ "key": "acs": "docCount": 21}
],
"source": [
{ "key": "U.S. Federal Maps and Apps", "docCount": 24 },
{ "key": "Miami-Dade County, Florida", "docCount": 11 }
]
}
}
}
}OpenData V2 API (beta): Paging
/api/v2/datasets?q=population&filter[content]=spatial%20dataset&page[size]=25
{
"data": [...],
"meta": {...},
"links": {
"first": "https://opendata.arcgis.com/api/v2/datasets?page[number]=1&page[size]=25&q=population&filter[content]=spatial%20dataset",
"next": "https://opendata.arcgis.com/api/v2/datasets?page[number]=2&page[size]=25&q=population&filter[content]=spatial%20dataset",
"last": "https://opendata.arcgis.com/api/v2/datasets?page[number]=8&page[size]=25&q=population&filter[content]=spatial%20dataset"
}
}
OpenData V2 API (beta): Standardized Errors
/api/v2/datasets?foo=bar
{
"errors": [
{
"title": "Unrecognized Parameter",
"detail": "'foo' is not a recognized parameter for this request.",
"status": 400,
"source": { "parameter": "foo=bar" },
"links": { "about": "coming-soon: link-to-help-docs" },
"meta": { detailed-information-to-debug-the-request }
}
]
}
OpenData V2 API (beta): More Standardized Errors
/api/v2/datasets?foo=bar&filter[pizza]=anchovies
All errors are processed before a response is generated
{
"errors": [
{
"title": "Unrecognized Parameter",
"detail": "'foo' is not a recognized parameter for this request.",
"status": 400,
"source": { "parameter": "foo=bar" },
"links": { "about": "coming-soon: link-to-help-docs" },
"meta": { detailed-information-to-help-debug-the-request }
},
{
"title": "Invalid filter key",
"detail": "'filter[pizza]' is not a valid filter key",
"status": 400,
"source": { "parameter": "filter[pizza]" },
"links": { "about": "coming-soon: link-to-help-docs" },
"meta": { detailed-information-to-help-debug-the-request }
}
]
}
OpenData V2 API (beta): Support
Spike on an ArcGIS Open Data API Sandbox at
API is being finalized. Expect documentation and an API console once the API is out of beta
What can you do with it?
Build an app!
ArcGIS Open Data
$.getJSON('http://my.api.url')
.done(function (response, status, xhr) {
/* stuff it into the DOM... */
})
.fail(function (xhr, status, error) {
/* do something about the error! */
});
$.getJSON('http://my.api.url')
.done(function (response, status, xhr) {
/* do something with the response! */
})
.fail(function (xhr, status, error) {
/* do something about the error! */
});
MV*
Key endpoints:
$.getJSON(MyApiUrl + '/api/v2/datasets?q=water&page[number]=1');$.getJSON(MyApiUrl + '/api/v2/datasets/abc123');And maybe:
$.getJSON(MyApiUrl + '/api/v2/datasets/abc123/related');$.getJSON(MyApiUrl + '/datasets/autocomplete?query=wat');Also:
Errors...
$.getJSON('http://my.api.url')
.done(function (response, status, xhr) {
if (response.error) {
/* you don't have to do this! */
}
})
.fail(function (xhr, status, error) {
switch (xhr.status) {
case 404:
/* handle 404 */
default:
/* handle other errors */
}
});
Gotcha's:
- SSL only (https://opendata.arcgis.com/api/v2)
- Currently no JSONP support
- Does support CORS
- But ... IE < 10
mjuniper.github.io/OpenData-Backbone
Ok,... how?
Fork it: github.com/mjuniper/OpenData-Backbone
then:
$ git clone
$ cd opendata-backbone
$ npm install
$ bower install
$ gulp serve
In backbone-land...
DatasetCollection = Backbone.Collection.extend({
url: function () {
//get the params (q=, page=, etc) from somewhere...
var queryParams = '';
return MyOD.config.api + 'datasets?' + queryParams;
},
parse: function (resp) {
return resp.data;
}
});
DatasetModel = Backbone.Model.extend({
url: function () {
return MyOD.config.api + 'datasets/' + this.get('id');
},
parse: function (response) {
return response.data;
}
});
mjuniper.github.io/opendata-ember
Fork it: github.com/mjuniper/opendata-ember
then:
$ npm install -g ember-cli
$ git clone
$ cd opendata-ember
$ npm install
$ bower install
$ ember serve
In ember-land...
import Ember from 'ember';
import DS from 'ember-data';
export default DS.JSONAPIAdapter.extend({
host: ENV.APP.API,
namespace: 'api/v2',
pathForType: function(type) {
const camelized = Ember.String.camelize(type);
return Ember.String.pluralize(camelized);
},
urlForFindRecord: function (id/*, modelName, snapshot*/) {
const host = this.get('host');
const namespace = this.get('namespace');
return `${host}/${namespace}/datasets/${id}`;
}
});import Ember from 'ember';
import DS from 'ember-data';
export default DS.JSONAPISerializer.extend({
normalizeResponse: function (store, primaryModelClass, payload/*, id, requestType*/) {
if (Ember.isArray(payload.data)) {
payload.data = payload.data.map(this._mapDataset);
} else {
payload.data = this._mapDataset(payload.data);
}
return payload;
},
_mapDataset: function (item) {
if(!item.attributes.name){
item.attributes.name = item.attributes.item_name;
}
return item;
}
});More to come?
- mjuniper.github.io/presentations/opendata-api-2016.html
- koopjs.github.io/
- jsonapi.org
- akharris.github.io/ember-arc-swag/ (not API feature-complete but still useful)
- emberjs.com/
- backbonejs.org/
Esri Dev Summit 2016 | Developing with ArcGIS Open Data
Developing with ArcGIS Open Data
Mike Juniper :: Washington DC R&D Center
Alex Harris :: Washington DC R&D Center
Jake Sower :: Washington DC R&D Center