Integrating Open Data
Mike Juniper :: Washington DC R&D Center
Alex Harris :: Washington DC R&D Center
ArcGIS Open Data
Interoperability
- Koop
- DCAT
- Open Data API
- Open Data Search web component
- Building an app against the API
Koop
Alex Harris
Connect APIs to ArcGIS
- Open Source (github.com/Esri/koop) R & D
- node js
- Converts third-party APIs into Feature Services
- export as kml, csv, shp
Koop Providers
- Responsible for translating API into GeoJSON
- Koop handles GeoJSON to Geoservices transformation
Koop Providers
Example: Socrata Provider
Registered as an item in ArcGIS Online:
dcdev.maps.arcgis.com/home/item.html?id=8b6ecc9845944908a77a197eed908838Shared through Open Data:
opendata.arcgis.com/datasets/dcdev::seattle-socrata-pmap-kbvrKoop Providers (for Koop >= 3.0)
Requirements
- Docs are found at koopjs.github.io/docs/specs/provider/
- Implement a single function
- That function fetches data from remote API and translates it into GeoJSON
- Koop handles the rest
- Slides for Koop Presentation: www.slideshare.net/DanielFenton2/koop-using-3rd-party-services-in-arcgis
Koop Demo
- Open Data from City of DC
- Koop Providers for Zillow, Craigslist, and Yelp
- Demo
DCAT
geohub.lacity.org/data.json
or:
<your site>-<your organization>.opendata.arcgis.com/data.json
DCAT enables integration with CKAN portals such as data.gov
Open Data API
What is the Open Data API?
It is not the Portal API
It is not the GeoServices API
It is used for finding Open Data Datasets & managing Open Data Sites
Simple API example
https://opendata.arcgis.com
/api/v2
/datasets
?q=crime&page[size]=5&page[number]=3
An Aside on the V1 API
Don't Use It
What uses the OpenData API?
OpenData "Umbrella" (Ember)
OpenData Custom Sites 2.X (Ember)
OpenData Admin 2.0 (Ember)
OpenData V2 API
Versioning via URL
API returns JSON
adheres to JSONAPI standard
/api/v2/{:resource}
/api/v2/{:resource}/api/v2/{:resource}/{:id}
/api/v2/{:resource}/{:id}/api/v2/{:resource}/{:id}/{:related_resource}
/api/v2/{:resource}/{:id}/{:related_resource}Why JSONAPI?
Standardizes JSON responses
Supports features like side-loading
Supports client-defined payloads
Has many client and server implementations
Go to http://jsonapi.org/implementations/ for client and server libraries
Go to http://jsonapi.org/format for the spec
OpenData V2 API: Query Abilities
Full text queries supported with the `q` parameter
Can 'side-load' related resources in a single request via `include` parameter
"Bring me all the datasets that match the query 'parcel' and also include the organizations that published those datasets"
OpenData V2 API: Response Format
Standardized response for a resource or a collection of resources
/api/v2/datasets -> {"data": []}/api/v2/datasets/:id -> {"data": {}}OpenData V2 API: Object Format
Each object has a standard format
{
"id": "f2e1c2ef9eb44f2899f4a310a80ecec9_2",
"type": "dataset",
"attributes": {
"name": ...,
"url": ...,
"recordCount": ...,
}
}
OpenData V2 API: Side-Loaded Resources
How are side-loaded resources returned?
{
"data": [...],
"included": [
{
"id": "afd7refdr",
"type": "organization",
"attributes": {
"name": ...,
"homePageUrl": ...,
}
}
]
}
OpenData V2 API: Metadata
Aggregations included on each request for collections
{
"data": [...],
"meta": {
"apiRoot": "https://opendata.arcgis.com/api/v2/",
"resourceRoot": "https://opendata.arcgis.com/api/v2/datasets/",
"queryParameters": {
query-parameters-of-request...
},
"stats": {
counts-and-other-aggregations
}
}
}
OpenData V2 API: Request Metadata Example
/api/v2/datasets?q=population&filter[content]=spatial%20dataset&page[size]=25
{
"meta": {
"apiRoot": "http://opendata.arcgis.com/api/v2",
"resourceRoot": "http://opendata.arcgis.com/api/v2/datasets",
"queryParameters": {
"page": {
"number": 1,
"size": 25
},
"q": "population",
"filter": {
"content": "spatial dataset"
}
},
"stats": {
"count": 25,
"totalCount": 186,
"aggs": {
"content": [{ "key": "spatial dataset", "docCount": 186 }],
"tags": [
{ "key": "census", "docCount": 101 },
{ "key": "population", "docCount": 97 },
{ "key": "demographics", "docCount": 71 },
{ "key": "acs": "docCount": 21}
],
"source": [
{ "key": "U.S. Federal Maps and Apps", "docCount": 24 },
{ "key": "Miami-Dade County, Florida", "docCount": 11 }
]
}
}
}
}OpenData V2 API: Pagination
/api/v2/datasets?q=population&filter[content]=spatial%20dataset&page[size]=25
{
"data": [...],
"meta": {...},
"links": {
"first": "https://opendata.arcgis.com/api/v2/datasets?page[number]=1&page[size]=25&q=population&filter[content]=spatial%20dataset",
"next": "https://opendata.arcgis.com/api/v2/datasets?page[number]=2&page[size]=25&q=population&filter[content]=spatial%20dataset",
"last": "https://opendata.arcgis.com/api/v2/datasets?page[number]=8&page[size]=25&q=population&filter[content]=spatial%20dataset"
}
}
OpenData V2 API: Standardized Errors
/api/v2/datasets?foo=bar
{
"errors": [
{
"title": "Unrecognized Parameter",
"detail": "'foo' is not a recognized parameter for this request.",
"status": 400,
"source": { "parameter": "foo=bar" },
"meta": { detailed-information-to-debug-the-request }
}
]
}
OpenData V2 API: More Standardized Errors
/api/v2/datasets?foo=bar&filter[pizza]=anchovies
All errors are processed before a response is generated
{
"errors": [
{
"title": "Unrecognized Parameter",
"detail": "'foo' is not a recognized parameter for this request.",
"status": 400,
"source": { "parameter": "foo=bar" },
"meta": { detailed-information-to-help-debug-the-request }
},
{
"title": "Invalid filter key",
"detail": "'filter[pizza]' is not a valid filter key",
"status": 400,
"source": { "parameter": "filter[pizza]" },
"meta": { detailed-information-to-help-debug-the-request }
}
]
}
OpenData V2 API Sandbox
Unoffical ArcGIS Open Data API Sandbox at
Embedded Open Data
What can you do with the Open Data API?
Build an app!
ArcGIS Open Data
$.getJSON('http://my.api.url')
.done(function (response, status, xhr) {
/* stuff it into the DOM... */
})
.fail(function (xhr, status, error) {
/* do something about the error! */
});
$.getJSON('http://my.api.url')
.done(function (response, status, xhr) {
/* do something with the response! */
})
.fail(function (xhr, status, error) {
/* do something about the error! */
});
MV*
Key endpoints:
And maybe:
Also:
Errors...
$.getJSON('http://my.api.url')
.done(function (response, status, xhr) {
if (response.error) {
/* you don't have to do this! */
}
})
.fail(function (xhr, status, error) {
switch (xhr.status) {
case 404:
/* handle 404 */
default:
/* handle other errors */
}
});
Gotcha's:
- SSL only (https://opendata.arcgis.com/api/v2)
- Currently no JSONP support
- Does support CORS
- But ... IE < 10
esridc.github.io/OpenData-Backbone
Ok,... how?
Fork it: github.com/esridc/OpenData-Backbone
then:
$ git clone https://github.com/esridc/OpenData-Backbone.git
$ cd opendata-backbone
$ npm install
$ bower install
$ gulp serve
In backbone-land...
DatasetCollection = Backbone.Collection.extend({
url: function () {
//get the params (q=, page=, etc) from somewhere...
var queryParams = '';
return MyOD.config.api + 'datasets?' + queryParams;
},
parse: function (resp) {
return resp.data;
}
});
var datasetCollection = new DatasetCollection();
datasetCollection.fetch({ ... });
DatasetModel = Backbone.Model.extend({
url: function () {
return MyOD.config.api + 'datasets/' + this.get('id');
},
parse: function (response) {
return response.data;
}
});
var datasetModel = new DatasetModel();
datasetModel.fetch({ id: myId });
esridc.github.io/opendata-ember
Fork it: github.com/esridc/opendata-ember
then:
$ npm install -g ember-cli
$ git clone https://github.com/esridc/opendata-ember.git
$ cd opendata-ember
$ npm install
$ bower install
$ ember serve
In ember-land...
import Ember from 'ember';
import DS from 'ember-data';
export default DS.JSONAPIAdapter.extend({
host: ENV.APP.API,
namespace: 'api/v2',
pathForType: function(type) {
const camelized = Ember.String.camelize(type);
return Ember.String.pluralize(camelized);
},
urlForFindRecord: function (id/*, modelName, snapshot*/) {
const host = this.get('host');
const namespace = this.get('namespace');
return `${host}/${namespace}/datasets/${id}`;
}
});import Ember from 'ember';
import DS from 'ember-data';
export default DS.JSONAPISerializer.extend({
normalizeResponse: function (store, primaryModelClass, payload/*, id, requestType*/) {
if (Ember.isArray(payload.data)) {
payload.data = payload.data.map(this._mapDataset);
} else {
payload.data = this._mapDataset(payload.data);
}
return payload;
},
_mapDataset: function (item) {
if(!item.attributes.name){
item.attributes.name = item.attributes.item_name;
}
return item;
}
});Or:
$ ember install ember-arcgis-opendata-services
// datasets route
model: function (params) {
const qryParams = ...
return this.store.query('dataset', qryParams);
},
// datasets.dataset route
model: function (params) {
return this.store.findRecord('dataset', params.id);
},
More to come?
- mjuniper.github.io/presentations/ds2017/integrating-opendata.html
- koopjs.github.io
- jsonapi.org
- akharris.github.io/ember-arc-swag (not API feature-complete but still useful)
- emberjs.com
- backbonejs.org
Esri Dev Summit 2017 | Integrating Open Data
Integrating Open Data
Mike Juniper :: Washington DC R&D Center
Alex Harris :: Washington DC R&D Center