Code Examples
In each of these code examples you will need to insert your generated app
identifier in place of { your app identifier } text. To generate an app
identifier follow the getting started guide.
1. Query a sub-category end point and loop through pages
Themes are the core data of the API. The results are paginated and so
multiple calls are needed to get the whole dataset. Below we query the
operational presence sub-category for Afghanistan and return all results
into a single object. To query a different sub-category or country change
the constant variable of THEME to another sub-category or LOCATION to
a different ISO3 country code.
import json
from urllib import request
def fetch_data(base_url, limit=1000):
"""
Fetch data from the provided base_url with pagination support.
Args:
- base_url (str): The base URL endpoint to fetch data from.
- limit (int): The number of records to fetch per request.
Returns:
- list: A list of fetched results.
"""
idx = 0
results = []
while True:
offset = idx * limit
url = f"{base_url}&offset={offset}&limit={limit}"
with request.urlopen(url) as response:
print(f"Getting results {offset} to {offset+limit-1}")
json_response = json.loads(response.read())
results.extend(json_response['data'])
# If the returned results are less than the limit,
# it's the last page
if len(json_response['data']) < limit:
break
idx += 1
return results
APP_IDENTIFIER = { your app identifier }
THEME = "coordination-context/operational-presence"
LOCATION = "AFG"
BASE_URL = (
f"https://hapi.humdata.org/api/v1/{THEME}?"
f"output_format=json"
f"&location_code={LOCATION}"
f"&app_identifier={APP_IDENTIFIER}"
)
LIMIT = 1000
results = fetch_data(BASE_URL, LIMIT)
print(results)
async function fetchData(baseUrl, limit = 1000) {
let idx = 0;
let results = [];
while (true) {
const offset = idx * limit;
const url = `${baseUrl}&offset=${offset}&limit=${limit}`;
console.log(`Getting results ${offset} to ${offset + limit - 1}`);
const response = await fetch(url);
const jsonResponse = await response.json();
results = results.concat(jsonResponse['data']);
// If the returned results are less than the limit, it's the last page
if (jsonResponse['data'].length < limit) {
break;
}
idx++;
}
return results;
}
const APP_IDENTIFIER = { your app identifer }
const THEME = "coordination-context/operational-presence"
const LOCATION = "AFG"
const BASE_URL = `https://hapi.humdata.org/api/v1/${THEME}?` +
`output_format=json` +
`&location_code=${LOCATION}` +
`&app_identifier=${APP_IDENTIFIER}`;
const LIMIT = 1000;
window.onload = async function() {
const results = await fetchData(BASE_URL, LIMIT);
console.log(results);
};
import fetch from 'node-fetch'
async function fetchData(baseUrl, limit = 1000) {
let idx = 0;
let results = [];
while (true) {
const offset = idx * limit;
const url = `${baseUrl}&offset=${offset}&limit=${limit}`;
console.log(`Getting results ${offset} to ${offset + limit - 1}`);
const response = await fetch(url);
const jsonResponse = await response.json();
results = results.concat(jsonResponse['data']);
// If the returned results are less than the limit, it's the last page
if (jsonResponse['data'].length < limit) {
break;
}
idx++;
}
return results;
}
const APP_IDENTIFIER = { your app identifer }
const THEME = "coordination-context/operational-presence"
const LOCATION = "AFG"
const BASE_URL = `https://hapi.humdata.org/api/v1/${THEME}?` +
`output_format=json` +
`&location_code=${LOCATION}` +
`&app_identifier=${APP_IDENTIFIER}`;
const LIMIT = 1000;
fetchData(BASE_URL, LIMIT).then(results => {
console.log(results);
});
library(jsonlite)
library(httr)
fetch_data <- function(base_url, limit = 1000) {
idx <- 0
results <- list()
while(TRUE) {
offset <- idx * limit
url <- paste0(base_url, "&offset=", offset, "&limit=", limit)
response <- GET(url)
cat("Getting results", offset, "to", offset + limit - 1, "\n")
json_response <- fromJSON(content(response, "text"))
results <- append(results, list(json_response$data))
# If the returned results are less than the limit, it's the last page
if(length(json_response$data) < limit) {
break
}
idx <- idx + 1
}
return(results)
}
APP_IDENTIFIER <- { your app identifier }
THEME <- "coordination-context/operational-presence"
LOCATION <- "AFG"
BASE_URL <- paste0(
"https://hapi.humdata.org/api/v1/", THEME, "?",
"output_format=json",
"&location_code=", LOCATION,
"&app_identifier=", APP_IDENTIFIER
)
LIMIT <- 1000
results <- fetch_data(BASE_URL, LIMIT)
print(results)
2. Filtering results
It is possible to add extra filters to the call to get a subset of results. To see the full set of filters that can be used for each sub-category, please check the API endpoint documentation.
Filter by Sector
Change the code to include a new parameter in the URL.
Filter by Admin 1 P-code
3. Filter for admin level
Some sub-categories have data at multiple admin levels. If you don't filter for a particular level, you could receive data from multiple levels in one call. To filter for only admin 1 data, add this parameter to the URL:
4. Get data from supporting tables
Each supporting table such as orgs, orgs_type, sector and more have a
unique URL to call to get the range of possible values. Below we show the
base URL for getting of the sector names and codes.
5. Getting dataset metadata for a resource
Each row returned from a sub-category endpoint contains the field
resource_hdx_id, which can be used to obtain the dataset metadata,
including information about the contributing organization,
from the resource endpoint
6. Get admin level data for a country
The admin 1 and admin 2 API endpoints provide data about subnational administrative boundary names and p-codes (place codes). Below is an example that will retrieve the admin 1 level information for Afghanistan:
The geometry associated with each admin boundary is not yet available via the API, but it can be downloaded from HDX.
7. Download as CSV
The code examples so far have been using JSON output and then processing this
data. To query this data as csv, change the output_format to csv as per
the examples below. Visiting this URL through the browser will download the
CSV to then be used on your computer.
8. Load data into Google Sheets using an app script and periodically update
1. Create a New Google Spreadsheet:
Create a new Google Spreadsheet where you want to load the data.
2. Open the Script Editor:
Click on Extensions -> Apps Script in the top menu to open the Google Apps Script editor.
3. Use this code:
A simple script that will fetch the API data and place it in the spreadsheet
function loadApiData() {
var baseUrl = "https://hapi.humdata.org/api/v1/coordination-context/" +
"operational-presence?output_format=json&" +
"location_code=AFG&app_identifier={ your app identifier }";
var limit = 10000;
var offset = 0;
var allData = [];
while (true) {
// Fetch data from the current page
var url = baseUrl + "&offset=" + offset + "&limit=" + limit;
var response = UrlFetchApp.fetch(url);
var jsonData = JSON.parse(response.getContentText());
// If there's no data or less data than the limit, break out of the loop
if (!jsonData.data.length || jsonData.data.length < limit) {
allData = allData.concat(jsonData.data);
break;
}
// Otherwise, store the data and increment the offset for the next page
allData = allData.concat(jsonData);
offset += limit;
}
// Convert the JSON data to a 2D array for the spreadsheet
var dataArray = [];
// If there's data, add headers from the first item's keys
if (allData.length > 0) {
dataArray.push(Object.keys(allData[0]));
}
allData.forEach(function(item) {
var row = [];
for (var key in item) {
row.push(item[key]);
}
dataArray.push(row);
});
// Get the active sheet and clear existing data
var sheet = SpreadsheetApp.getActiveSpreadsheet().getActiveSheet();
sheet.clear();
// Write the headers and data to the spreadsheet
if (dataArray.length > 0) {
sheet.getRange(1, 1, dataArray.length,
dataArray[0].length).setValues(dataArray);
}
}
4. Set up a Daily Trigger:
- Click on the clock icon on the left sidebar to view the project's triggers.
- Click on the + Add Trigger button at the bottom right.
- Set the function to loadApiData, the deployment to Head, the event source to Time-driven, and then select Day timer to choose a specific time of day.
- Click Save.
5. Authorize the Script:
When you run the script for the first time or set up a trigger, it will ask for permissions. Make sure to grant them so the script can access the external API and modify your Google Spreadsheet.
Now, the script will run daily at the time you specified and load the API data into your Google Spreadsheet.