Metadata
HDX Metadata
All of the data in HDX HAPI comes from publicly-available datasets on HDX, and
the HDX metadata can be used refer back to the original data to see what
has been simplified or omitted in the API.
The two primary HDX metadata tables are
dataset and resource, and they
contain a small subset of the metadata available through the CKAN API or
HDX library.
The HAPI data records contain enough information to access the full records from CKAN (see the parameters tables for more details). All subcategory tables link back to a resource record, and all resources link back to a dataset record.
Dataset
Used in: Resource, all sub-category tables
This table contains the HDX-specific metadata associated with all datasets used to populate the HAPI sub-category tables. Every dataset has at least one child resource.
Parameters Returned
The table below describes the parameters returned from this endpoint. For available query parameters, please see the API sandbox.
| Parameter | Description |
|---|---|
dataset_hdx_id |
Unique dataset UUID on HDX, which does not change even when changes are made to the dataset. It can be used to construct a URL to access the dataset on HDX: https://data.humdata.org/dataset/[dataset_hdx_id], which can be found in the hdx_link field. It can also be used to access the dataset through the HDX CKAN API: https://data.humdata.org/api/action/package_show?id=[dataset_hdx_id], as shown in the hdx_api_link field. |
dataset_hdx_stub |
The unique, URL-safe name of the dataset on HDX, used to create a human-readable URL to the dataset on HDX: https://data.humdata.org/dataset/[dataset_hdx_stub]. The stub may change at any time (unlike the UUID). |
dataset_hdx_title |
Dataset title on HDX |
hdx_provider_stub |
The unique, URL-safe dataset provider name, used to access the the dataset provider's organization page on HDX through the following URL pattern: https://data.humdata.org/organization/[hdx_provider_stub], which can be found in the provider_hdx_link field. It can also be used to access the provider's information through the HDX CKAN API: https://data.humdata.org/api/action/organization_show?id=[hdx_provider_stub], as shown in the provider_hdx_api_link field. |
hdx_provider_name |
The name of the organization that provided the dataset to HDX |
hdx_link |
A link to the dataset on HDX, constructed using the UUID from the dataset_hdx_id field |
hdx_api_link |
A link to the dataset on the HDX CKAN API, constructed using the UUID from the dataset_hdx_id field |
provider_hdx_link |
A link to the dataset provider's organization page, constructed using the data provider stub from the hdx_provider_stub field |
provider_hdx_api_link |
A link to the dataset provider's information via the HDX CKAN API, constructed using the data provider stub from the hdx_provider_stub field |
Resource
Used in: All sub-category tables
This table contains the HDX-specific metadata associated with all resources used to populate the HAPI sub-category tables. Every resource has one parent dataset.
Parameters Returned
The table below describes the parameters returned from this endpoint. For available query parameters, please see the API sandbox.
| Parameter | Description | Source |
|---|---|---|
resource_hdx_id |
Unique resource UUID on HDX, which does not change even when changes are made to the resource. In combination with the dataset UUID from the dataset_hdx_id field, it can be used to construct a URL to the resource on HDX: https://data.humdata.org/dataset/[dataset_hdx_id]/resource/[resource_hdx_id], which can be found in the hdx_link field. It can also be used to access the resource through the HDX CKAN API: https://data.humdata.org/api/action/resource_show?id=[resource_hdx_id], which is shown in the hdx_api_link field. |
|
name |
The resource name on HDX. In combination with the dataset UUIDs from the dataset_hdx_id and resource_hdx_id fields respectively, it can be used to construct a URL to download the resource: https://data.humdata.org/dataset/[dataset_hdx_id]/resource/[resource_hdx_id]/download/[name], which can be found in the download_url field. |
|
hdx_link |
A link to the resource on HDX, constructing using the UUIDs from the dataset_hdx_id and resource_hdx_id fields |
|
hdx_api_link |
A URL to access the resource on the HDX CKAN API, constructed using the UUID from the resource_hdx_id field |
|
download_url |
nan | nan |
dataset_hdx_id |
Unique dataset UUID on HDX, which does not change even when changes are made to the dataset. It can be used to construct a URL to access the dataset on HDX: https://data.humdata.org/dataset/[dataset_hdx_id], which can be found in the dataset_hdx_link field. It can also be used to access the dataset through the HDX CKAN API: https://data.humdata.org/api/action/package_show?id=[dataset_hdx_id], as shown in the dataset_hdx_api_link field. |
Dataset |
format |
The format of the resource file, such as csv or Excel | |
update_date |
The date the resource was last updated | |
is_hxl |
A boolean to indicate whether the resource data is HXLated | |
hapi_updated_date |
The date that the resource was ingested into HDX HAPI | |
dataset_hdx_stub |
The unique, URL-safe name of the dataset on HDX, which is used to create a human-readable URL to the dataset on HDX: https://data.humdata.org/dataset/[dataset_hdx_stub]. The stub may change at any time (unlike the UUID). |
Dataset |
dataset_hdx_title |
A URL to directly download the resource file from HDX, in the format specified in the format field |
Dataset |
dataset_hdx_link |
A URL to access the dataset on HDX, constructed using the UUID from the dataset_hdx_id field |
Dataset |
dataset_hdx_api_link |
A URL to access the dataset on the HDX CKAN API, constructed using the UUID from the dataset_hdx_id field |
Dataset |
dataset_hdx_provider_stub |
The unique, URL-safe dataset provider name, used to access the the dataset provider's organization page on HDX through the following URL pattern: https://data.humdata.org/organization/[dataset_hdx_provider_stub], which can be found in the provider_hdx_link field. It can also be used to access the provider's information through the HDX CKAN API: https://data.humdata.org/api/action/organization_show?id=[dataset_hdx_provider_stub], as shown in the provider_hdx_api_link field. |
Dataset |
dataset_hdx_provider_name |
The name of the organization that provided the dataset to HDX | Dataset |
provider_hdx_link |
A URL to access the dataset provider's organization page, constructed using the data provider stub from the dataset_hdx_provider_stub field |
Dataset |
provider_hdx_api_link |
A URL to access the dataset provider's information via the HDX CKAN API, constructed using the data provider stub from the dataset_hdx_provider_stub field |
Dataset |
Geographical Metadata
HAPI supports three hierarchical levels of geographical metadata: location (a country or country-like entity), admin 1, and admin 2. An entry in the location table does not necessarily imply UN recognition of statehood. The subcategory data tables link to the lowest administrative level used by that data type; it will usually be admin 2, but in some cases may be admin 1 or location.
The names and p-codes are read in from a
global p-code list
taken from the
common operational dataset
(COD) gazetteers and administrative boundaries.
In the geographical tables, the code (p-code) field is unique only in
combination with reference_period_start,
since p-codes may be reused in different versions of geographical metadata.
When bringing in data from sub-categories, p-codes are used if
present. Sometimes, these p-codes may not strictly follow the standard with
variations such as shorter or longer country or admin unit codes.
HDX HAPI utilises p-code handling provided by the
hdx-python-country
library, which has a p-code length matching algorithm to
ensure correct admin units are determined.
This algorithm works as follows: First, a p-code is verified against the expected p-code format. Then, a length-matching check is performed, ensuring that for each segment (country code or admin unit code), the length matches the predefined format for the country. For example, a 3-letter country code could be converted to a 2-letter one or vice-versa. Digits can be prefixed by 0's to increase the length, or have 0's dropped from the prefix.
Location
Country or country-like entities in HDX HAPI, from the CODs global p-code list.
Parameters Returned
The table below describes the parameters returned from this endpoint. For available query parameters, please see the API sandbox.
| Parameter | Description | Source |
|---|---|---|
from_cods |
Whether the source of the entry is the common operational datasets (CODs) | |
has_hrp |
Whether the location has a "Humanitarian Response Plan" | |
in_gho |
Whether the location is in the Global Humanitarian Overview | |
reference_period_start |
The start date for which the data are applicable | |
reference_period_end |
The end date for which the data are applicable | |
code |
Location p-code, based on the ISO-3 (ISO 3166 alpha-3) standard | |
name |
Location name, based on the "short name" from the UN M49 Standard | |
id |
Unique identifier associated with each location entry |
Admin 1
Admin 1 level names and p-codes in HDX HAPI, from the CODs global p-code list.
Parameters Returned
The table below describes the parameters returned from this endpoint. For available query parameters, please see the API sandbox.
| Parameter | Description | Source |
|---|---|---|
code |
Admin 1 p-code | |
name |
Admin 1 name | |
id |
A stable, unique identifier associated with each admin 1 entry | |
from_cods |
Whether the source of the entry is the common operational datasets (CODs) | |
reference_period_start |
The start date for which the data are applicable | |
reference_period_end |
The end date for which the data are applicable | |
location_code |
Location p-code, based on the ISO-3 (ISO 3166 alpha-3) standard | Location |
location_name |
Location name, based on the "short name" from the UN M49 Standard | Location |
location_ref |
A reference to the id of the associated entry in the location table |
Location |
Admin 2
Admin 2 level names and p-codes in HDX HAPI, from the CODs global p-code list.
Parameters Returned
The table below describes the parameters returned from this endpoint. For available query parameters, please see the API sandbox.
| Parameter | Description | Source |
|---|---|---|
code |
Admin 2 p-code | |
name |
Admin 2 name | |
id |
A stable, unique identifier associated with each admin 2 entry | |
from_cods |
Whether the source of the entry is the common operational datasets (CODs) | |
reference_period_start |
The start date for which the data are applicable | |
reference_period_end |
The end date for which the data are applicable | |
location_code |
Location p-code, based on the ISO-3 (ISO 3166 alpha-3) standard | Location |
location_name |
Location name, based on the "short name" from the UN M49 Standard | Location |
location_ref |
A reference to the id of the associated entry in the location table |
Location |
admin1_code |
Admin 1 p-code | Admin 1 |
admin1_name |
Admin 1 name | Admin 1 |
admin1_ref |
A reference to the id of the associated entry in the admin 1 table |
Admin 1 |
Sub-category Metadata
Currency
Used in:
Food Prices
The currency table is populated using the WFP VAM Data Bridges API.
Parameters Returned
The table below describes the parameters returned from this endpoint. For available query parameters, please see the API sandbox.
| Parameter | Description |
|---|---|
code |
ISO-4217 currency code |
name |
Currency name |
Org
Used in:
Who is Doing What Where - Operational Presence
The organization table is populated from the 3W data, using the following methodology:
- Organization name and acronym strings are normalised. If an acronym isn’t available, the first 32 characters of the name are used.
- This organization mapping is used for common alternative names
- Organizations must have an associated organization type. If available, the organization type code is taken directly from the 3W data, otherwise the name string is normalised and matched to the org type names. In the absence of a direct match, phonetic matching is used for strings > 5 characters. If no match is found, the organization is skipped.
Parameters Returned
The table below describes the parameters returned from this endpoint. For available query parameters, please see the API sandbox.
| Parameter | Description | Source |
|---|---|---|
acronym |
The organization acronym | |
name |
The organization name | |
org_type_code |
The code referring to the organization type | Org Type |
org_type_description |
A description of the organization type | Org Type |
Org Type
Used in:
Org,
Who is Doing What Where - Operational Presence
The table is initially populated using the OCHA Digital Services organization types list. The following rows are then added:
| Code | Description |
|---|---|
| 501 | Civil Society |
| 502 | Observer |
| 503 | Development Programme |
| 504 | Local NGO |
Organization types all have an associated description and code.
Parameters Returned
The table below describes the parameters returned from this endpoint. For available query parameters, please see the API sandbox.
| Parameter | Description |
|---|---|
code |
The code referring to the organization type, derived either from the OCHA Digital Services organization types list, or created for HDX HAPI |
description |
A description of the organization type |
Sector
Used in:
Who is Doing What Where - Operational Presence,
Humanitarian Needs
This table is initially populated using the Global Coordination Groups dataset. The following rows are then added:
| Code | Name |
|---|---|
| Cash | Cash programming |
| Hum | Humanitarian assistance (unspecified) |
| Multi | Multi-sector (unspecified) |
| Intersectoral | Intersectoral |
Sectors all have an associated name and code.
Parameters Returned
The table below describes the parameters returned from this endpoint. For available query parameters, please see the API sandbox.
| Parameter | Description |
|---|---|
code |
The sector code, derived either from the Global Coordination Groups dataset, or created for HDX HAPI |
name |
The name of the sector |
WFP Commodity
Used in:
Food Prices
The commodity table tracks all food items, and their associated commodity category, present in the food prices data.
Parameters Returned
The table below describes the parameters returned from this endpoint. For available query parameters, please see the API sandbox.
| Parameter | Description | Source |
|---|---|---|
code |
The unique code identifying the commodity | |
name |
The name of the commodity | |
category |
The food group that the commodity belongs to | Commodity Category |
WFP Market
Used in:
Food Prices
Markets are defined as the physical locations where buyers and sellers
come together to trade goods and services.
This table is populated from the food prices data, which contains location
names down to admin 2 as well as
a latitude and longitude, but is not p-coded. Consequently, admin 1 and 2
names must be matched to p-codes using the algorithm provided by the
hdx-python-country
library which uses phonetic name matching and
manual overrides.
Parameters Returned
The table below describes the parameters returned from this endpoint. For available query parameters, please see the API sandbox.
| Parameter | Description | Source |
|---|---|---|
code |
The unique code identifying the market | |
name |
The name of the market | |
lat |
The market's latitude | |
lon |
The market's longitude | |
location_code |
Location p-code, based on the ISO-3 (ISO 3166 alpha-3) standard | Location |
location_name |
Location name, based on the "short name" from the UN M49 Standard | Location |
location_ref |
A reference to the id of the associated entry in the location table |
Location |
admin1_code |
Admin 1 p-code | Admin 1 |
admin1_name |
Admin 1 name | Admin 1 |
provider_admin1_name |
Admin 1 names provided in the original source data | |
admin1_ref |
A reference to the id of the associated entry in the admin 1 table |
Admin 1 |
admin2_code |
Admin 2 p-code | Admin 2 |
admin2_name |
Admin 2 name | Admin 2 |
provider_admin2_name |
Admin 2 names provided in the original source data | |
admin2_ref |
A reference to the id of the associated entry in the admin 2 table |
Admin 2 |
HDX HAPI Metadata
Data Availability
The data availability endpoint in HDX HAPI shows what data is accessible by sub-category and administrative level. If relevant data exists for a specific sub-category and region, it will be included in the query results, along with the date that it was last updated in HDX HAPI.
Where data are available to the country (admin 0) level, then the admin 1 and 2 code and name fields will contain empty strings. Where the data are available to the admin 1 level, then the admin 2 code and name fields will contain empty strings. Note that some sub-categories can have data available at multiple administrative levels.
Parameters Returned
The table below describes the parameters returned from this endpoint. For available query parameters, please see the API sandbox
| Parameter | Description | Source |
|---|---|---|
category |
The category of data, based on the HDX data grids | |
subcategory |
The sub-category of data, based on the HDX data grids | |
hapi_updated_date |
Date that the data was last updated in HAPI. Note that this differs from the data's last update on HDX and does not reflect how current the data is | |
location_code |
Location p-code, based on the ISO-3 (ISO 3166 alpha-3) standard | Location |
location_name |
Location name, based on the "short name" from the UN M49 Standard | Location |
admin1_code |
Admin 1 p-code | Admin 1 |
admin1_name |
Admin 1 name | Admin 1 |
admin2_code |
Admin 2 p-code | Admin 2 |
admin2_name |
Admin 2 name | Admin 2 |