DFx ETL Pipeline

A Python package containing an ETL pipeline for processing indicator data used by the Data Futures Exchange (DFx).

Warning

The package is currently undergoing a major revamp. Some features may be missing or not working as intended. Feel free to open an issue.

Table of Contents

1. Introduction
2. Installation
3. Usage
4. Sources
5. Contributing
6. License
7. Contact

---

Introduction

The Data Futures Exchange (DFx) is an analytics platform designed to support data-driven decision-making at UNDP. This repository hosts a Python package developed to collect, process and consolidate indicator data from various sources for the use on the DFx. The project is structured as follows:

.
├── .github/                  # GitHub templates and settings
│   └── ...
├── src/                      # package source code
│   └── dfx_etl/
│       ├── data/             # auxiliary data shipped with the package
│       │   └── unsd-m49.csv  # https://unstats.un.org/unsd/methodology/m49
│       ├── pipelines/        # ETL pipelines for various sources
│       │   ├── __init__.py
│       │   ├── _base.py      # base classes to inherit from for new sources
│       │   ├── energydata_info.py 
│       │   ├── ...
│       │   └── world_bank.py # source-specific ETL steps
│       ├── storage/          # supported storage backends
│       │   ├── __init__.py
│       │   ├── _base.py      # base class to inherit from for new backends
│       │   ├── azure.py      # Azure Blob Storage backend
│       │   └── local.py      # local storage backend
│       ├── __init__.py
│       ├── utils.py          # utility functions
│       └── validation.py     # validation schemas and functions
├── .env.example              # .env files example
├── .gitattributes            # special git attributes
├── .gitignore                # untracked path patterns
├── CONTRIBUTING.md           # guidelines for contributors
├── LICENSE                   # license terms for the package
├── main.ipynb                # main notebook for execuring pipelines
├── Makefile                  # make commands for routine operations
├── pyproject.toml            # Python package metadata
├── README.md                 # this file
├── requirements_dev.txt      # development dependencies
└── requirements.txt          # core dependencies

Installation

ensure git is installed and configured

using git

 pip install git+https://github.com/undp-data/dfx-etl-pipeline

See VCS Support for more details.

using uv

1. ensure uv is installed

2. clone the repo

 git clone https://github.com/UNDP-Data/dfx-indicators-etl.git

3. setup

 uv venv #create a virtual env in local folder
 uv pip install . # install the dependencies 
 

Python API based usage

The package provides ETL – or rather RTL – routines for a range of supported sources. Running a pipeline involves 3 steps:

1. Retriving raw data using a source-specific Retriever.
2. Transforming raw data using a source-specific Transformer, which also validates the data.
3. Loading validated data to a supported storage backend.

These 3 steps are abstracted away in a single Pipeline class. The basic usage is demonstrated below.

# import a supported backend and source
from dfx_etl.storage import AzureStorage
from dfx_etl.pipelines import who_int as source, Pipeline

# instantiate a pipeline with relevant metadata
pipeline = Pipeline(
    url="https://ghoapi.azureedge.net/api/",
    retriever=source.Retriever(),
    transformer=source.Transformer(),
    storage=storage,
)

# run the ETL steps one by one
pipeline.retrieve()
pipeline.transform()
pipeline.load()
# or just call the `pipeline` to run them all at once
pipeline()

For more details see main.ipynb.

Warning

Running the pipeline requires the .env file at the root of the project. Depending on the storage backend, you might need to set different types of variables. See .env.example.

Sources

Below is the list of sources currently supported by the package.

Name	URL	Manual filename
IHME	GBD 2021 dataset	IHME_GBD*
ILO	ILOSTAT SDMX API
IMF	IMF DataMapper API
SIPRI	SIPRI Milex dataset	SIPRI-Milex-data*
UNAIDS	UNAIDS Key Population Atlas	KPAtlasDB*
UNICEF	UNICEF SDMX API
UN Stats	UN Stats Database Zip	UNSTATS_ZIP*
WHO	WHO GHO API
World Bank	World Bank WDI CSV Datasheet	WDICSV*

Contributing

See CONTRIBUTING.md.

License

This project is licensed under the GNU General Public License. See the LICENSE file.

Contact

This project is part of Data Futures Exchange (DFx) at UNDP. If you are facing any issues or would like to make some suggestions, feel free to open an issue. For enquiries about DFx, visit Contact Us.