# Country Pages Data

Versioned country data fetched from providers (currently Superset) and published through the UNDRR asset library.

## Structure

```
data/
├── v1.0.0/
│   ├── countries/
│   │   ├── AFG.json
│   │   ├── ALB.json
│   │   └── ...
│   ├── index.json
│   ├── metadata.json
│   └── provenance.json
├── v1.0.1/
│   └── ...
└── latest/ -> v1.0.1 (symlink)
```

**Source script:** [`scripts/fetch-country-data.js`](../../../scripts/fetch-country-data.js) in the repository root.

## Data Format

### Country File (`countries/{ISO3_CODE}.json`)

```json
{
  "countryCode": "AFG",
  "countryName": "Afghanistan",
  "provider": "superset",
  "fetchedAt": "2026-02-03T12:00:00.000Z",
  "data": {
    // Provider-specific data structure
  }
}
```

### Index File (`index.json`)

```json
{
  "version": "v1.0.0",
  "publishedAt": "2026-02-03T12:00:00.000Z",
  "provider": "superset",
  "countries": ["AFG", "ALB", "..."],
  "totalCount": 195
}
```

### Metadata File (`metadata.json`)

```json
{
  "version": "v1.0.0",
  "publishedAt": "2026-02-03T12:00:00.000Z",
  "source": {
    "provider": "superset",
    "baseUrl": "https://dash.giri.unepgrid.ch",
    "exploreUrl": "https://dash.giri.unepgrid.ch/explore/p/VoAYOvNKR4P/",
    "permalinkId": "VoAYOvNKR4P",
    "provenance": "v1.0.0/provenance.json"
  },
  "structure": {
    "countries": "v1.0.0/countries/{ISO3_CODE}.json",
    "index": "v1.0.0/index.json"
  },
  "usage": {
    "example": "https://assets.undrr.org/projects/pw-country-pages/data/v1.0.0/countries/AFG.json"
  },
  "statistics": {
    "totalCountries": 195,
    "failedCountries": 0
  }
}
```

### Provenance File (`provenance.json`)

Version-level record of query specification and source details, so published data can be traced back to the Superset visualization and API endpoint.

```json
{
  "version": "v1.0.0",
  "publishedAt": "2026-02-03T12:00:00.000Z",
  "source": {
    "provider": "superset",
    "baseUrl": "https://dash.giri.unepgrid.ch",
    "permalinkId": "VoAYOvNKR4P",
    "datasourceId": 10,
    "datasourceType": "table",
    "exploreUrl": "https://dash.giri.unepgrid.ch/explore/p/VoAYOvNKR4P/"
  },
  "apiEndpoint": "/api/v1/chart/data",
  "query": {
    "columns": ["climate_scenario", "risk_metric", "..."],
    "metrics": [
      { "column": "value_axis_1", "aggregate": "SUM" }
    ],
    "rowLimit": 50000
  },
  "attribution": "Data: GIRI Risk Data, UNEP-GRID Geneva (https://giri.unepgrid.ch)",
  "fetchScript": "scripts/fetch-country-data.js"
}
```

### Per-country provenance (`_provenance` in each country JSON)

Each country file includes `_provenance.exploreUrl`, which opens Superset with the datasource, dimensions, metrics, and country filter pre-filled. The `attribution` field provides a ready-made credit line for UIs, tooltips, or overlays.

## Usage

### Fetching Data

From the repository root:

```bash
# Fetch all countries using default provider (Superset)
npm run fetch-country-data

# Fetch specific countries
npm run fetch-country-data -- --countries AFG,ALB,DZA

# Use mock provider for testing
npm run fetch-country-data -- --provider mock

# Dry run (preview without writing files)
npm run fetch-country-data -- --dry-run --verbose

# Specify custom version
npm run fetch-country-data -- --version v2.0.0
```

### Configuration

Set environment variables for Superset authentication:

```bash
export SUPERSET_BASE_URL="https://dash.giri.unepgrid.ch"
export SUPERSET_PERMALINK_ID="VoAYOvNKR4P"
export SUPERSET_AUTH_TOKEN="your-auth-token-here"
```

Or create a `.env` file in the repository root:

```
SUPERSET_BASE_URL=https://dash.giri.unepgrid.ch
SUPERSET_PERMALINK_ID=VoAYOvNKR4P
SUPERSET_AUTH_TOKEN=your-auth-token-here
```

## Public URLs

Once published, data is available via the asset library:

- **Specific version**: `https://assets.undrr.org/projects/pw-country-pages/data/v1.0.0/countries/AFG.json`
- **Latest version**: `https://assets.undrr.org/projects/pw-country-pages/data/latest/countries/AFG.json`
- **Index**: `https://assets.undrr.org/projects/pw-country-pages/data/latest/index.json`

## Adding New Data Providers

To add a new data provider, extend the `DataProvider` class in `scripts/fetch-country-data.js`. Every provider must implement three methods:

```javascript
class CustomProvider extends DataProvider {
  async fetchCountryData(countryCode) {
    // Fetch data for a specific country
    return {
      countryCode,
      provider: 'custom',
      fetchedAt: new Date().toISOString(),
      _provenance: { /* per-country provenance -- see below */ },
      data: { /* your data structure */ }
    };
  }

  async fetchAllCountries() {
    // Fetch and group data by ISO3 code
    const countries = {};
    // ... populate countries, each with _provenance
    return countries;
  }

  getProvenance() {
    // Return source details for metadata.json and provenance.json.
    // At minimum, include source.provider, attribution, and fetchScript.
    return {
      source: {
        provider: 'custom',
        baseUrl: 'https://example.com',
        exploreUrl: 'https://example.com/your-dataset',
      },
      attribution: 'Data: Your Organisation Name (https://example.com)',
      fetchScript: 'scripts/fetch-country-data.js',
    };
  }
}
```

Then register it in the provider factory:

```javascript
function createProvider(name, config = {}) {
  switch (name.toLowerCase()) {
    case 'custom':
      return new CustomProvider(config);
    // ...
  }
}
```

The `getProvenance()` convention keeps every dataset self-documenting regardless of source.

## Updating the data

Data fetching is **not CI-automated**: run locally, review output, then commit and push the JSON files.

```bash
# 1. Fetch (writes to projects/pw-country-pages/data/<version>/)
npm run fetch-country-data

# 2. Review the diff
git diff --stat projects/pw-country-pages/data/

# 3. Commit and push
git add projects/pw-country-pages/data/
git commit -m "Update country data to <version>"
git push
```

CI/CD deploys committed files only; it does not run the fetch script.

## Working with Apache Superset

Data currently comes from [UNEP-GRID's Superset instance](https://dash.giri.unepgrid.ch). Key notes:

**Key concepts**

- **Datasource**: A database table or view exposed through Superset (e.g. `risk.risk_drought`). Identified by a numeric ID and type (`table`).
- **Permalink**: A short ID (e.g. `VoAYOvNKR4P`) that resolves to a saved explore configuration via `/api/v1/explore/permalink/{id}`. The script uses this to discover the datasource.
- **Chart Data API**: `POST /api/v1/chart/data` is the main data endpoint. You send a payload with the datasource, columns, metrics (with aggregation), and filters; it returns rows.
- **Explore UI**: Superset's interactive chart builder at `/explore/`. You can pre-fill it via `/superset/explore/?form_data={JSON}` with datasource, dimensions, metrics, and filters URL-encoded as JSON.

**How the fetch script uses Superset**

1. Resolve permalink for datasource ID/type
2. POST to Chart Data API with columns, metrics, and optional country filter
3. Group rows by ISO3 and write per-country JSON files
4. Write a pre-filled Superset `exploreUrl` to each file's `_provenance`

**Useful URLs**

- Superset home: https://dash.giri.unepgrid.ch
- Datasets list: https://dash.giri.unepgrid.ch/tablemodelview/list/
- Charts list: https://dash.giri.unepgrid.ch/chart/list/
- API docs (Swagger): https://dash.giri.unepgrid.ch/swagger/v1

**Authentication**: The Superset instance requires a bearer token for API access. Set `SUPERSET_AUTH_TOKEN` as an environment variable. The explore UI links work without authentication for public datasets.

## Versioning Strategy

- **Patch version** (v1.0.X): Data updates, no structure changes
- **Minor version** (v1.X.0): New fields added, backward compatible
- **Major version** (vX.0.0): Breaking changes in data structure

## Caching

- **Production**: 2-week CDN cache (same as other static assets)
- **Testing**: 5-minute CDN cache
- Use versioned URLs for long-term caching
- Use `latest/` for always-current data (shorter cache)

## Notes

- All versions remain accessible indefinitely (never remove old versions)
- ISO3 country codes are used consistently (e.g., AFG, ALB, DZA)
- Data files are formatted with 2-space indentation for readability
- The `latest` symlink/copy is updated automatically after each fetch