1. Overview

The WIS2 Downloader exposes a REST API for managing subscriptions and monitoring the system.

Replace localhost:5002 with your server’s address if deployed remotely. The base URL is configured via WIS2DOWNLOADER_SUBSCRIPTION_MANAGER_URL in .env.

2. Authentication

The API does not require authentication. In production, place it behind a reverse proxy with authentication (see the Access Control section of the Admin Guide).

3. Subscriptions API

Each subscription is identified by a UUID and binds together an MQTT topic, a save path, an optional filter, optional credentials, and a queue. Multiple subscriptions can share the same topic — the MQTT connection is opened once and messages are routed to all matching subscriptions.

3.1. List Subscriptions

Returns all active subscriptions grouped by topic.

GET /subscriptions

3.1.1. Response

Status Description

200

Success — map of topic → map of subscription UUID → subscription detail

503

Redis unavailable

3.1.2. Example

curl http://localhost:5002/subscriptions
{
  "cache/a/wis2/+/data/core/weather/surface-based-observations/#": {
    "550e8400-e29b-41d4-a716-446655440000": {
      "save_path": "surface-obs",
      "filter": {}
    },
    "661f9511-f30c-52e5-b827-557766551111": {
      "save_path": "surface-obs-bufr",
      "filter": {
        "name": "bufr-only",
        "rules": [
          { "id": "accept-bufr", "order": 1, "match": { "media_type": { "equals": "application/bufr" } }, "action": "accept" }
        ]
      }
    }
  }
}

3.2. Create Subscription

Creates a new subscription. Assigns a UUID and returns the full subscription record.

If this is the first subscription for the given topic, an MQTT connection is opened. Additional subscriptions for the same topic reuse the existing connection.

POST /subscriptions
Content-Type: application/json

3.2.1. Request Body

Field Type Required Description

topic

string

Yes

WIS2 MQTT topic pattern. Supports + (single-level) and # (multi-level) wildcards.

target

string

No

Subdirectory under the data root where files are saved.

filter

object

No

Rule-based filter. See Filter Schema.

credentials

object

No

HTTP credentials for access-controlled datasets. See Credentials Schema.

queue

string

No

Download queue: high_priority, small_files (default), or large_files.

3.2.2. Response

Status Description

201

Created — returns the full subscription record

400

Bad request — missing or invalid fields

503

Service unavailable — Redis connection failed

3.2.3. Response Headers

Header Description

Location

URL to retrieve the created subscription (/subscriptions/{id})

3.2.4. Examples

# Simple subscription
curl -X POST http://localhost:5002/subscriptions \
  -H "Content-Type: application/json" \
  -d '{
    "topic": "cache/a/wis2/+/data/core/weather/surface-based-observations/#",
    "target": "surface-obs"
  }'
# Subscription with a filter
curl -X POST http://localhost:5002/subscriptions \
  -H "Content-Type: application/json" \
  -d '{
    "topic": "cache/a/wis2/+/data/core/weather/surface-based-observations/#",
    "target": "surface-obs-bufr",
    "filter": {
      "name": "bufr-only",
      "rules": [
        { "id": "accept-bufr", "order": 1, "match": { "media_type": { "equals": "application/bufr" } }, "action": "accept" },
        { "id": "default-reject", "order": 999, "match": { "always": true }, "action": "reject" }
      ]
    }
  }'
# Subscription to an access-controlled dataset using basic auth
curl -X POST http://localhost:5002/subscriptions \
  -H "Content-Type: application/json" \
  -d '{
    "topic": "cache/a/wis2/+/data/recommended/weather/surface-based-observations/#",
    "target": "restricted-obs",
    "credentials": { "type": "basic", "username": "myuser", "password": "s3cr3t" }
  }'
# Route large downloads to the large-files queue
curl -X POST http://localhost:5002/subscriptions \
  -H "Content-Type: application/json" \
  -d '{
    "topic": "cache/a/wis2/fr-meteofrance/data/core/weather/space-based-observations/noaa-21/cris",
    "target": "space-obs",
    "queue": "large_files"
  }'
{
  "id": "550e8400-e29b-41d4-a716-446655440000",
  "topic": "cache/a/wis2/+/data/core/weather/surface-based-observations/#",
  "save_path": "surface-obs",
  "filter": {},
  "credentials": null,
  "queue": "small_files"
}

3.3. Get Subscription

Returns the full record for a specific subscription.

GET /subscriptions/{id}

3.3.1. Path Parameters

Parameter Type Description

id

string (UUID)

Subscription UUID returned when the subscription was created

3.3.2. Response

Status Description

200

Success

404

Subscription not found

3.3.3. Example

curl http://localhost:5002/subscriptions/550e8400-e29b-41d4-a716-446655440000
{
  "id": "550e8400-e29b-41d4-a716-446655440000",
  "topic": "cache/a/wis2/+/data/core/weather/surface-based-observations/#",
  "save_path": "surface-obs",
  "filter": {},
  "credentials": null,
  "queue": "small_files"
}
Credential secrets (password, token) are never returned. Only type and username (for basic auth) are included in responses.

3.4. Update Subscription

Updates the save path, filter, credentials, or queue of an existing subscription. The topic cannot be changed — delete and recreate to change the topic.

PUT /subscriptions/{id}
Content-Type: application/json

3.4.1. Request Body

All fields are optional. Only provided fields are updated.

Field Type Description

target

string

New subdirectory under the data root

filter

object

New rule-based filter (replaces existing filter)

credentials

object

New credentials (replaces existing credentials)

queue

string

New queue: high_priority, small_files, or large_files

3.4.2. Response

Status Description

200

Updated subscription record

400

Invalid field value

404

Subscription not found

503

Redis unavailable

3.4.3. Example

curl -X PUT http://localhost:5002/subscriptions/550e8400-e29b-41d4-a716-446655440000 \
  -H "Content-Type: application/json" \
  -d '{ "target": "surface-obs-v2", "queue": "high_priority" }'

3.5. Delete Subscription

Removes a subscription. If it is the last subscription for its topic, the MQTT connection for that topic is closed.

DELETE /subscriptions/{id}

3.5.1. Response

Status Description

200

Successfully deleted

404

Subscription not found

503

Redis unavailable

3.5.2. Example

curl -X DELETE http://localhost:5002/subscriptions/550e8400-e29b-41d4-a716-446655440000
{ "status": "deleted", "id": "550e8400-e29b-41d4-a716-446655440000" }

4. Monitoring API

4.1. Health Check

GET /health

Always returns 200. Check the status field for actual health.

curl http://localhost:5002/health
{ "status": "healthy" }

4.2. Prometheus Metrics

Returns metrics in Prometheus text format.

GET /metrics

4.2.1. Available Metrics

Metric Type Labels Description

wis2downloader_notifications_total

Counter

status

Total notifications processed by Celery

wis2downloader_downloads_total

Counter

cache, media_type

Successfully downloaded files

wis2downloader_downloads_bytes_total

Counter

cache, media_type

Total bytes downloaded

wis2downloader_skipped_total

Counter

reason

Skipped notifications by reason

wis2downloader_failed_total

Counter

cache, reason

Failed downloads

wis2downloader_celery_queue_length

Gauge

queue_name

Current tasks across all Celery queues

4.2.2. Example

curl http://localhost:5002/metrics

4.3. OpenAPI Specification

GET /openapi

4.4. Swagger UI

Interactive API documentation available at:

GET /
GET /swagger

5. Error Responses

All error responses follow this format:

{ "error": "Error message describing the problem" }

5.1. Common Error Messages

Status Error Cause

400

"No topic provided"

POST /subscriptions without a topic field

400

"credentials.type must be 'basic' or 'bearer'"

Invalid credentials type

400

"queue must be one of: high_priority, large_files, small_files"

Unknown queue name

404

"Subscription '{id}' not found"

UUID does not exist

503

"Failed to connect to Redis: …​"

Redis unavailable

503

"Failed to queue subscription command…​"

Redis pub/sub failed

6. Data Schemas

6.1. Subscription

Full subscription record as stored and returned by the API.

Field Type Description

id

string (UUID)

Unique identifier assigned on creation

topic

string

WIS2 MQTT topic pattern

save_path

string | null

Subdirectory under the data root where files are saved

filter

object

Rule-based filter (empty object {} means no filtering)

credentials

object | null

Credentials used for downloading. Secrets are redacted in responses.

queue

string

Download queue: high_priority, small_files, or large_files

6.2. Filter Schema

Filters use an ordered list of rules. Rules are evaluated in order; the first matching rule’s action is applied. If no rule matches, the file is accepted.

Field Type Description

name

string

Filter name for identification

rules

array

Ordered list of filter rules

6.2.1. Filter Rule Schema

Field Type Description

id

string

Unique rule identifier

order

integer

Execution order — lower values run first

match

object

Condition to match against. See Match Conditions.

action

string

accept, reject, or continue

6.2.2. Match Conditions

Condition Description

media_type

Match on the file’s MIME type

size

Match on file size in bytes

centre_id

Match on the originating centre ID (e.g., de-dwd, fr-meteofrance). The first field is the ISO2C country code, the second field is a descriptive name for the centre.

data_id

Match on the WIS2 data ID

href

Match on the download URL

always: true

Unconditionally match (use as a catch-all default rule)

String operators: equals, not_equals, in, not_in, pattern (glob), regex

Size/numeric operators: gt_bytes, lt_bytes, gte_bytes, lte_bytes, between_bytes

Logical combinators: all (AND), any (OR), not

6.2.3. Filter Example

Accept only BUFR files smaller than 10 MB; reject everything else:

{
  "name": "bufr-small",
  "rules": [
    {
      "id": "accept-small-bufr",
      "order": 1,
      "match": {
        "all": [
          { "media_type": { "equals": "application/bufr" } },
          { "size": { "lt_bytes": 10485760 } }
        ]
      },
      "action": "accept"
    },
    {
      "id": "default-reject",
      "order": 999,
      "match": { "always": true },
      "action": "reject"
    }
  ]
}

6.3. Credentials Schema

Used to authenticate HTTP downloads from access-controlled datasets.

password and token are write-only — they are accepted in requests but never returned in responses.
Field Type Description

type

string

basic or bearer

username

string

Required when type is basic

password

string (write-only)

Required when type is basic

token

string (write-only)

Required when type is bearer

7. Topic Pattern Reference

7.1. WIS2 Topic Structure

{origin|cache}/a/wis2/{centre-id}/data/{data-policy}/{earth-system-domain}/{category}/...

  • origin — Original data from the data provider

  • cache — Cached copy from a Global Cache (recommended for reliability)

  • {centre-id} — Centre identifier. The first field is the ISO2C country code and the second is a descriptive name for the centre (e.g., de-dwd for Deutscher Wetterdienst, fr-meteofrance for Météo-France)

  • {data-policy} — WMO Unified Data Policy: core (freely available) or recommended (may require credentials)

  • {earth-system-domain} — Earth system domain from the WMO Unified Data Policy

7.2. Wildcards

Pattern Symbol Description

Single-level

+

Matches exactly one level

Multi-level

#

Matches zero or more levels (must be at end)

7.3. Examples

Pattern Matches

cache/a/wis2/de-dwd/data/#

All data from Deutscher Wetterdienst

cache/a/wis2/+/data/core/weather/surface-based-observations/#

Surface observations from all centres

cache/a/wis2/+/data/core/#

All core data from any centre

cache/a/wis2//data//weather/#

All weather data (any policy) from any centre