DS2 MCL Module: Service Portal

Services

EDC control planes

A provider and a consumer connector on Eclipse EDC. They publish assets, answer catalog requests over the Dataspace Protocol, negotiate contracts and start, suspend or terminate every transfer.

Checking…

Provider /edc-provider/api/management
Consumer /edc-consumer/api/management
Data planes …/api/management/v4/dataplanes

Contract governance

Stream Data Transfer

Topic-to-topic forwarding of a continuous event stream by a Data Plane Signaling data plane, which pumps records only while the agreement permits it and stops when the consumer suspends.

Checking…

Data plane /stream-dataplane/dataflows
Source /stream-provider/stats
Sink /stream-consumer/stats
Demo Scenario 1: governed stream

Real-time · Kafka-PUSH

Vast Data Transfer

Large files under a contract: partitioned, pushed in a bounded parallel window, verified part by part, written at their byte offset, and resumed from whatever survived a failure.

Checking…

Provider /vast-provider/transfers
Sources /vast-provider/source/files
Consumer /vast-consumer/ingest/{id}/state
Demo Scenario 2: governed large file

EDC-governed large files

Intelligent Data Placement

A placement engine decides, per transfer, whether an asset is worth caching and where. A regional cache then serves repeat transfers in place of the provider, under the same agreement.

Checking…

Engine /idp-engine/placement/v1/nodes
Controller /idp-controller/placement/v1/regions
Caches /idp-cache-regional/cache/v1/capacity
Demo Scenario 3: data placement

Data caching

Discovery Catalogue

Browses the datasets the DS2 catalogue advertises, so a consumer can find what to negotiate for before it opens a contract negotiation.

Checking…

Health /discovery/health
Catalogue /discovery/catalog
Dataset /discovery/dataset/{id}

Finding data

Blockchain logs proxy

Turns MCL events into DRM component logs on the DS2 ledger. The data planes and the control-plane audit bridge write through it; a failing ledger never stops a transfer.

Checking…

Create /blockchain-logs/create-log
Read /blockchain-logs/read-log/component/{id}

Audit

Blockchain audit logs (DRM)

The proxy turns MCL logs into DRM createComponentLog calls with componentLog.type: "MCL". payload.componentName names the transfer path (vast data transfer, stream data transfer, push/pull transfer), alongside message and timestamp. Use a different 0x… id for every log, or omit id so the proxy generates one.

Via Nginx Public: /blockchain-logs/ · Docker: http://blockchain-logs:3000

Example: POST to proxy

{
  "logType": "component",
  "id": "0xa1b2c3d4",
  "logData": {
    "componentName": "vast data transfer",
    "message": "[vast] transfer 3f2a… completed, sha256 verified",
    "timestamp": "1760290000"
  },
  "userID": "user2Org1TEST",
  "organization": "Org1MSP"
}

Successful create returns logId (the same id sent to DRM). Read with GET …/read-log/component/{id}. Each data plane also keeps a local mirror of what it sent, readable on the demo pages even while the ledger is down.

Proxy /blockchain-logs/create-log (POST), /blockchain-logs/read-log/component/{id} (GET)
DRM Manager https://drm-manager.ds2.atc.gr/ledger/createComponentLog

Audit & traceability

Choosing a transfer path

All three paths share one rule: a transfer runs under a contract. They differ in what is being moved. Intelligent data placement is not a fourth path; it sits behind the large-file path and changes where the bytes come from, not who may receive them.

Path	Use it for	How it moves	Try it
Stream	continuous events: IoT, telemetry	topic to topic, suspend without loss	Scenario 1
Vast	files of gigabytes, up to 100 GB	verified parts at byte offsets, resumable	Scenario 2
Vast + placement	the same large file for many consumers	repeat transfers served from a regional cache	Scenario 3
Push/Pull	small or medium payloads, on demand	request/response or scheduled push	runs separately: `push_pull/run_demo.sh`

Testing

The integration suites drive the same Management API calls the demo pages make. Run them on the host where the stack runs, from the root of the mcl_module repository.

Integration suites (bash)

Each suite pre-flights on the data planes' registration-aware /ready first.

Run the suites

docker compose up -d
tests/stream_data_extension/run_all.sh
tests/vast_data_transfer/run_all.sh
tests/intelligent_data_placement/run_all.sh
# The demo-page checks find the proxy themselves; to point them elsewhere:
UI_BASE=https://this-site tests/vast_data_transfer/test_vast_demo_ui.sh

Unit tests need no stack: node --test test/ in each data plane, cache and edc/common/dps package. See tests/README.md for prerequisites.

Postman

Import ds2_data_sharing.postman_collection.json and set the collection variables to this deployment:

Collection variables

vastProviderBaseUrl    = https://this-site/vast-provider
vastConsumerBaseUrl    = https://this-site/vast-consumer
providerManagementUrl  = https://this-site/edc-provider/api/management
consumerManagementUrl  = https://this-site/edc-consumer/api/management
providerDspUrl         = http://edc-provider:8082/api/protocol/2025-1

providerDspUrl stays internal: it is the address the consumer's connector uses to reach the provider's, not one your machine calls. Run the requests in order; each saves the id the next one needs.

See README.md and tests/README.md in the repo for full details.