DS2: Data Space | Data Share Digiotouch

DS2 · MCL module demo · Scenario 2

Vast Data Transfer

Sending multi-gigabyte data (stored in multiple cloud storage) between dataspace participants as a single stream is fragile: one dropped connection or restarted service forces the whole transfer to start again, and integrity can only be checked after the last byte has arrived. Vast Data Transfer addresses this with a custom data plane, built for the MCL module rather than taken from the Eclipse Dataspace Connector (EDC), which the EDC control planes start only under a negotiated contract agreement. pX's side of the data plane splits the file into fixed-size parts and pushes them over a bounded number of parallel connections, compressing a part only when that reduces its size. pY's side verifies the SHA-256 checksum of each part on arrival, writes it directly at its byte offset and records it in a durable journal, so no reassembly step is needed and an interrupted transfer resumes from the parts already verified. Once the last part arrives, the checksum of the whole file is compared with the source. The module is designed for files of up to 100 GB.

Provider

pX
Holds the data in multi-cloud infrastructure
→

DS2 MCL module

Vast Data (Push)
Custom data plane; partitioned, parallel, verified part by part
→

Consumer

pY
Receives a verified copy

Running the demo. Step 1 lists the datasets pX holds: tick the ones to share with pY. In Step 2, start the transfer. The page publishes the offer on pX, negotiates one contract for pY, and then transfers the selected datasets one after another under that single agreement. Step 3 shows each dataset's progress. Use Step 4 to suspend, resume or terminate the current transfer. Step 5 shows which parts of the current transfer are already verified and stored at pY, so an interrupted transfer resumes from them.

1

Data Provider (Multi-Cloud Storage)

Public cloud storage

Amazon S3

Amazon Web Services · object storage · S3 API

Available

Azure Blob Storage

Microsoft Azure · object storage

Available

Google Cloud Storage

Google Cloud · object storage · S3-compatible interoperability API

Available

Connected storage

2

Negotiate a contract and start the transfer

Selected datasets none yet: tick datasets in Step 1

Waiting for a transfer…

3

Progress

Current transfer

Parts
—
verified at the consumer
In flight
—
of the parallel window
Throughput
—
end to end
ETA
—
at the current rate

—

4

Suspend, resume, terminate

These drive the consumer's EDC Management API, the same calls a real participant makes. Suspend stops the provider refilling its window; parts already in flight are allowed to land. Terminate ends the current transfer and cancels the datasets still queued behind it.

no transfer
5

Resume state: what survives a failure

Every green cell below is a part whose bytes are on disk and fsynced, and whose checksum has been verified.

verified & durable in flight not yet sent kept across a restart