DS2: Data Space | Data Share Digiotouch

DS2 · MCL module demo · Scenario 3

Data Caching for Vast Data Transfer

Vast Data Transfer (Scenario 2) moves very large datasets between participants under a contract. When the same dataset is wanted again, by several consumers in one region or by the same consumer later, sending it from the provider each time spends the provider's uplink on bytes it has already sent, and every consumer waits on the full route back to the source. Data caching removes that repetition. The placement engine weighs each dataset's policy, its size and how often it is asked for, and decides whether to cache it and where. The first transfer is served by the provider, and a copy of each part is placed on a cache in the consumer's region as it goes. Every later transfer of that dataset is served from the regional cache: the consumer gets it from close by, the provider's uplink carries almost nothing, and the contract still governs every byte.

Provider

pX
Holds the original and pays the uplink for every miss
→

DS2 MCL module

Intelligent Data Placement
The engine decides, the controller places, the caches serve
→

Consumer

pY
Asks from a regional cache
Provider checking… Engine checking… Controller checking…

Running the demo. Step 1 shows the cache nodes and what each holds. In Step 2, select the data to transfer, the cache policy and the consumer's region, and run the transfer. With the cache policy on, the first run is a miss, served by pX while a copy of each part is placed on that region's cache. When Step 3 reports the population job complete, run the same dataset again from the same region: this run is a hit, served from the cache. Step 3 shows where each transfer's bytes came from and why the engine decided as it did, and Step 4 sets the miss and the hit side by side.

1

The cache nodes, and what they hold

A provider-tier cache sits next to the provider and holds parts already prepared for sending. Regional caches sit near the consumers, one per region; the engine resolves the consumer's region to its cache by name.

cache nodeschecking…
2

Run a transfer the engine may cache

The provider publishes the data as an asset with the cache policy chosen below, and the consumer negotiates and asks for it from the chosen region. With the cache policy on, run it once for the miss, wait until the population job below reports complete, then run the same data again for the hit. With it off, every transfer is served by the provider. Data is uploaded in Scenario 2, Step 1.

pX is using the Vast Data Transfer push method to share data with pY.

—

3

Placement: where these bytes came from

This transfer
—
hit, miss or partial
Parts from cache
—
of the parts outstanding
Served by
—
tier / node
Provider uplink
bytes the provider sent

No population job yet.

The engine's recommendation, and why

—

  • no recommendation yet
4

Miss against hit

TransferOutcomeWall clockProvider uplinkFrom cache

Run one transfer to populate the cache, then another of the same file.