Blog metadata (draft, for the CMS)
Blog image
assets/four_worlds_poster.jpg (alt: assets/hero_poster.jpg)
Publish date & time
TBD
Author(s)
Alicia Chua, Alexandr Plashchinsky, Nickhil Gupta, Abhishek Sriram, Abhishek Gupta
Blog type
Product
Eng reviewer
TBD
Target keywords
Ray, Anyscale, Ray Serve, Ray Data, multi GPU, video generation, world foundation models, NVIDIA Cosmos, Cosmos Transfer 2.5, synthetic data, off-road autonomy, data augmentation, batch inference, model multiplexing
Related resources
https://docs.ray.io/en/latest/serve/model-multiplexing.html
https://docs.ray.io/en/latest/data/data.html
https://docs.ray.io/en/latest/ray-core/actors.html
SEO description
Repaint real off-road video into new weather with NVIDIA Cosmos Transfer 2.5 on Ray: 288 clips on 8 A100s, plus a live Ray Serve app. (133 characters)
Blog URL
/paint-the-trail-cosmos-transfer-ray
Ray Data · Ray Core · Ray Serve · NVIDIA Cosmos

Painting New Weather onto Real Off-Road Video with Ray and NVIDIA Cosmos Transfer 2.5

By Alicia Chua, Alexandr Plashchinsky, Nickhil Gupta, Abhishek Sriram and Abhishek Gupta

Night rain, blizzards and wildfire smoke are rare in off-road driving logs, and those are the conditions where a perception model most needs examples. We took real ATV footage from TartanDrive 2.0 and used NVIDIA Cosmos Transfer 2.5 to repaint each clip in a new condition while keeping the trail, the bumps and the camera path of the original drive. On 8 A100 GPUs, Ray produced 288 weather variants in 51.5 minutes, and every request succeeded on its first attempt.

The same 5.8 seconds of real driving, repainted six times. Each wipe switches to a different Cosmos Transfer 2.5 output, all generated from one camera clip and its edge map.

We built the workflow from four Ray pieces on one cluster: Ray Data prepares clips and scores outputs, persistent Ray actors keep the 2B-parameter model loaded on each GPU, a small scheduler built on ray.wait handles retries and resume, and Ray Serve turns the same actors into an app where anyone can pick a drive, type a condition and watch 8 GPUs paint at once. The template includes a replay of stored results that runs without GPUs; generating new clips requires GPU resources.

288 / 288clips succeeded on the first attempt, 0 out-of-memory errors
51.5 minfor the full run on 8 A100s (6.45 GPU-hours of work)
6.98xgeneration speedup on 8 GPUs vs 1 (87% efficiency)
100.9 sfor 8 drives painted live from one click in Ray Serve

Try it

Pick a drive and a condition. The player below replays results we generated with the batch pipeline, at the model's native 1024×544, with the original camera on the left of the line and the repaint on the right. Switch to "All 9 drives" to see one prompt applied across every scene.

Interactive replay: stored outputs, no GPU neededopen full screen

The live app, start to finish

The replay mirrors a live Ray Serve application. In the recording below, a first-time visitor picks a drive, chooses golden hour and sends it to all 8 GPUs. The page shows each GPU's progress while Cosmos runs, then lays out the 8 repaints with a comparison slider.

Screen recording of the live app on Ray Serve with safety guardrails on. The GPU wait is fast-forwarded 12x. All 8 clips finished 100.9 seconds after the click.

Motivation

Ground robots that leave the road meet conditions their training logs barely cover. Capturing a night rainstorm on a specific trail means waiting for the weather, sending a crew out in it and labeling the result again. Simulators offer control, but their terrain rarely looks like a real hillside of wet grass and loose gravel.

World foundation models offer a third route. Cosmos Transfer 2.5 takes a real video, a control signal derived from it and a text prompt, and generates a new video that follows the control. We use the edge map of every frame as the control. Because the edges come from the real recording, the generated clip keeps the trail's layout and the vehicle's motion, while the prompt sets the light, weather, season and surface.

Four conditions generated from the same real clip and the same edge map: golden hour, night rain, autumn and blizzard.

How a clip is repainted

Dataset: TartanDrive 2.0

ItemValue
SourceTartanDrive 2.0, CC BY 4.0, ATV front camera (image_left_color)
Drives9 recorded runs
Clips27 prepared (3 per drive); 24 used in the main run, 9 showcase scenes for the extra conditions
Download13.39 GB in 82.75 s
Clip format93 frames, 1024×544, 16 fps

Results

Scaling: per-request time stays flat

We ran the same 24 requests on 1, 2, 4 and 8 actors, each with one A100, in fresh namespaces with no resume. The median request took 79.79 seconds on one actor and 80.27 seconds with eight running side by side: actors do not slow each other down.

ActorsWall timeSpeedup, generationSpeedup, end to endMedian per request
12067.75 s1.00x1.00x79.79 s
21107.90 s1.96x1.87x80.19 s
4624.94 s3.75x3.31x80.26 s
8398.39 s6.98x5.19x80.27 s

End-to-end speedup includes a fixed cold start of about 106 seconds per actor (loading the model), spread over only 3 requests per actor in the 8-actor case. On longer runs that cost fades: the main run below spent 88 seconds before its first dispatch and 51.5 minutes generating.

Speedup and parallel efficiency for 1, 2, 4 and 8 actors
Speedup over one actor (left) and parallel efficiency (right). The generation window excludes the model cold start.
Timeline of 8 actors each loading the model then generating 3 requests back to back
Every actor loads the model once, then generates three requests back to back with no idle gaps.

The main run

The main run crossed 24 clips with 4 conditions (fog, rain, night, snow) and 3 seeds: 288 requests on 8 A100s. All 288 succeeded on the first attempt in 3089.64 seconds, with no out-of-memory errors, no infrastructure failures and no replacement actors. Run one after another, the same work would take 23,219 seconds of GPU time.

Where the time goes

The 4 denoising steps account for 79% of each request. Everything around them is small, so the throughput lever is the number of GPUs working in parallel.

Stage (median of 288)SecondsShare
Denoising, 4 steps63.5779.1%
Video guardrail5.036.3%
Writing output and control videos4.856.0%
Decode and postprocessing4.035.0%
Preprocessing1.181.5%
Text guardrail0.640.8%
Total per request80.32100%

One prompt, many drives

Because the trail structure comes from each recording, one prompt produces a different but consistent scene on every drive. Night rain scored highest of the ten extra conditions in our automated review.

The night rain prompt applied to all nine showcase drives. Headlight falloff, wet ground and rain streaks follow each trail's own layout.
Autumn on a rocky trail under trees.
Blue hour on a wet gravel road.

What we can measure automatically

We scored every output with a Ray Data pass on CPU. Edge F1 compares the output's edges with the source clip's edges within 2 pixels; it measures how much structure the repaint keeps, not how good it looks. Fog scores lowest, which is expected because fog hides distant edges.

Condition (72 outputs each)Edge F1 vs sourceChange in brightness
Snow0.820+17.3
Rain0.779−33.9
Night0.640−82.4
Fog0.529+7.8

Two lessons came out of the metrics. First, HSV saturation overstates color on dark frames because it divides by brightness: night outputs read 170.7 against 41.8 for the source. A brightness-independent measure, CIELAB chroma, shows a real but smaller rise from 8.47 to 15.44. Second, an automated vision-language judge (Cosmos-Reason1-7B, blind to the requested condition) is weak evidence: it labeled 64 of 71 rain clips "overcast". We used its scores only to rank showcase candidates. Human review of the outputs is still to come.

Where it breaks

The lowest-scoring output out of 371 judged: wildfire smoke on a winding gravel path. The sun disc dissolves and smoke puffs appear and vanish between frames.

Our automatic checks flagged 6 of the 288 main-run outputs, all for frozen frames and all from one source clip where the vehicle barely moves. That clip already repeats frames, because the camera's 10 Hz recording is stretched to 16 fps, so we treat the flags as pointers for review rather than confirmed defects.

System design on Ray

Preparing clips, holding a large model on every GPU, keeping 8 GPUs busy, scoring hundreds of videos and serving a live app are five different scaling problems. Each maps to a Ray abstraction, and all of them share one cluster of 8 nodes with one A100 each.

Ray Data: prepare every clip once

Clip preparation decodes the raw camera frames, checks timestamps, resamples to 16 fps, encodes the model input, verifies it by decoding it back and computes the edge control. A Ray Data pipeline fans the work out one clip per block across the GPU nodes' CPUs, so the GPUs stay free for generation:

Ray Data · Clip preparationPython · excerpt
ds = (ray.data.from_items(runs, override_num_blocks=len(runs))
      .flat_map(plan_clips, num_cpus=1, resources=WORKER_ONLY)       # choose clip windows per drive
      .repartition(target_num_rows_per_block=1)                      # one clip per block
      .map(prepare_clip, num_cpus=2, resources=WORKER_ONLY,          # decode, resample, edges, verify
           compute=ray.data.TaskPoolStrategy(size=concurrency)))
ds.write_parquet(catalog_dir)                                         # the prepared clip catalog
The Ray advantage: preparing 9 clips from scratch took 24.06 seconds of wall time for 129.49 seconds of per-clip work. Each clip writes into a directory keyed by a hash of its settings, so a rerun over all 27 clips with the same settings finished in 7.28 seconds.

Ray Core: keep the model loaded in its own environment

Cosmos Transfer 2.5 pins its own PyTorch and CUDA stack, which differs from the driver's environment. Each generation worker is a Ray actor that starts one long-lived child process under the Cosmos interpreter and talks to it over a pipe. The model loads once per actor, in 52 to 57 seconds, and then serves every request sent to that GPU:

Ray Core · Persistent GPU actorPython · excerpt
class CosmosActor:                               # created with ray.remote(CosmosActor), one GPU each
    def _start_child(self):
        rfd, wfd = os.pipe()                      # replies come back on a dedicated pipe
        self.proc = subprocess.Popen(
            [self.s["cosmos_python"], "-m", "offroad_ray.inference.subprocess_worker"],
            stdin=subprocess.PIPE, env=self._child_env(), pass_fds=(wfd,))

    def generate(self, req: dict) -> dict:     # model stays loaded between calls
        ...
The Ray advantage: the actor owns the GPU, the child process and its logs. If the child dies, the actor reports it and the scheduler replaces the actor on a healthy node, while the conflicting Cosmos dependencies never touch the driver.

A scheduler on ray.wait: one request in flight per GPU

Generation requests vary little in cost, so the scheduler keeps exactly one request in flight per actor and hands out the next one as soon as any finishes. It retries infrastructure failures on a replacement actor, never retries a guardrail block, validates every output and writes a record per request, so an interrupted run resumes from what is already done:

Ray Core · Dispatch loopPython · simplified
while pending or in_flight:
    done, _ = ray.wait(list(in_flight), num_returns=1, timeout=5.0)
    for ref in done:
        i, req, t0 = in_flight.pop(ref)
        record(req, ray.get(ref))                 # validate with ffprobe, write the record
        free.append(i)
    while free and pending:                    # refill idle GPUs immediately
        i, req = free.popleft(), pending.popleft()
        in_flight[actors[i].generate.remote(req)] = (i, req, time.time())
The Ray advantage: because requests are dispatched one at a time from a shared queue, a GPU never waits on another. In the main run each of the 8 actors served exactly 36 requests and was busy for 2905 to 2957 of the 3090 seconds.

Ray Data: score 288 videos on CPU

The quality pass reads every output, computes edge agreement, brightness, contrast, color and temporal metrics, and aggregates them per condition. It ran on all 288 outputs in 60.12 seconds and also builds side-by-side review images for a static review page:

Ray Data · Quality metricsPython · simplified
ds = ray.data.from_items(rows, override_num_blocks=len(rows))
metrics = ds.map(compute_output_metrics, num_cpus=2, resources=WORKER_ONLY,
                 compute=ray.data.TaskPoolStrategy(size=32))
metrics.write_parquet(metrics_dir)
summary = ray.data.read_parquet(metrics_dir).groupby("condition").aggregate(
    Count(), Mean("edge_f1"), Std("edge_f1"), Mean("delta_luma"), ...).to_pandas()

Ray Serve: the live app

The live app reuses the same actor class inside a Serve deployment. Each GPU replica accepts one request at a time. A CPU front end built on FastAPI serves the page, queues jobs and lets the browser poll for progress, so a request that takes 80 seconds never holds a connection open. Scene IDs act as multiplexed model IDs, which sends repeat requests for a drive to a replica that has already encoded it:

Ray Serve · GPU tier and scene affinityPython · excerpt
@serve.deployment(max_ongoing_requests=1, ray_actor_options={"num_gpus": 1, "num_cpus": 4})
class Painter:
    def __init__(self, settings):
        self.engine = CosmosActor(idx, settings)      # same engine as the batch pipeline

    @serve.multiplexed(max_num_models_per_replica=12)
    async def _scene(self, scene_id): return scene_id

    async def paint(self, req):
        await self._scene(serve.get_multiplexed_model_id())
        return await asyncio.to_thread(self.engine.generate, req)

# In the FastAPI front end: route each job to a replica that already holds its scene
self.painter.options(multiplexed_model_id=job["scene_id"]).paint.remote(req)
The Ray advantage: Serve's multiplexing doubles as a cache-affinity router. The two video encodes that depend only on the drive are cached on the GPU and reused bit for bit, and identical requests replay instantly from shared storage.

With guardrails on, 8 replicas were ready 121 seconds after deployment. In the recorded session, 8 drives finished 100.9 seconds after the click, with each clip taking 97.9 to 100.5 seconds. Those were the first requests on each replica, and the guardrail models were still warming up: the video check took 13.56 seconds against a median of 5.03 seconds in the batch run.

What did not make a single clip faster

We also tried to shorten the wait for one clip. Splitting a clip across GPUs with context parallelism was slower than a single GPU in both 2-node and 4-node configurations, because each node in this cluster has one GPU and the split adds network traffic to every step. Generating at 480p was about 3 times faster, but this distilled model is trained for 720p and the output turned to blur. torch.compile shaved a few seconds per clip, after warm-up runs of up to three minutes. Depth control, the other way to condition the model, needs the non-distilled model with 35 steps: 867.5 seconds per clip, 10.8 times the edge model. On this hardware the better strategy is more clips in parallel, which is what Ray makes easy.

Limitations

Cost

The batch work in this post, from the first smoke test through the ten extra conditions, used 3.32 hours of cluster time on 8 GPU nodes: 26.55 node-hours held and 17.40 node-hours of active generation. The main run alone held 6.87 node-hours.

Key takeaways

A world foundation model can turn a few hours of real off-road driving into the same trails at night, in rain, in smoke or in autumn, while the geometry of every frame stays true to the recording. The work is dominated by a fixed 4-step sampler on each clip, so throughput comes from running many clips at once: per-request time stayed flat from 1 to 8 GPUs, and 288 clips finished in under an hour without a failed request.

Ray supplies the execution layer around the model: Ray Data for preparation and scoring, persistent actors that isolate the model's dependencies, a dispatch loop that keeps every GPU busy and resumes cleanly, and Ray Serve to put the same engine behind an interactive app.

Get started

Open the template on Anyscale (DRAFT: replace with the template link before publishing). It includes the batch pipeline, the quality pass, the Ray Serve app and the replay page shown above, which runs without GPUs. Enable GPU execution when you are ready to generate new clips.

Acknowledgements. Driving footage from TartanDrive 2.0 (CC BY 4.0); we thank its authors for making it openly available. Generation uses NVIDIA Cosmos Transfer 2.5 and the automated review uses Cosmos-Reason1-7B.

Get involved with Ray

To connect with the Ray community, join the Ray Slack or ask questions on the Discuss forum. To run this workload on managed Ray, get started with Anyscale. To learn more about Ray, see ray.io and docs.ray.io.