Blog metadata (draft, for the CMS)
Blog image
assets/four_worlds_poster.jpg (alt: assets/hero_poster.jpg)
Publish date & time
TBD
Author(s)
Alicia Chua, Alexandr Plashchinsky, Nickhil Gupta, Abhishek Sriram, Abhishek Monangi, Omar Shorbaji
Blog type
Product
Eng reviewer
TBD
Target keywords
Ray, Anyscale, Ray Serve, Ray Data, multi GPU, video generation, world foundation models, NVIDIA Cosmos, Cosmos Transfer 2.5, synthetic data, off-road autonomy, data augmentation, batch inference, model multiplexing
Related resources
https://docs.ray.io/en/latest/serve/model-multiplexing.html
https://docs.ray.io/en/latest/data/data.html
https://docs.ray.io/en/latest/ray-core/actors.html
SEO description
Generate weather variations of real off-road video with NVIDIA Cosmos Transfer 2.5 on Ray: 288 clips on 8 A100s and a live Ray Serve application. (145 characters)
Blog URL
/every-season-every-trail-ray-cosmos-transfer
Ray Data · Ray Core · Ray Serve · NVIDIA Cosmos

Every Season on Every Trail: Scaling Off-Road Video Augmentation with Ray and NVIDIA Cosmos Transfer 2.5

By Alicia Chua, Alexandr Plashchinsky, Nickhil Gupta, Abhishek Sriram, Abhishek Monangi and Omar Shorbaji

Off-road perception models need examples of night rain, snow and smoke, yet these conditions are scarce in recorded driving data. We used NVIDIA Cosmos Transfer 2.5 [2] to generate new weather and lighting for real ATV footage from TartanDrive 2.0 [1], a public off-road driving dataset from the AirLab at Carnegie Mellon University, preserving the terrain, vehicle motion and camera path of each recording. On 8 NVIDIA A100 GPUs, Ray [6] generated 288 variants in 51.5 minutes, and every request succeeded on its first attempt.

A real 5.8-second recording followed by Cosmos Transfer 2.5 outputs for golden hour, night rain, autumn, blue hour, blizzard and rain. Every output is generated from the same camera clip and its edge map.

The workflow runs on a single Ray cluster. Ray Data prepares clips and scores outputs, persistent Ray actors keep the 2B-parameter model loaded on each GPU, a scheduler built on ray.wait handles retries and resumption, and Ray Serve exposes the same actors through an interactive application in which users select a drive, describe a condition and watch 8 GPUs generate results in parallel. The template includes a replay of stored results that runs without GPUs; generating new clips requires GPU resources.

288 / 288clips succeeded on the first attempt, 0 out-of-memory errors
51.5 minfor the full run on 8 A100s (6.45 GPU-hours of work)
6.98xgeneration speedup on 8 GPUs vs 1 (87% efficiency)
100.9 sto generate 8 drives in the live Ray Serve application

Interactive demo

Select a drive and a condition. The player replays outputs from the batch pipeline at the model's native 1024×544 resolution, with the original footage to the left of the divider and the generated version to the right. Select "All 9 drives" to view the same prompt applied to every scene.

Interactive replay: stored outputs, no GPU neededopen full screen

The live application

The replay reproduces a live Ray Serve application. In the recording below, a new user selects a drive, chooses golden hour and submits the request to all 8 GPUs. The page reports the progress of each GPU during generation and then displays the results with a comparison slider.

Screen recording of the live application on Ray Serve with safety guardrails enabled. The generation period is shown at 12x speed. All 8 clips completed 100.9 seconds after submission.

Motivation

Autonomous ground vehicles operating off-road encounter conditions that their training data rarely covers. Collecting footage of a night rainstorm on a specific trail requires waiting for the weather, deploying a crew in it and labeling the new data. Simulators provide control, but they rarely reproduce the appearance of real terrain such as wet grass and loose gravel.

World foundation models provide an alternative [4]. Cosmos Transfer 2.5 [2] takes a real video, a control signal derived from it and a text prompt, and generates a new video that follows the control. We use the edge map of every frame as the control. Because the edges come from the real recording, the generated clip keeps the trail's layout and the vehicle's motion, while the prompt sets the light, weather, season and surface.

Golden hour, night rain, autumn and blizzard, generated from the same recording and edge map.

Generation pipeline

Dataset: TartanDrive 2.0

ItemValue
SourceTartanDrive 2.0 [1], AirLab, Carnegie Mellon University; CC BY 4.0; ATV front camera (image_left_color)
Drives9 recorded runs
Clips27 prepared (3 per drive); 24 used in the main run, 9 showcase scenes for the extra conditions
Download13.39 GB in 82.75 s
Clip format93 frames, 1024×544, 16 fps

Results

Scaling

We ran the same 24 requests on 1, 2, 4 and 8 actors, each with a dedicated A100, in fresh namespaces with no resume. The median request took 79.79 seconds on a single actor and 80.27 seconds with eight running concurrently, so parallel actors do not slow individual requests.

ActorsWall timeSpeedup, generationSpeedup, end to endMedian per request
12067.75 s1.00x1.00x79.79 s
21107.90 s1.96x1.87x80.19 s
4624.94 s3.75x3.31x80.26 s
8398.39 s6.98x5.19x80.27 s

End-to-end speedup includes a fixed cold start of about 106 seconds per actor (loading the model), spread over only 3 requests per actor in the 8-actor case. On longer runs this cost becomes small: the main run below spent 88 seconds before its first dispatch and 51.5 minutes generating.

Speedup and parallel efficiency for 1, 2, 4 and 8 actors
Speedup relative to a single actor (left) and parallel efficiency (right). The generation window excludes the model cold start.
Timeline of 8 actors each loading the model then generating 3 requests back to back
Every actor loads the model once, then processes its requests back to back with no idle gaps.

The main run

The main run combined 24 clips with 4 conditions (fog, rain, night, snow) and 3 seeds: 288 requests on 8 A100s. All 288 succeeded on the first attempt in 3089.64 seconds, with no out-of-memory errors, no infrastructure failures and no replacement actors. Executed sequentially, the same work would require 23,219 seconds of GPU time.

Time per request

The denoising steps account for 79% of each request, so throughput scales primarily with the number of GPUs working in parallel.

Stage (median of 288)SecondsShare
Denoising, 4 steps63.5779.1%
Video guardrail5.036.3%
Writing output and control videos4.856.0%
Decode and postprocessing4.035.0%
Preprocessing1.181.5%
Text guardrail0.640.8%
Total per request80.32100%

Consistency across drives

Because the scene structure comes from each recording, the same prompt produces results that follow the layout of every drive. Night rain received the highest automated review score among the additional conditions.

The night rain prompt applied to all nine showcase drives. Headlight falloff, wet ground and rain streaks follow each trail's own layout.
Autumn on a rocky trail under trees.
Blue hour on a wet gravel road.

Automated quality metrics

We scored every output with a Ray Data pass on CPU. Edge F1 compares the edges of each output with those of the source clip within 2 pixels and measures how much of the original structure is preserved. Fog scores lowest, as expected, because fog obscures distant edges.

Condition (72 outputs each)Edge F1 vs sourceChange in brightness
Snow0.820+17.3
Rain0.779−33.9
Night0.640−82.4
Fog0.529+7.8

HSV saturation overstates color on dark frames because it is normalized by brightness: night outputs measure 170.7 against 41.8 for the source footage. CIELAB chroma, which is independent of brightness, shows a smaller increase from 8.47 to 15.44. The automated vision-language judge (Cosmos-Reason1-7B [3], blind to the requested condition) proved to be weak evidence, labeling 64 of 71 rain clips as "overcast", so its scores were used only to rank showcase candidates. Human review of the outputs is planned.

Failure cases

The lowest-scoring of 371 judged outputs: wildfire smoke on a winding gravel path. The sun disc dissolves and smoke puffs appear and vanish between frames.

Our automatic checks flagged 6 of the 288 main-run outputs, all for frozen frames and all from the same source clip, where the vehicle barely moves. That clip already repeats frames, because the camera's 10 Hz recording is stretched to 16 fps, so the flags identify clips for manual review.

System design on Ray

Clip preparation, model hosting, GPU scheduling, quality scoring and interactive serving each present a distinct scaling problem. Each maps to a Ray abstraction, and all of them run on the same cluster of 8 nodes with an A100 GPU on each.

Ray Data: clip preparation

Clip preparation decodes the raw camera frames, checks timestamps, resamples to 16 fps, encodes the model input, verifies it by decoding it back and computes the edge control. A Ray Data pipeline distributes the work with a single clip per block across the GPU nodes' CPUs, so the GPUs stay free for generation:

Ray Data · Clip preparationPython · excerpt
ds = (ray.data.from_items(runs, override_num_blocks=len(runs))
      .flat_map(plan_clips, num_cpus=1, resources=WORKER_ONLY)       # choose clip windows per drive
      .repartition(target_num_rows_per_block=1)                      # one clip per block
      .map(prepare_clip, num_cpus=2, resources=WORKER_ONLY,          # decode, resample, edges, verify
           compute=ray.data.TaskPoolStrategy(size=concurrency)))
ds.write_parquet(catalog_dir)                                         # the prepared clip catalog
The Ray advantage: preparing 9 clips from scratch took 24.06 seconds of wall time for 129.49 seconds of per-clip work. Each clip writes into a directory keyed by a hash of its settings, so a rerun over all 27 clips with the same settings finished in 7.28 seconds.

Ray Core: persistent model actors

Cosmos Transfer 2.5 pins its own PyTorch and CUDA stack, which differs from the driver's environment. Each generation worker is a Ray actor that starts a long-lived child process under the Cosmos interpreter and talks to it over a pipe. The model loads once per actor, in 52 to 57 seconds, and then serves every request sent to that GPU:

Ray Core · Persistent GPU actorPython · excerpt
class CosmosActor:                               # created with ray.remote(CosmosActor), one GPU each
    def _start_child(self):
        rfd, wfd = os.pipe()                      # replies come back on a dedicated pipe
        self.proc = subprocess.Popen(
            [self.s["cosmos_python"], "-m", "offroad_ray.inference.subprocess_worker"],
            stdin=subprocess.PIPE, env=self._child_env(), pass_fds=(wfd,))

    def generate(self, req: dict) -> dict:     # model stays loaded between calls
        ...
The Ray advantage: the actor owns the GPU, the child process and its logs. If the child dies, the actor reports it and the scheduler replaces the actor on a healthy node, and the Cosmos dependencies remain isolated from the driver.

Scheduling with ray.wait

Generation requests vary little in cost, so the scheduler keeps a single request in flight per actor and dispatches the next request as soon as an actor becomes free. It retries infrastructure failures on a replacement actor, never retries a guardrail block, validates every output and writes a record per request, so an interrupted run resumes from the last completed request:

Ray Core · Dispatch loopPython · simplified
while pending or in_flight:
    done, _ = ray.wait(list(in_flight), num_returns=1, timeout=5.0)
    for ref in done:
        i, req, t0 = in_flight.pop(ref)
        record(req, ray.get(ref))                 # validate with ffprobe, write the record
        free.append(i)
    while free and pending:                    # refill idle GPUs immediately
        i, req = free.popleft(), pending.popleft()
        in_flight[actors[i].generate.remote(req)] = (i, req, time.time())
The Ray advantage: because requests are dispatched individually from a shared queue, a GPU never waits on another. In the main run each of the 8 actors served exactly 36 requests and was busy for 2905 to 2957 of the 3090 seconds.

Ray Data: quality scoring

The quality pass reads every output, computes edge agreement, brightness, contrast, color and temporal metrics, and aggregates them per condition. It ran on all 288 outputs in 60.12 seconds and also builds side-by-side review images for a static review page:

Ray Data · Quality metricsPython · simplified
ds = ray.data.from_items(rows, override_num_blocks=len(rows))
metrics = ds.map(compute_output_metrics, num_cpus=2, resources=WORKER_ONLY,
                 compute=ray.data.TaskPoolStrategy(size=32))
metrics.write_parquet(metrics_dir)
summary = ray.data.read_parquet(metrics_dir).groupby("condition").aggregate(
    Count(), Mean("edge_f1"), Std("edge_f1"), Mean("delta_luma"), ...).to_pandas()

Ray Serve: the interactive application

The live application reuses the same actor class inside a Serve deployment. Each GPU replica processes a single request at a time. A CPU front end built on FastAPI serves the page, queues jobs and lets the browser poll for progress, so a request that takes 80 seconds never holds a connection open. Scene IDs act as multiplexed model IDs, which sends repeat requests for a drive to a replica that has already encoded it:

Ray Serve · GPU tier and scene affinityPython · excerpt
@serve.deployment(max_ongoing_requests=1, ray_actor_options={"num_gpus": 1, "num_cpus": 4})
class Painter:
    def __init__(self, settings):
        self.engine = CosmosActor(idx, settings)      # same engine as the batch pipeline

    @serve.multiplexed(max_num_models_per_replica=12)
    async def _scene(self, scene_id): return scene_id

    async def paint(self, req):
        await self._scene(serve.get_multiplexed_model_id())
        return await asyncio.to_thread(self.engine.generate, req)

# In the FastAPI front end: route each job to a replica that already holds its scene
self.painter.options(multiplexed_model_id=job["scene_id"]).paint.remote(req)
The Ray advantage: Serve's model multiplexing also provides cache-affinity routing. The video encodes that depend only on the drive are cached on the GPU and reused bit for bit, and identical requests replay instantly from shared storage.

With guardrails on, 8 replicas were ready 121 seconds after deployment. In the recorded session, 8 drives finished 100.9 seconds after the click, with each clip taking 97.9 to 100.5 seconds. Those were the first requests on each replica, and the guardrail models were still warming up: the video check took 13.56 seconds against a median of 5.03 seconds in the batch run.

Single-clip latency

We also evaluated ways to reduce the latency of an individual clip. Splitting a clip across GPUs with context parallelism was slower than a single GPU in both 2-node and 4-node configurations, because each node in this cluster has a single GPU and the split adds network traffic to every step. Generating at 480p was about 3 times faster, but this distilled model is trained for 720p and the output turned to blur. torch.compile shaved a few seconds per clip, after warm-up runs of up to three minutes. Depth control, an alternative conditioning method, needs the non-distilled model with 35 steps: 867.5 seconds per clip, 10.8 times the edge model. On this hardware, the more effective strategy is to process more clips in parallel, which Ray supports directly.

Limitations

Cost

The batch work in this post, from the first smoke test through the ten extra conditions, used 3.32 hours of cluster time on 8 GPU nodes: 26.55 node-hours held and 17.40 node-hours of active generation. The main run alone held 6.87 node-hours.

Key takeaways

A world foundation model can turn a few hours of real off-road driving into the same trails at night, in rain, in smoke or in autumn, while the geometry of every frame stays true to the recording. The work is dominated by a fixed 4-step sampler on each clip, so throughput comes from running many clips at once: per-request time stayed flat from 1 to 8 GPUs, and 288 clips finished in under an hour without a failed request.

Ray supplies the execution layer around the model: Ray Data for preparation and scoring, persistent actors that isolate the model's dependencies, a dispatch loop that keeps every GPU busy and resumes cleanly, and Ray Serve to put the same engine behind an interactive app.

Get started

Open the template on Anyscale (DRAFT: replace with the template link before publishing). It includes the batch pipeline, the quality pass, the Ray Serve application and the replay page shown above, which runs without GPUs. Enable GPU execution when you are ready to generate new clips.

Acknowledgements. We thank the TartanDrive 2.0 team at the AirLab, Carnegie Mellon University, for releasing their off-road driving dataset openly, and the NVIDIA Cosmos team for the Cosmos Transfer 2.5, Cosmos-Reason1 and Cosmos-Guardrail1 models and code.

References

  1. M. Sivaprakasam, P. Maheshwari, M. Guaman Castro, S. Triest, M. Nye, S. Willits, A. Saba, W. Wang and S. Scherer. TartanDrive 2.0: More Modalities and Better Infrastructure to Further Self-Supervised Learning Research in Off-Road Driving Tasks. arXiv:2402.01913, 2024. arxiv.org/abs/2402.01913. Dataset: theairlab.org/TartanDrive2.
  2. NVIDIA. World Simulation with Video Foundation Models for Physical AI. arXiv:2511.00062, 2025. arxiv.org/abs/2511.00062. Code: github.com/nvidia-cosmos/cosmos-transfer2.5. Model: huggingface.co/nvidia/Cosmos-Transfer2.5-2B. Project page: research.nvidia.com/labs/dir/cosmos-transfer2.5.
  3. NVIDIA. Cosmos-Reason1: From Physical Common Sense To Embodied Reasoning. arXiv:2503.15558, 2025. arxiv.org/abs/2503.15558. Model: huggingface.co/nvidia/Cosmos-Reason1-7B.
  4. NVIDIA. Cosmos World Foundation Model Platform for Physical AI. arXiv:2501.03575, 2025. arxiv.org/abs/2501.03575. Guardrail model: huggingface.co/nvidia/Cosmos-Guardrail1. Platform: nvidia.com/en-us/ai/cosmos.
  5. H. Zhao, C. Yuan, F. Huang, X. Hu, Y. Zhang, A. Yang, B. Yu, D. Liu, J. Zhou, J. Lin and others. Qwen3Guard Technical Report. arXiv:2510.14276, 2025. Model: huggingface.co/Qwen/Qwen3Guard-Gen-0.6B.
  6. P. Moritz, R. Nishihara, S. Wang, A. Tumanov, R. Liaw, E. Liang, M. Elibol, Z. Yang, W. Paul, M. I. Jordan and I. Stoica. Ray: A Distributed Framework for Emerging AI Applications. 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2018. arxiv.org/abs/1712.05889. Project: ray.io, documentation: docs.ray.io, code: github.com/ray-project/ray.
  7. X. Zhai, B. Mustafa, A. Kolesnikov and L. Beyer. Sigmoid Loss for Language Image Pre-Training. arXiv:2303.15343, 2023. Model: huggingface.co/google/siglip-so400m-patch14-384 (loaded by the Cosmos guardrail stack).

Licenses and attribution

Get involved with Ray

To connect with the Ray community, join the Ray Slack or ask questions on the Discuss forum. To run this workload on managed Ray, get started with Anyscale. To learn more about Ray, see ray.io and docs.ray.io.