NVIDIA and Siemens Healthineers Just Rewired Ultrasound from the Ground Up

I’ve been watching ultrasound AI for a while now, and most of what gets hyped is just fancy filters applied to the final image. NVIDIA and Siemens Healthineers just did something different.

They released a model called NV-Raw2Insights-US that skips the traditional reconstruction pipeline entirely. Instead of working from the pretty pictures clinicians see on screen, it learns directly from the raw sensor data—the actual echoes coming back from the body. That might sound like a technicality, but it changes everything.

Why raw data matters

Ultrasound isn’t an image. It’s sound. What you see on the monitor is a reconstruction built from millions of tiny echoes, processed through decades-old assumptions. The biggest one: sound travels at a constant speed through all tissue. Anyone who’s ever had an ultrasound knows that’s not true. Fat, muscle, bone, fluid—sound moves through each differently. Traditional beamforming just picks one number and hopes for the best.

NV-Raw2Insights-US estimates a personalized sound-speed map for each patient in a single AI pass. That was previously a complex, time-consuming computation. Now it’s real-time. The model corrects image focus on the fly, adapting to how sound actually behaves in that specific body.

This is higher quality than I expected from a first-generation model. The team calls this class of models “Raw2Insights,” and the vision is end-to-end AI for ultrasound imaging. This is the first step.

The hardware story is interesting too

Raw ultrasound channel data is bandwidth-heavy. Most clinical scanners don’t even expose it. NVIDIA’s solution is Holoscan Sensor Bridge (HSB), an open-source FPGA IP that streams data from the scanner’s DisplayPort outputs over Ethernet to an NVIDIA IGX system. They call it Data over DisplayPort. It’s clever—leverages existing hardware without invasive modifications.

Once data hits GPU memory, inference runs on a Blackwell-class GPU. The sound-speed estimate streams back to the scanner, improving focus in the live feed. The whole loop is real-time.

This approach has been tried before in research labs with custom hardware. NVIDIA’s bet is that HSB makes it practical for clinical deployment. I’m skeptical about how easily this integrates into existing hospital workflows, but the technical execution is solid.

What this unlocks

Three things stand out:

First, software-only integration. If your scanner has DisplayPort, you can potentially add this capability without hardware changes. That lowers the barrier for existing devices.

Second, software-defined ultrasound. Once raw data is in GPU memory, you can update the AI models without touching the scanner hardware. Continuous improvement becomes possible in a field that’s been locked into fixed pipelines.

Third, modular expansion. Other AI models can be plugged into the same data stream. This isn’t just about sound-speed estimation—it’s a platform for whatever comes next.

The catch

This is investigational. Not cleared for clinical use. The paper references IEEE and arXiv publications, so the science is peer-reviewed, but real-world validation is still pending. NVIDIA and Siemens Healthineers collaborated closely, which gives it credibility, but I’d want to see independent replication before getting excited about clinical adoption.

Also, the hardware stack is NVIDIA-specific. IGX, Holoscan, Blackwell GPU—if you’re not already in that ecosystem, the barrier to entry is significant. The open-source FPGA IP helps, but it’s still a bespoke setup.

Bottom line

NV-Raw2Insights-US is a genuine step forward, not just another AI filter slapped on a reconstructed image. Learning directly from raw sensor data and adapting to patient-specific physics is the right direction. The modular architecture and software-defined approach make it more than a one-off model—it’s a foundation for future work.

Whether it translates to better clinical outcomes remains to be seen. But technically, this is the most interesting ultrasound AI I’ve seen in years.

You can dig into the code and weights on GitHub if you want to play with it yourself. The dataset is also available. NVIDIA and Siemens Healthineers put the resources out there, which is more than most medical AI projects do.