What GPU Memory Actually Limits in a Depth Pipeline
VRAM is the silent bottleneck in every depth and stereo pipeline. Understanding what fills it — and what does not — changes how you configure your processing.
Every AI model that runs on a GPU needs to fit its parameters, activations, and input/output tensors into GPU memory (VRAM). When there is not enough VRAM, the operation either fails with an out-of-memory error or silently degrades by falling back to CPU processing at a fraction of the speed. Understanding what consumes VRAM lets you make informed trade-offs.
Model weights are the fixed cost. A typical depth estimation model (Depth Anything V2 at the "small" configuration) uses approximately 100MB of VRAM for its parameters. The "base" model uses 400MB. The "large" model uses 1.2GB. These numbers do not change with input resolution — they are the cost of loading the model, period.
Activations are the variable cost, and they scale with input resolution. When a model processes a 1080p frame, the intermediate feature maps (activations) might consume 2-4GB of VRAM. At 4K, the same model's activations consume 8-16GB. This is why a GPU with 8GB of VRAM can process 1080p frames comfortably but chokes on 4K — the model fits, but the activations do not.
The practical implications are clear. If you have a GPU with 8GB of VRAM (like an RTX 3060 or 4060), you can run most depth models at 1080p without issues. At 4K, you need to either tile the input (process it in overlapping patches and stitch the results) or use a model that was designed for lower memory usage. Tiling works but introduces potential seam artifacts at patch boundaries.
For a full conversion pipeline — upscaling, interpolation, depth, stereo — the VRAM requirements compound because each model needs to be loaded simultaneously or swapped in and out. Loading all models at once is faster (no swap overhead) but requires more VRAM. Sequential loading is slower but fits in less memory.
The desktop-first approach matters here. When you run the pipeline on your own GPU, you know exactly how much VRAM you have, and the software can configure itself accordingly. Cloud processing with high-end GPUs (A100, H100) removes the VRAM constraint entirely, but at a cost per minute. The hybrid model — free desktop processing for standard resolutions, paid cloud processing for 4K and above — maps naturally to the economics of VRAM.