Engineering / Case study 02

Running local AI on 8 GB of VRAM: what actually fits

Three failures, three fixes, and why the smallest model usually wins.

By Bryan's Lab · October 2026 · 10 min read

The setup

I wanted a full podcast clipping pipeline with zero cloud calls. Audio would become a transcript, a local LLM would pick highlight moments, and ffmpeg would reframe and render the selected clips. The hardware was an RTX 5060 Ti with 8 GB VRAM. Every stage had to work within that machine's resources.

The first case study covers the clipping pipeline and its delivery. This one is about getting the local AI stages to coexist. A model that loads successfully on its own can still leave too little memory for the next job. I found that out after fixing a dependency error and then watching the pipeline sit at 0%.

These are the readings and failures from my Windows debugging session. The memory estimates below are a planning budget for this workload, not a benchmark across every model configuration.

GPU: RTX 5060 Ti, 8 GB VRAM.
Transcription: faster-whisper through CTranslate2, initially using the small model.
Highlight selection: Ollama running granite4.1:8b at 16K context.
Reframing and rendering: ffmpeg, with AutoClip coordinating the job.

Failure 1: CUDA DLL resolution

Symptom: faster-whisper crashed with Library cublas64_12.dll is not found or cannot be loaded. Transcription never reached the point where model size mattered.
Root cause: CTranslate2 did not bundle the CUDA 12 runtime libraries this installation needed. The NVIDIA driver alone did not supply those DLLs. A working driver and a visible GPU were not enough to make the Python loader find cuBLAS.
Fix: I installed the CUDA wheels in the virtual environment, then copied their DLLs into CTranslate2's folder under site-packages, next to the loading binary.

uv pip install nvidia-cublas-cu12 nvidia-cuda-runtime-cu12

The copied files were cublas64_12.dll, cublasLt64_12.dll, and cudart64_12.dll.

On Linux, these libraries can be resolved through the configured linker path. Windows uses DLL search rules, and the wheel installation did not put the libraries where this loader could resolve them. Placing the DLLs adjacent to the loading binary fixed this environment. That is the workaround I used, not a claim that every Windows installation requires copying DLLs.

I would record the wheel versions and copied files with the environment. Otherwise, rebuilding the venv can bring back the same error even though the GPU driver has not changed. Fixing this lookup got transcription past startup. It did not create any more VRAM.

Failure 2: VRAM collision

Symptom: The pipeline stalled at 0% for 6+ minutes. The GPU was pegged at 99%, but there was no progress. AutoClip eventually reported Ollama timed out. Large transcripts on a small GPU can exceed the limit.
Root cause: With granite4.1:8b loaded at 16K context, ollama ps showed an 8.2 GB model and 24% CPU / 76% GPU spillover. nvidia-smi showed 7801 / 8151 MiB used, or 95% VRAM occupancy. faster-whisper small needed ~1.5 GB and the ffmpeg buffers in this workload needed ~1 GB. They could not fit in the remaining space.
Fix: Reduce the resident workload before retrying. The useful choices were a smaller LLM, a smaller context window, or a smaller transcription model. Repeating the same request against the same allocation would not change the budget.

ollama ps
nvidia-smi

The two readings describe different things. The 8.2 GB model figure from ollama ps includes a workload split across CPU and GPU. The 7801 / 8151 MiB reading describes occupied device memory. Neither says that 8.2 GB of model weights sat entirely on the card. Together, they show why loading the model was a poor test of whether the full pipeline would work.

Only 350 MiB remained in that device-memory reading. That was nowhere near the ~1.5 GB transcription allocation, before allowing for the ~1 GB of ffmpeg buffers. GPU utilization at 99% looked busy, but six minutes at 0% was the useful result. The workload had no room to proceed as configured.

The context window also costs memory. Keeping a larger KV cache lets the LLM retain more context during inference, but that allocation competes with everything else on the card. The model's parameter count and download size are not the whole runtime budget.

Swap to qwen2.5:3b-instruct-q4_K_M, at 1.9 GB. In this budget, the smaller model leaves ~4 GB headroom for the other stages and working memory. This is where I would start for highlight selection.
Reduce Ollama's context window from 16K to 4K. A smaller KV cache lowers the memory requirement while keeping the 8B model. The transcript still needs to fit the usable context alongside the prompt and response; longer episodes need smaller input passages.
Use the smallest Whisper model, tiny/base, instead of small. This reduces transcription's share of memory. Check the resulting transcript before relying on it to select and caption a clip.

The task was "pick 3 highlight moments from a transcript." Starting with granite 8B at 16K context spent most of the GPU budget before I knew whether that size improved the selections. For this task, it was overkill. I would establish a usable result with the 3B model before giving a larger one more memory.

There is also a scheduling decision. Transcription, selection, and rendering happen in sequence, but their processes can keep allocations alive between stages. A sequential pipeline does not automatically have a sequential memory budget. Each handoff needs to release memory that the next stage requires.

Failure 3: Orphaned processes

Symptom: ollama stop granite4.1:8b returned successfully, but VRAM stayed at 7.8 GB. The unload command had finished. The allocation had not gone away.
Root cause: In this session, AutoClip died mid-request and left a llama-server.exe child process holding the model. The API response did not mean that the remaining worker had exited. That runner name and failure path describe this setup.
Fix: I identified the process holding the GPU allocation and killed the orphaned llama-server.exe directly with Stop-Process. Then I checked device memory again before starting another job.

nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv

Use the PID belonging to the confirmed orphan, rather than terminating every model runner. In PowerShell, replace <orphan-PID> below with that process ID.

Stop-Process -Id <orphan-PID>
nvidia-smi

My rule after any cancelled AI job is to verify that VRAM returns to baseline before starting another. A closed client window and a successful stop response are not measurements of free memory. The GPU reading is what tells me whether the next run has room.

This also changes how I interpret a failed retry. If the previous worker is still resident, a new job is competing with an abandoned one. Until that allocation is gone, the retry cannot tell me whether the smaller model or context change solved the original problem.

What actually fits on 8 GB

Approximate model-stage VRAM budget for this setup. These estimates exclude the additional ~1 GB of ffmpeg buffers and other device allocations.
Task	Model	VRAM	Fits?
Transcription only	faster-whisper small	~1.5 GB	Yes
Transcription + 3B LLM	whisper small + qwen2.5:3b	~3.5 GB	Yes
Transcription + 8B LLM (4K ctx)	whisper small + granite4.1:8b	~6.8 GB	Tight but yes
Transcription + 8B LLM (16K ctx)	whisper small + granite4.1:8b	~9.7 GB	No

"Tight but yes" applies to the model stages in the table. It is not a guarantee that the full pipeline fits with rendering buffers still resident. Quantization, context, transcription settings, and other GPU users change the peak. Leave space for the next stage instead of planning to consume every last MiB.

The architectural lesson

"Running locally" means "managing resources like a 2010s sysadmin." I had to resolve DLLs, inspect memory, identify a worker process, and clean it up before the useful work could continue. Owning the hardware meant owning those handoffs too.

Cloud APIs hide the VRAM problem because I am paying for someone else's orchestration. Cloud brings rate limits and quotas; local execution brings memory management and process cleanup. Neither choice is wrong, but I want to know which tax I am paying before committing a workflow to it.

What I'd do differently

Start with qwen2.5:3b, not 8B, on any GPU under 12 GB. The task of "pick 3 moments from a transcript" doesn't need 8B parameters as a starting requirement.
Pin CUDA wheel versions in the venv. Don't rely on system CUDA to supply the Python environment's runtime dependencies.
Check for and kill confirmed orphaned LLM processes before every run. Verify that VRAM has returned to baseline before loading another model.
Compress output video from the first clip, not after three uploads. Save the 6× bandwidth cost upfront. The first case study's sample went from approximately 15 MB to 2.5 MB; I would make that delivery pass part of the first export.

When to use cloud instead

If this workflow runs 20+ times a day, I would treat cloud as cheaper than my time managing VRAM. That is my operating rule, not a measured break-even price for every provider or transcript length. Local processing makes sense when keeping source material on the machine matters or when the workload fits comfortably. If repeated stalls and cleanup consume the day, I would pay for orchestration and spend that time reviewing the clips.