August 2, 2026

How an Infinite Rectangle Broke CosmiCut's 2D-to-3D Conversion

Fixing black spatial videos, 4K memory crashes, and the failure paths that let both slip through.

cosmicut
swift
coreimage
spatialvideo
depthpro
debugging

CosmiCut’s 2D-to-3D conversion broke in a particularly rude way: sometimes it crashed, and sometimes it finished successfully and handed you a completely black video.

The second result was arguably worse. At least you know something went wrong when it crashes.

The conversion pipeline has several expensive pieces stacked together. We decode a regular video frame, estimate its depth, stabilize that depth across time, generate a slightly different image for each eye, tag both images, and write the pair into an MV-HEVC movie.

2D frame → depth map → temporal stabilization → left/right warp → MV-HEVC

So… I found two separate bugs in that pipeline:

The stereo renderer was accidentally doing math with an effectively infinite image rectangle.
Depth Pro’s temporal processing was holding a giant working set at full source resolution.

One explains the black frames. The other explains why 4K conversions could knock the app over. Let’s fix both.

The Infinite Rectangle

Core Image is lazy. A CIImage is less like a finished bitmap and more like a recipe for producing one. That is extremely useful, but it also means image geometry can behave differently than you might expect if you are thinking in terms of ordinary pixel buffers.

We use clampedToExtent() before blurring and warping frames. Clamping repeats the edge pixels forever, which prevents a blur from pulling transparent pixels in from outside the image. That part was intentional.

The problem was was the next part.

After clamping, we read the image’s extent and used it to calculate the frame width, parallax shift, kernel coordinates, and output dimensions. But a clamped Core Image extent is no longer the original 1920×1080 rectangle. It is effectively infinite.

So a real video frame went into the renderer, then this happened conceptually:

let clampedColor = adjustedColor.clampedToExtent()
let frameWidth = Float(clampedColor.extent.width) // Extremely, comically large.
let maxShift = frameWidth * maxParallaxFraction

Nothing good follows from sending those values into a stereo warp kernel. The kernel produced invalid sampling coordinates, the writer received black eye buffers, and the conversion could still (at least sometimes, when it doesn’t just straight up crash) reach the end without reporting an error.

Core Image was doing exactly what we asked. We just asked a rectangle with no meaningful boundary how wide the video was.

Keep the Finite Extent, Then Clamp

The fix is to capture and validate the adjusted frame’s real extent before clamping anything. Then we clamp only for sampling and immediately crop the result back to that finite rectangle.

In simplified form:

let sourceExtent = adjustedColorImage.extent
let renderExtent = try validatedStereoRenderExtent(sourceExtent)

let stereoInput = adjustedColorImage
    .clampedToExtent()
    .cropped(to: renderExtent)

let blurredDepth = scaledDepth
    .clampedToExtent()
    .applyingFilter("CIGaussianBlur", parameters: [
        kCIInputRadiusKey: 2.0
    ])
    .cropped(to: renderExtent)

The renderer now rejects extents that are infinite, null, empty, or contain non-finite values. It also verifies that the output buffers match the expected dimensions before rendering either eye.

That gave the stereo kernel a boring, finite coordinate system again. Boring is basically always better in code. Interesting gets you bugs XD.

I also changed the renderer to reuse one CIContext instead of constructing a new context for every frame. A Core Image context owns caches and GPU resources; throwing it away hundreds or thousands of times during a conversion is expensive for no benefit. And because CIContext is doing the rendering, the manual base-address locks around those GPU renders were unnecessary. Those are gone too.

The 4K Memory Problem

Fixing the infinite extent restored the image, but Depth Pro conversions could still crash. This turned out to be a more traditional resource problem.

CosmiCut stabilizes Depth Pro’s output in two stages:

Correspondence stabilization uses optical flow to line up depth between neighboring frames.
Hysteresis smoothing reduces smaller frame-to-frame changes that would otherwise look like flickering depth.

Both are important. Without temporal stabilization, a depth model can assign slightly different values to the same object in consecutive frames. A still frame may look great while the resulting video seems to shimmer or breathe in 3D.

The mistake was running all of that work at the video’s full resolution.

A single 3840×2160 32-bit buffer is about 32 MiB. Temporal correspondence needs multiple color, depth, flow, and history buffers alive at once. Add intermediate textures and both stabilization stages, and a 4K frame can push the live working set past 500 MB. Repeat that for a long video while the GPU and writer are also busy and iOS will eventually make a very reasonable decision about your app’s continued existence.

The tempting fix would be to turn temporal stabilization off. That would save memory, but it would also remove one of the things that makes Depth Pro video look good. So we kept it.

Stabilize at a Bounded Working Resolution

Depth Pro already operates at a native 1024-pixel scale. There is little value in upscaling its result to 4K, running optical flow and history there, and paying the memory cost for millions of interpolated pixels.

The new pipeline calculates an aspect-preserving temporal working size with a 1024-pixel longest edge. Dimensions are kept even for predictable pixel-buffer allocation.

let scale = min(1, 1024 / max(sourceWidth, sourceHeight))

let workingSize = CGSize(
    width: even(sourceWidth * scale),
    height: even(sourceHeight * scale)
)

That turns a 3840×2160 landscape frame into 1024×576. Portrait video becomes 576×1024. Smaller inputs stay at their original size.

For each frame, CosmiCut now:

Downscales the aligned color and direct depth buffers to the bounded working size.
Runs correspondence stabilization there.
Runs hysteresis smoothing there.
Flattens the stabilized depth.
Upscales the result once to the source dimensions for the stereo render.

At 1024×576, a comparable 32-bit buffer is 2.25 MiB instead of about 32 MiB. The exact total still depends on the optical-flow implementation and temporary GPU resources, but the working set is now bounded by the stabilization scale instead of the source video resolution.

If those smaller buffers still cannot be allocated, the pipeline resets its temporal history and uses that frame’s direct depth result. It does not retry the same work at full resolution. One less-stabilized frame is much better than a crashed conversion.

A Failed Conversion Should Actually Fail

The black-video bug also exposed a separate design problem: too many failure points returned nil or were allowed to fall through as if conversion had succeeded.

The conversion path now checks all the unglamorous but critical stuff:

The asset reader actually started.
The writer actually started.
Depth generation succeeded for every video frame.
Both stereo pixel buffers were allocated.
Every tagged left/right pair was appended.
Audio timing could be rebased and appended.
The reader and writer both completed successfully.
At least one video frame was written.

If any of those checks fail, conversion throws an actionable error and removes the temporary movie. Most importantly, CosmiCut never replaces the original clip with a failed or empty output.

I also removed a redundant second decoder from the conversion path. The encoder already owns a BGRA reader configured with the clip’s trim range, so conversion now uses that reader for both the fast and Depth Pro paths. Video and audio timestamps are rebased to zero from the trim start, which keeps the converted clip’s duration and synchronization aligned with what the user selected.

Testing the Actual Failure

This bug needed tests at the same boundaries where it escaped.

The new regression coverage checks that:

16:9, portrait, 1080p, and 4K inputs all produce finite stereo extents.
Temporal working dimensions never exceed 1024 pixels on their longest edge and remain even.
Synthetic color and depth images survive the real warp path with the expected dimensions.
The rendered warp contains non-black image data and both eye images keep the expected extent.
An infinite Core Image extent is rejected instead of reaching the kernel.

The broader conversion path now also has explicit seams for depth, allocation, and writer failures so those cases cannot quietly turn into a “successful” empty movie again.

tl;dr

CosmiCut’s black 2D-to-3D output came from reading a CIImage extent after clampedToExtent() had made it effectively infinite. We now preserve and validate the finite frame rectangle, clamp only for sampling, and crop back before running the stereo kernel.

The crashes were a second problem: Depth Pro’s optical flow and temporal history ran at full 4K resolution. Temporal stabilization is staying, but it now runs at an aspect-preserving size capped to 1024 pixels on the longest edge, then upscales once for output.

One infinite rectangle, several hundred megabytes of avoidable buffers, and a failure path that was far too optimistic. A very normal week in video software.

return to the bug pile