Why an open-source pipeline, in the age of Gemini Omni and Runway Aleph
Hosted generative models now cover a significant part of this problem space: Google’s Gemini Omni edits video conversationally, and Runway’s Aleph performs prompt-based object replacement on existing footage. For many tasks these are the most efficient option. They have, however, two structural limitations.
The first is content policy. Modifying a specific person’s head or face — for example, compositing a cyberpunk mask onto the driver of a car in a scene — is the kind of identity edit that hosted models broadly restrict, regardless of whether the footage and the actor are the user’s own. For legitimate VFX work this is a hard blocker, not an inconvenience.
The second is data locality. The footage must be uploaded to a third-party service, which is often unacceptable for client work or unreleased material.
A locally executed open-source pipeline is subject to neither constraint. The cost is integration work: the components exist, but they have to be assembled into a working whole. That is what this repository provides.
Pipeline overview
The workflow chains four stages inside ComfyUI:
- Text-based detection with Qwen2.5-VL. The target is specified in natural language — for example “the head, hair and neck area of the female driver” — rather than with manual coordinates. The vision-language model locates the region on a chosen keyframe.
- Segmentation and tracking with SAM2. SAM2 converts the detection into a precise mask and propagates it across the entire chunk — in both directions, using the patch described below.
- Inpainting with Wan 2.1 VACE. The masked region is regenerated, guided by a reference image of the intended content. The pipeline uses the 1.3B fp16 VACE model with the CausVid LoRA, which fits within 16GB of VRAM.
- Automated chunking and reassembly. A Python script drives the ComfyUI API, renders the video in 81-frame chunks, and concatenates the results losslessly with ffmpeg.
Mask-grow and blur nodes sit between stages 2 and 3, so the inpainted region blends into its surroundings without visible seams.
The mask is a separate deliverable. In addition to the final render, the workflow writes the tracked segmentation to a standalone black-and-white matte video. This output carries over directly into post-production: imported as an external matte — in DaVinci Resolve, for instance — it allows secondary color correction, denoising or relighting to be applied precisely to the modified region, without repeating any tracking work.
The core problem: SAM2 only tracked forward from frame 0
Kijai’s ComfyUI-segment-anything-2 nodes hardcode the tracking start at frame_index = 0 and propagate forward only. This is sufficient when the subject is clearly visible on the first frame; in practice it frequently is not.
When the target is detected with a VLM, the optimal keyframe is wherever the subject is largest and least occluded — which may be frame 72 of an 81-frame chunk, the moment the driver turns toward the camera. With the stock node, every frame preceding the detection frame receives no mask.
The patch modifies two things in nodes.py:
- frame_index becomes an input on the Sam2VideoSegmentationAddPoints node, so the tracking anchor can be set dynamically — for example, fed directly from the detection stage.
- propagate_in_video runs twice — forward from the anchor frame to the end of the chunk, then in reverse from the anchor back to frame 0 — and the two passes are merged into a single consistent mask sequence.
As a result, the track can be anchored on the most informative frame of the chunk rather than the first one, while the mask still covers every frame. The patch includes installation instructions and remains under the original Apache 2.0 license, with the modifications documented as the license requires.
Chunked rendering under VRAM constraints
On a 16GB card, Wan 2.1 runs out of memory beyond roughly 81 frames (approximately 3.3 seconds at 24 fps). The pipeline treats this as a fixed constraint rather than attempting to work around it: render_full_video.py slices the input video into 81-frame chunks by adjusting skip_first_frames and frame_load_cap on the LoadVideo node, queues each chunk through the ComfyUI HTTP API, polls until completion, and finally concatenates the rendered segments with a lossless ffmpeg concat.
The configured workflow is exported once with ComfyUI’s “Save (API)” function; from that point the script handles slicing, queuing and reassembly without manual intervention.
Two practices that determine output quality
The infrastructure above is necessary but not sufficient. In testing, two workflow decisions had the largest effect on the plausibility of the result.
A fresh retouched reference frame for every chunk. VACE is guided by a reference image describing the intended content of the masked region. Reusing a single reference for the entire video degrades consistency. The reliable procedure: export the keyframe used for detection, apply the intended modification to that still image, crop it tightly as an RGB JPG with a neutral background (not a transparent PNG), and use it as the reference. When rendering continues with the next 81-frame chunk, the procedure is repeated with a frame from that chunk. Lighting, angle and motion blur then match the source material, and consistency across chunk boundaries improves substantially.
A lower denoise value than intuition suggests. Wan 2.1 VACE integrates edits into their surroundings aggressively. At high denoise values this works against the operator: the model reconstructs a plausible version of the original scene and the intended modification is progressively suppressed. The denoise value should be reduced until the modification remains clearly present while still inheriting the scene’s lighting and motion.
Responsible use
This pipeline modifies the appearance of people in video. It is intended for VFX work on footage the user holds the rights to — costume and prop replacement, previsualization and comparable tasks. It should not be used to depict real persons without their consent or to produce misleading content. The repository’s README states the same conditions.
Repository and requirements
The project is available on GitHub under the MIT license (the SAM2 patch remains Apache 2.0): tenpel/ComfyUI-Bidirectional-SAM2-Inpaint.
Requirements: ComfyUI with VideoHelperSuite, the Qwen-VL detection nodes, Kijai’s SAM2 nodes with the included patch, and the Wan 2.1 model family.
The README documents the setup step by step.
Issues and feedback are welcome in the repository.