VPdata_processing

Extracted VideoPainter VPData assets, plus documentation for the processing scripts that turn them into inpainting / outpainting control-video training pairs.

A "control video" here is the original footage with part of it painted solid black — the region a generative model is asked to reconstruct. Inpainting blacks out the inside of an object's box; outpainting blacks out everything outside a visible box.

---

Paths

The data lives here; the scripts live in the sibling VPData/processing/. Every command below is anchored on these two roots, so it runs from any working directory:

export VPDATA=/mnt/dataset/xinyuy/datasets/VPData
export PROC=/mnt/dataset/xinyuy/datasets/VPdata_processing

---

What's in this folder

Extracted from the VPData zips

Layout follows the VPData README's canonical structure, matching what the VideoPainter training code expects and what ../VPBench/ already uses.

The videovo raw videos are deliberately not extracted here. They remain as 74 zips in $VPDATA/videovo_raw_videos/ (475 GB, 73,297 clips). Nothing in this pipeline uses them — it operates on the Pexels subset only. To unpack them into the canonical videovo/raw_video/<group>/<clip>.mp4 layout:

mkdir -p $PROC/videovo/raw_video
ls $VPDATA/videovo_raw_videos/*.zip | parallel -j 12 unzip -qq -o {} -d $PROC/videovo/raw_video

Note that four clips are 0 bytes upstream — declared 0-length inside the source zips, so no re-extraction will fix them: 000005036/000005036288.0.mp4, 000005050/000005050262.0.mp4, 000005055/000005055471.0.mp4, 000005065/000005065795.0.mp4.

Pipeline outputs (hardlinked from VPData/, so they cost no extra disk)

inpainting ⊃ overlay (3,402 of 3,404) ⊃ outpainting (all 3,000). Clip counts exceed selected_videos because the folders accumulated across runs with different inputs; the per-videoId dedupe applies within a single run, not across runs.

---

Pipeline at a glance

pexels.csv ─────────────────► download_pexels.py ──► selected_videos/<videoId>.mp4
                                                              │
video_inpainting/pexels/*/all_masks.npz ─┐                    │
pexels_videovo_train_dataset.csv ────────┤                    │
                                         ▼                    ▼
                    ┌───────────────────────────────────────────────┐
                    │ overlay_masks.py   (one decode pass, 2 outputs)│
                    │   --overlay-dir      ──► overlay/      (QA)    │
                    │   --inpaint-mask-dir ──► inpainting/   (train) │
                    ├───────────────────────────────────────────────┤
                    │ outpaint_masks.py                             │
                    │   --outpaint-dir     ──► outpainting/  (train) │
                    └───────────────────────────────────────────────┘
                                         │
                                         ▼
                    build_inpainting_outpainting_dataset.py
                                         │
                                         ▼
        <output>/{stem}.mp4  +  {stem}_painting.mp4  +  dataset.json

Stages 2a and 2b are independent — run either or both. Stage 3 needs both folders.

---

Requirements

pip install requests av opencv-python numpy tqdm

av (PyAV) is used for encoding rather than cv2.VideoWriter: the bundled FFmpeg's VideoWriter backend can only open the mp4v fourcc (MPEG-4 Part 2), which most players and browsers refuse to play. All scripts write H.264 / yuv420p instead.

---

Input formats

$VPDATA/pexels.csv

The resolution filter in download_pexels.py parses _<W>_<H>_<fps>fps out of link.

$VPDATA/pexels_videovo_{train,val,test}_dataset.csv

391,416 / 1,000 / 568 rows.

One raw video can appear in many rows (different objects / ranges). Both mask scripts dedupe to the first row per videoId, so each raw video yields at most one clip.

all_masks.npz

Single key arr_0: a uint8 array of shape (num_frames, H, W). 0 is background, 1..N are per-instance ids. A row's mask for frame i is arr_0[i] == mask_id. Mask resolution may differ from the video's; overlay_masks.py nearest-neighbour resizes to match.

Both mask scripts accept --masks-dir pointing at a folder containing extracted <index>_<videoId>/ folders, pexels-*.zip archives, or a mix — build_mask_index() indexes whichever it finds and reads .npz bytes straight out of the zip when needed. Point it at $PROC/video_inpainting/pexels (extracted) rather than $VPDATA/pexels_masks (zips) to skip reading 1,247 zip central directories at every startup.

---

Scripts

1. download_pexels.py — fetch raw Pexels videos

Samples rows from pexels.csv and downloads them concurrently to <out>/<videoId>.mp4.

python3 $VPDATA/processing/download_pexels.py \
  --csv $VPDATA/pexels.csv \
  --mask-info-csv $VPDATA/pexels_videovo_train_dataset.csv \
  --out $PROC/selected_videos \
  -n 3000 --resolution 1920x1080 --workers 8 --seed 42

Behaviour worth knowing: downloads go to a .part file and are renamed only on success, so a killed run leaves no truncated .mp4. Existing non-empty files are skipped, making re-runs resumable. Failed URLs (Pexels links do expire) are replaced by drawing further samples until -n successes are reached — so the output count is exact, but the specific videos depend on how many links were dead.

---

2a. overlay_masks.py — QA overlays and inpainting control clips

Produces up to two outputs from a single decode pass over each clip's frame range:

• --overlay-dir — the frame alpha-blended with a deterministic colour per instance id. Visualization only, for eyeballing what the mask actually selected.
• --inpaint-mask-dir — the frame with the mask_id region's axis-aligned bounding box painted solid black.

python3 $VPDATA/processing/overlay_masks.py \
  --csv $VPDATA/pexels_videovo_train_dataset.csv \
  --videos-dir $PROC/selected_videos \
  --masks-dir $PROC/video_inpainting/pexels \
  --overlay-dir $PROC/overlay \
  --inpaint-mask-dir $PROC/inpainting \
  --alpha 0.5 --workers 32

Two deliberate design choices in the inpainting output:

• A bounding box, not the precise silhouette. Using the exact mask contour would leak the object's shape into the "inpaint this region" signal, letting a model cheat.
• A hard black replace, not an alpha blend, encoded losslessly (crf=0). The control clip is read back downstream as a strict binary mask; lossy encoding would smear the box edges into intermediate values.

The overlay output is not lossless — it is only ever looked at.

Output naming: {index}_{videoId}_{start}-{end}.mp4.

---

2b. outpaint_masks.py — outpainting control clips

The inverse of the inpainting output: everything outside a visible box is painted black, so the model sees a crop and must hallucinate the surrounding context.

python3 $VPDATA/processing/outpaint_masks.py \
  --csv $VPDATA/pexels_videovo_train_dataset.csv \
  --videos-dir $PROC/selected_videos \
  --masks-dir $PROC/video_inpainting/pexels \
  --outpaint-dir $PROC/outpainting \
  --bbox-frac-range 0.4-0.8 --seed vpdata --workers 32

How the visible box is sized (--bbox-frac-range, default 0.4-0.8):

1. One target fraction is drawn per clip, not per frame — otherwise the visible region would jitter every time the object drifted in and out of range.
2. For each frame, if the object's own bbox already covers 40–80% of the frame area, it is used as-is.
3. Otherwise the box is rescaled about its centre to hit the target fraction — grown outward if the object was too small, shrunk inward (cutting into the object) if too large — then shifted to stay inside the frame. Very elongated boxes that cannot fit are clamped to the frame extent, so the achieved area can fall short of the target.
4. Frames where mask_id is absent become fully black.

--seed takes a string and is combined per-clip as random.Random(f"{seed}:{key}"). random.Random(str) is stable across processes and PYTHONHASHSEED values (unlike hash()), so a seeded run is exactly reproducible across the worker pool. Omitting --seed is nondeterministic.

Also encoded losslessly, for the same binary-mask reason as above. Clips whose mask_id never appears in the frame range are skipped before the video is decoded, which is the expensive part.

---

3. build_inpainting_outpainting_dataset.py — assemble the training set

Selects distinct videos across inpainting/ and outpainting/ and materializes a flat dataset directory.

python3 $VPDATA/processing/build_inpainting_outpainting_dataset.py \
  --csv $VPDATA/pexels_videovo_train_dataset.csv \
  --videos-dir $PROC/selected_videos \
  --inpainting-dir $PROC/inpainting \
  --outpainting-dir $PROC/outpainting \
  --output $PROC/train_dataset \
  --num-videos 2000 --inpaint-ratio 0.5 --seed 0 --link-mode hardlink

Output layout:

train_dataset/
├── {stem}.mp4            # raw video, freshly trimmed to [start_frame, end_frame]
├── {stem}_painting.mp4   # the control clip (hardlinked from inpainting/ or outpainting/)
├── ...
└── dataset.json          # [{caption, media_path, reference_path}, ...]

Two things this script gets deliberately right:

• {stem}.mp4 is re-encoded, not hardlinked. The control clips cover only a [start_frame, end_frame] sub-range, while selected_videos/<videoId>.mp4 is the full video. Linking the full video would leave the pair frame-misaligned, so the raw footage is re-trimmed to the identical range.
• Distinct-videoId allocation. Every outpainting clip's videoId also has an inpainting clip (outpainting is a strict subset), so naive independent sampling would put the same source video in both halves. allocate() fills the smaller pool first and excludes its videoIds from the other pool's candidates. If a pool runs short it warns and returns fewer rather than failing.

--link-mode hardlink (default) costs no extra disk and falls back to a real copy across filesystems. Use copy if the output must survive deletion of inpainting/.

Only clips with both a present raw video and a caption in --csv are eligible.

---

Gotchas

Three built-in defaults are stale — pass these flags explicitly (the commands above already do).

The last one still works — the scripts read masks out of zips fine — but it re-indexes 1,247 zip central directories on every startup.

Other notes

• --workers defaults to every logical CPU. Each clip's decode+encode is independent and CPU-bound, so it scales near-linearly — lower it on a shared machine.
• end_frame is inclusive in the CSVs but the loops use min(end_frame + 1, num_masks), so clips are silently truncated when the mask array is shorter than the CSV claims.
• Failures are reported per clip and tallied, but never abort the run. Check the final ok/fail line.
• /mnt/dataset/xinyuy/datasets/processing/ (build_dataset/, select_and_trim/) contains 0-byte placeholder files — unrelated to this pipeline and not runnable.
• This pipeline uses the Pexels subset only. The videovo masks are extracted here for other consumers; the videovo raw videos stay zipped (see "What's in this folder").