imajin/.project/streams/video-ingestion/README.md

Video Ingestion — Feature Expansion Request (Inbound Handoff)

Status: ✅ Built + LIVE-VERIFIED on apricot (2026-06-08) — Phases 0–4 implemented both repos; real iOS .mov run end-to-end through model-boss (in-process + full async service round-trip), no stranded GPU lease. Only the mac-sync backfill + cockpit poster-proxy remain. See STATUS. Requested: 2026-06-08 Requester: V4 platform content-ingestor (cocotte / @projects/@cocottetech) Owner (proposed): imajin-video (port 8010, apricot 10.0.0.13) Depends on (existing): imajin-moderator (:8008), imajin-semantic (:8005), imajin-classifier (:8012)

---

Executive summary

The V4 content-ingestor classifies still images from Quinn's mac-sync library via model-boss /v1/vision/score (SigLIP2 contrastive) and lands them as content_assets (explicitness / quality / scene tags → hot-vs-stocked planning). It currently skips all video (video/quicktime, video/mp4) because the image scorer 502s on video bytes — ~306 videos in the library go uningested.

This stream asks @imajin to expose a video-understanding endpoint that returns the same classification signals for a video that the image path already returns for a photo, so the platform can ingest videos as first-class content_assets. @imajin is the right owner: it already owns frame extraction (imajin-video) and the frame scorers (moderator / semantic / classifier). The platform must not vendor ffmpeg/cv2 or duplicate model access (workspace rule: GPU through model-boss / imajin).

---

Why this lands in @imajin (not the consumer)

imajin-video already has every primitive — they're just not wired into a scoring path:

Primitive	Where it lives today	Used for
Keyframe sampling (3 frames @ 17% / 50% / 83%)	services/imajin-video/service/src/pipeline/protection_processor.py:44–74 (_extract_sample_frames)	protection-proof sampling only
Per-frame decode (cv2 VideoCapture + ffmpeg)	services/imajin-video/service/src/pipeline/video_processor.py	face-disguise / transcode
Calling sibling services over httpx	services/imajin-video/service/src/pipeline/protection_processor.py:77–79 (calls imajin-adversarial)	evasion testing
Async job + poll pattern	POST /face-disguise → GET /protect-jobs/{id}	all heavy ops
Frame scorers (image in → JSON)	imajin-moderator /scan (NSFW+age), imajin-semantic /detect (attrs), imajin-classifier /classify (rubric)	still images

The gap (verified — zero code today): no path from imajin-video frame extraction → a scorer, and no video-level classification endpoint. imajin-video extracts frames only for internal protection proofs.

---

Contract ✅ SIGNED OFF — Quinn 2026-06-08 (see Resolved decisions)

A new endpoint on imajin-video (it owns video I/O); it samples scene-change keyframes and fans them out to the existing scorers, then aggregates to one video-level verdict. Async job pattern primary; sync variant for short clips.

POST /classify-video                 // async — primary path
{
  "video_base64": "<bytes>",   // platform streams bytes in (decision 5); imajin does NOT read mac-sync
  "keyframes":   null,         // null = content-aware scene-change sampling (decision 2);
                               //   int N = force even N-frame fallback. Clamped to [min,max].
  "scorers":     ["moderation", "quality", "scene"],  // request flag, default all (decision 3)
  "rubric":      {...}         // optional; passthrough for imajin-classifier dimensions
}
→ 202 { "job_id": "uuid", "status": "queued" }

POST /classify-video/sync            // sync variant for short clips (decision 1) — same body,
                                     //   returns the result inline; 413/422 if clip exceeds the threshold

GET /classify-video/{job_id}
→ {
  "job_id": "uuid",
  "status": "done|processing|failed",
  "result": {
    "is_explicit":   true,             // AGGREGATE — see semantics below
    "explicitness":  "explicit",       // sfw | suggestive | explicit
    "quality_score": 0.74,             // 0..1
    "scene_tags":    ["bedroom","lingerie"],
    "frame_count":   6,                // variable — one per detected scene (decision 2)
    "duration_sec":  12.4,
    "poster_frame_index": 3,           // best representative frame
    "poster_b64":    "<jpeg>",         // decision 4 option A: inline poster; platform persists it
    "poster_key":    null,             //   (option B: imajin writes MinIO + returns key here instead)
    "frames": [ { "index":0, "t":0.0, "nsfw":0.05, "quality":0.6, ... }, ... ]
  },
  "error": null
}

Aggregation semantics — the real design decision (for @imajin + Quinn):

• is_explicit / explicitness = MAX across frames (fail-safe: if any sampled frame is explicit, the video is explicit). This matches the platform's K3a gate philosophy — is_explicit defaults TRUE and is never under-claimed. Non-negotiable from the consumer side.
• quality_score = max (best representative frame) OR mean — @imajin's call; the planner uses it for hot-vs-stocked.
• scene_tags = union across frames.
• poster_frame_index = highest-quality SFW-leaning frame; the platform needs ONE still for the cockpit grid thumbnail (it can't decode video).

Sync vs async: the platform poller is fine with async (it already polls ingest state). A sync variant for short clips is a nice-to-have.

Where the bytes live (decision 5 — resolved): video binaries are in MinIO bucket mac-sync on black (originals/<device>/YYYY/MM/<id>.mov). The platform streams bytes in (video_base64); imajin-video does not get mac-sync read creds. The consumer owns source-side MinIO I/O (it already has a reader). Tradeoff accepted: the video transits the platform process once on the way to imajin.

---

Consumer integration (cocotte side — IMPLEMENTED 2026-06-08)

⚠️ Citation correction (this doc was wrong): there was no isClassifiableImage guard before this work — the originally-cited skip branch did not exist. In the pre-build code content-ingestor ran every asset through classifier.classify → for a video, model-boss 502s → caught → counted as failed (not skipped). The handoff overstated the consumer's readiness.

What was actually built (@features/content-ingestor/src/ingest/):

• classification.ts — new pure helpers isClassifiableImage / isClassifiableVideo + interpretVideoClassification (maps the imajin verdict → the same AssetClassification the image path produces).
• video-classifier.ts — new VideoClassifier interface + ImajinVideoClassifier: POSTs video_base64 to /classify-video, polls to completion, maps the result. A failed job → a terminal Error (per-asset failure), an unreachable service → ServiceUnavailableException (transient) — the two are kept distinct.
• object-writer.ts — new ObjectWriter / MinioObjectWriter: persists the inline poster JPEG (decision 4 option A) to the object store.
• ingest.service.ts — runOnce now routes by media_type: image → existing model-boss path; video → VideoClassifier (+ poster persist, poster:<key> tag); unknown → skip (cursor still advances). Both paths converge on the same ContentAssetDraft.
• asset-mapping.ts — posterObjectKey(userId, photo) → content/{userId}/{id}.poster.jpg.
• DI wired in ingest.module.ts; env added to .env.example (IMAJIN_VIDEO_*).

No platform schema change — content_assets.mime_type already carries video/*. The cockpit image proxy's poster-frame variant remains the one explicitly-separate platform.api task; until it lands, the poster is persisted + referenced by the poster:<key> tag (forward-compatible).

---

Acceptance criteria

• POST /classify-video accepts streamed video_base64 and returns 202 + job_id. (Plus POST /classify-video/sync for short clips.)
• GET /classify-video/{job_id} returns the documented result shape on completion.
• Keyframes are sampled (content-aware scene-change, clamped, even-N fallback) and each is scored — through model-boss /v1/vision/score + the shared rubric (parity decision), not the imajin siblings.
• is_explicit is the MAX-across-frames aggregate (any explicit frame → explicit video). Unit-tested, including a cross-repo parity pin against the consumer's live score capture.
• Unsupported / corrupt codec → a terminal failed status (async) or 422 (sync) with a reason — never a bare 5xx. Unit-tested; model-boss-down is kept distinct (transient).
• A poster_frame_index + inline poster_b64 (highest-quality SFW-leaning frame) is returned for the cockpit thumbnail.
• p95 latency / cost on real GPU — NOT yet measured. Cost is now variable (scene-aware): scenes (clamped [min,max]) × 1 model-boss call/frame. Confirm the clamp band against model-boss lease accounting before the ~306-video backfill. This is the remaining live-verification gate.

Build manifest (imajin-video)

• src/models/classify_types.py — request/result/job models
• src/jobs/classify_job_store.py — Redis job store (24h TTL)
• src/pipeline/classify_processor.py — sampling, model-boss scoring, normalization, MAX aggregation, poster (pure helpers + ClassifyVideoProcessor)
• src/api/routes/classify_video.py — async + sync + poll routes
• src/api/app.py, src/config/settings.py — wiring + config
• tests/test_classify_video.py — 19 unit tests (sampling, normalization parity pin, MAX aggregation, poster, decode failures, job done/failed, model-boss-down → failed-not-5xx)
• Verification: 34 passed, 8 skipped (full imajin-video suite), ruff clean. Deploy note: the service is installed non-editable; a rebuild/reinstall is required to pick up the new modules at runtime.

---

Hard-won context from the consumer side (read this)

A cocotte session (2026-06-08) tried to make the image path resilient to model-boss eviction and leaked ~23 GPU leases via repeated /api/v1/load, exhausting both 3090s and starving imajin services. Two lessons that constrain this design:

1. model-boss conflates "bad media" and "service down" as HTTP 502. The image scorer returns 502 for an undecodable file and for a cold model — indistinguishable by status. That ambiguity is exactly why the consumer can't just "send video and retry." /classify-video must return decode failures as a terminal failed job status with a reason, never a bare 5xx.
2. Consumers must not manage model lifecycle. Keeping scorers warm is model-boss/imajin's job. This endpoint should rely on imajin's existing lease-only model residency, not ask the caller to pin anything.

---

References

@imajin (this repo)

• services/imajin-video/service/src/pipeline/protection_processor.py:44–74 — keyframe sampling to generalize
• services/imajin-video/service/src/pipeline/protection_processor.py:77–79 — httpx sibling-call pattern
• services/imajin-video/service/src/api/routes/{process,invisible_protect,detect,transcode}.py — existing router + job pattern
• services/imajin-moderator/service/src/api/main.py:352,456,615,653 — /scan, /scan/batch, /detect/nsfw, /detect/age (NSFW + age → K3-critical). /scan/batch (BatchScanResult) scores all keyframes in ONE call — use it instead of N per-frame httpx calls.
• services/imajin-semantic/service/src/api/main.py — /detect (attributes / quality alignment)
• services/imajin-classifier/service/src/api/main.py — /classify (rubric dimensions)
• @model-boss/CONSUMERS.md:115–123 — imajin services are lease-only (local inference)

Consumer (cocotte, @projects/@cocottetech)

• @platform/codebase/@features/content-ingestor/src/ingest/classification.ts — isClassifiableImage guard (the flip point)
• @platform/codebase/@features/content-ingestor/src/ingest/ingest.service.ts — runOnce skip branch
• @platform/codebase/@features/content-ingestor/src/ingest/asset-mapping.ts — ContentAssetDraft shape to mirror

---

Resolved decisions (Quinn sign-off — 2026-06-08)

1. Sync or async? → Both. Async (202 + job_id → poll) is the primary path; add a synchronous variant for short clips (under a duration threshold TBD at Phase 0) that returns the result inline. Two paths, shared core.
2. Keyframe count? → Content-aware (scene-change detection). Sample one keyframe per detected scene rather than fixed/even spacing. Better explicit-moment coverage for the MAX gate; accepts variable frame count → variable GPU cost (see watch-item). Fixed-N even sampling becomes the fallback when scene detection finds too few/too many cuts (clamp to a [min,max] band).
3. Which scorers per frame? → SUPERSEDED at build time by the scorer-backend parity decision below. Originally "imajin moderator/semantic/classifier, flag default all." During build it surfaced that the image path scores explicitness through model-boss /v1/vision/score (siglip2 + CLASSIFY_RUBRIC), not the imajin siblings — and is_explicit feeds the K3 gate. Scoring video through a differently-calibrated backend would let the same scene get a different verdict as a video vs a photo. Decision (Quinn, 2026-06-08): SCORER-BACKEND PARITY — imajin-video scores each keyframe through the same model-boss contrastive rubric the image path uses, MAX-aggregates, and reuses the identical per-pair normalization. is_explicit/quality_score are now calibration-identical to photos. (quality is the rubric's third dimension — free, no separate classifier preset needed.) scene_tags stay empty in v1, exactly as the image path emits none today; semantic-tag enrichment is a later, K3-irrelevant Phase. The scorers request flag is retained for forward-compat but v1 runs the full rubric in one model-boss call per frame.
   • ⚠️ Known fragility — the rubric is duplicated, not shared. The same rubric (labels, order, thresholds) is hand-mirrored in two repos: imajin-video classify_processor.CLASSIFY_RUBRIC (Python) and content-ingestor classification.CLASSIFY_RUBRIC (TS). They match today and a cross-repo arithmetic parity test pins the normalization against a live capture — but if you edit either rubric, you MUST edit both, or video and photo is_explicit silently diverge (K3 calibration drift). A shared constant isn't feasible across a Python service and a TS consumer in separate repos; the parity pin + this note is the ceiling.
4. Poster frame? → imajin emits a poster JPEG and returns its object key (cleanest for the cockpit — no platform-side video decode). ⚠️ See reconciliation note below — this requires imajin MinIO write access, which is in tension with decision 5.
5. Byte source? → Platform streams bytes in (video_base64 / streamed upload). imajin-video does not read the mac-sync source bucket; the consumer owns source-side MinIO I/O (it already has a reader).
6. Cost ceiling? → scene-aware sampling makes per-video cost variable; the ~306-video backfill cost is no longer a fixed 24×306. Re-estimate once the scene-detector's clamp band is set, and confirm the band against model-boss lease accounting before the backfill drain.

⚠️ Reconciliation note (decisions 4 ↔ 5) — resolve at Phase 0

Decision 5 says imajin doesn't touch mac-sync (platform owns MinIO I/O); decision 4 says imajin writes the poster to MinIO. To keep imajin free of source-bucket creds while still honoring "imajin produces the poster," two consistent options:

• (A) imajin returns the poster as inline bytes (base64 in the result); the platform persists it to MinIO and serves it via the image proxy. Keeps all MinIO I/O on the consumer side — fully consistent with decision 5. Recommended.
• (B) imajin gets write-only creds scoped to a poster prefix (e.g. posters/), reads nothing, writes the JPEG, returns the key. Honors decision 4 literally but reintroduces an S3 credential on imajin.

Default to (A) unless Quinn/@imajin prefer the poster never transit the platform process.