Skip to main content

Set video recording

Per‑set video capture, frame‑accurate sync to sensor rep data, technique‑tab review, and deferred upload. Branch: video-recording. Status (2026‑06‑17): iOS recording + review device‑tested and working; upload pipeline built; Android not started. This doc is the source of truth + handoff for all video work.

1. The core problem this feature solves​

Frame‑accurate video ↔ sensor‑rep sync. Each saved video stores a single wall‑clock timestamp of its first frame; every rep already carries a wall‑clock created and concentric‑relative raw‑data time t. Sync is subtraction:

videoTime(rep, t) = (rep.created − concentricDurationMs − firstFrameDate)/1000 + t

firstFrameDate is the trimmed video's first‑frame wall‑clock (epoch ms). It is the one anchor everything rests on — persisted on the set, sent to the backend, and written into the MP4 metadata.

Verified consistent: WorkoutRep.created == event.timeStamp (reps.ts), and the duration metric == event.t — the exact values the trimmer uses.

2. Architecture — EnodeVideo Capacitor plugin​

Native‑only (iOS AVFoundation / Android Camera2). Mirrors EnodeBle: a TS bridge + native implementations; all product logic stays in shared TS. Gate every call on isVideoSupported() (@enode/core/video) — web rejects.

  • TS bridge + types: packages/core/src/video/capacitor-video.ts
  • Barrel + isVideoSupported() + DEFAULT_VIDEO_CONTEXT_BUFFER_MS = 2000: packages/core/src/video/index.ts
  • Upload deps wiring: packages/core/src/video/upload.ts

Bridge methods: checkCapabilities, requestVideoPermissions, setupPreview ({stream?}), updatePreviewRect, switchCamera, setExposure (no‑op stub), startRecording, stopRecording, cancelRecording, trimAndSave ({contextBufferMs}), teardownPreview, getVideoPath, uploadVideo, moveVideoToUploaded, listVideos, deleteVideo. Events: previewReady, recordingStarted, firstFrameEncoded, recordingStopping, recordingCompleted, previewFrame (stream mode).

iOS (apps/tracking/ios/App/App/Plugins/)​

  • VideoRecorder.swift — AVCaptureSession + AVCaptureVideoDataOutput → AVAssetWriter/pixel‑buffer adaptor (push model). Skips the first 30 warm‑up frames (iOS drops early frames → t=0 offset; the skip fixes it consistently), anchors writerStartTime/firstFrameEpochMs on the first kept frame. 1080p 60→30 selection; orientation follows the UI interface orientation; auto‑exposure. Stream‑mode tap: downscales (≤720px) + JPEG (q0.5), throttled to ~24 Hz, base64 → onPreviewFrame.
  • VideoTrimmer.swift — crops to [firstRep.created − concentricMs − buffer, lastRep.created + buffer], re‑encodes (frame‑accurate start), writes commonIdentifierCreationDate (ISO‑8601 UTC) = creationTimestampMs = max(firstFrameEpochMs, desiredStart).
  • EnodeVideoPlugin.swift — CAPBridgedPlugin wrapper, preview placement (hole‑punch mode), stream wiring, permissions, upload/file methods.
  • Registration (Cap 8 gotcha): bridge?.registerPluginInstance(EnodeVideoPlugin()) in AppViewController.swift capacitorDidLoad() — without it JS gets "not implemented on ios". Also force‑ref in AppDelegate + 4 project.pbxproj entries per Swift file.

Android — FULL PIPELINE PORTED (not device-tested)​

EnodeVideoPlugin.kt (ai.enode.tracking.plugins, registered in MainActivity.java) now implements the whole TS contract. Files (all in app/src/main/java/ai/enode/tracking/plugins/):

  • Camera2VideoRecorder.kt — Camera2 + MediaCodec + Mp4Muxer, 30/60 fps, first encoded frame's presentationTimeUs → wall-clock via clockOffsetNanos. Ported 1:1 from the POC EXCEPT preview: instead of the POC's native ViewfinderView (hole-punch), it taps a downscaled (~VGA) ImageReader (YUV_420_888 → NV21 → JPEG → base64) and emits previewFrame events — STREAM mode, matching the iOS default + the shipped overlay. The encoder still gets camera frames directly, so the tap never touches recording/sync.
  • VideoTrimmer.kt — Media3 Transformer + InAppMp4Muxer, DateTimeOriginal re-stamp. Trim ±buffer comes from JS (contextBufferMs, default 2000), not the POC's hardcoded 1000. NOTE: output is moov-at-end (not faststart), so the WebView <video> stalls ~60s over the local file server — fix that at the PLAYBACK layer (read the file into a blob), NOT by swapping the muxer (a custom streamable Mp4Muxer.Factory broke the Transformer export).
  • VideoConstants.kt, VideoFileName.kt, AppLogger.kt, VideoTime.kt.
  • Plugin + file/upload methods (getVideoPath/uploadVideo/move/list/delete).

Deviations from the POC: (1) STREAM preview (above); (2) java.time removed — the app is minSdk 24 (POC was 28), so clockOffsetNanos uses System.currentTimeMillis() and the ISO-8601 metadata string uses SimpleDateFormat (VideoTime.kt), avoiding core-library desugaring; (3) media3 1.9.1 deps added to app/build.gradle, CAMERA/RECORD_AUDIO + camera feature in the manifest. Android key is still DateTimeOriginal (MdtaMetadataEntry).

Verified: :app:compileDebugKotlin + :app:assembleDebug BUILD SUCCESSFUL under JDK 21 (Android Studio JBR). Two deprecation warnings (Mp4Muxer.Builder, ExportResult.durationMs) match the POC — harmless. NOT device-tested: camera open, the ImageReader→JPEG preview (color/stride/orientation — preview is emitted unrotated; the saved file is corrected via Mp4OrientationData), and the record→trim→upload→review loop all need a physical Android device.

3. Recording UI​

Entry: camera button bottom‑right of the tracking view (next to insights), ExerciseDetailView.tsx. Opens video-recording-overlay.tsx.

  • Shared rep‑feedback pieces (rep-feedback-shared.tsx) — metric cards, REP/TIMEOUT/Threshold stats row, sensor pill, set‑end/timeout logic — used by both this overlay and PendingRepsOverlay (single origin). The auto rep overlay is suppressed while the video overlay is open.
  • Lifecycle hook use-video-recording.ts — permission → setup → record on open → stop+trim on ✓ / cancel on discard. Accept always autocommits the set's reps and stamps videoID/firstFrameDate in one completeActiveSet (its applySessionChange is controlled — can't chain two calls).
  • Background trim — BACKGROUND_VIDEO_TRIM flag (reversible) in the hook. See §9 "Saving a set does not wait for the trim".
  • Preview strategy — PREVIEW_MODE flag (reversible) in the overlay:
    • "stream" (default): native records full‑fps and emits ~24 Hz downscaled JPEG frames the web paints into a normal <img> (imperative img.src, no React churn). Overlay is an ordinary modal — no hole‑punch, no clip‑path, no WebView transparency → no compositing glitches. Decoupled from recording, so sync is unaffected.
    • "holepunch": native preview behind a transparent WebView, shown through a clip‑path hole in #app-shell. Full‑fidelity but fragile; kept as fallback.
  • Trim window: ±2 s (DEFAULT_VIDEO_CONTEXT_BUFFER_MS). Center‑square is a display concern (CSS object-cover), never baked into the file.

4. Review (technique tab)​

technique-tab.tsx + video-review-player.tsx. Square <video> synced to the shared crosshairTime: videoTime = baseOffset + crosshairTime, with baseOffset = (rep.created − concentricMs − set.firstFrameDate)/1000. Chart scrub seeks the (paused) video; playing the video drives the crosshair.

getVideoSrc(videoId) (@enode/core/video) → native getVideoPath → Capacitor.convertFileSrc. Player keyed by videoID (remounts per video).

Full-screen set review — stepping between sets​

The fullscreen control opens set-review.tsx → SetReviewOverlay (the whole set, one video + the scrub timeline + best reps). Its timeline picks a mode by pointer type, not screen size: (pointer: coarse) — phone and pad — gets the filmstrip (SetTimeline mode="scroll", centred playhead, the strip scrolls under the finger, 15 s of window on a phone / 30 s on wider screens); mouse and trackpad keep the absolute track. The prev/next-set chevrons live in the timeline's leading / trailing slots so they align with the scrub band rather than the video footer.

Landscape (lg+) follows the app's frosted-header model: the header floats over the view (lg:absolute, clamped and centred on the content width) and the best-reps rail reserves HEADER_OVERLAY_PT inside its own scroll box, so the charts travel up behind the bar. The square video is height-driven, so the rail takes all the width it leaves.

Stepping between sets must not remount the overlay: it is portaled and fixed inset-0, so a remount tears the portal down for a frame and the app flashes through behind it. Instead the mounted overlay is re-pointed at the new set — technique-tab.tsx renders <SetReview> unkeyed, SetReviewOverlay takes a sourceKey and resets the playhead/duration when it changes, and set-review.tsx holds the current src on screen until the next one resolves. The step is a startTransition, so the (covered) technique tab's own chart rebuild can't block the press.

The Insights header is a frosted overlay over a full‑height scroll (its measured height is published as --insights-top); the review video + value header ride up in a sticky block (z‑50, above the header z‑30) and pin over the header on scroll, charts blurring beneath. Flags: STICKY_REVIEW_VIDEO.

5. Data model & upload​

  • WorkoutSet.videoID?: UUID + firstFrameDate?: number (epoch ms — sync anchor and wire value; renamed from videoStartEpochMs 2026‑06‑17).
  • Wire: SetCreateDto.video?: VideoCreateDto = { id, firstFrameDate } (mirrors iOS). Deprecated videoID is sent as null. Built by api/mappers/sessions.ts. Backend uses millisecondsSince1970 for all dates → firstFrameDate is epoch ms, no conversion. Backend reads it directly (no longer reliant on parsing the MP4, though the metadata is still written).
  • Upload outbox packages/core/src/offline/video-queue.ts — IndexedDB outbox (idempotent on videoId), enqueueVideos / flushVideos / videoQueueHealth (7‑day retention warn). Bytes go via a presigned Cloudflare URL: getVideoUploadUrl (api/videos.ts) → native uploadVideo (bare PUT, no Content-Type — the URL is pre‑signed) → moveVideoToUploaded. Deps wired in video/upload.ts (getVideoUploadDeps()), drained alongside the JSON upload queue via use-upload-drain.ts / orchestrator.ts / use-finish-training.ts. File lifecycle: active_session/ → pending_upload/ → uploaded (native).

6. Camera config & permissions​

1080p @ 60→30 fallback, auto‑exposure (manual ISO/shutter deferred — setExposure is a no‑op stub), front/back switch, orientation follows the UI. iOS Info.plist: NSCameraUsageDescription + NSMicrophoneUsageDescription.

7. Build / run​

Web changes: npm run build tracking → cd apps/tracking && npx cap sync ios → Xcode ⌘R. Native (Swift) changes also need ⌘R. Verify Swift compiles headless: xcodebuild build -project apps/tracking/ios/App/App.xcodeproj -scheme App -sdk iphonesimulator -destination 'generic/platform=iOS Simulator' CODE_SIGNING_ALLOWED=NO. Camera needs a physical device (no simulator camera). See reference_ios_native_build_loop in memory.

8. Open tasks — plan for next agents​

Status: iOS record→trim→upload→review device-tested working; Android record + orientation + live preview device-tested working; Android replay fix (blob playback, below) implemented in code, not yet device-verified. Tasks are ordered by priority. Each lists where, how, acceptance, and gotchas.

P0 — Android replay loads ~60s (moov-at-end) — FIXED IN CODE, verify on device​

  • Symptom (was confirmed on device): the trimmed clip plays, but the review <video> sits black "loading" ~60s first.
  • Root cause: Android's Media3 trim writes a moov-at-end MP4. Served over Capacitor's local file server (convertFileSrc), the WebView pulls the whole file before it can start. iOS avoids this via faststart (shouldOptimizeForNetworkUse).
  • Do NOT fix it by swapping the trim muxer — a custom streamable Mp4Muxer.Factory (setAttemptStreamableOutputEnabled(true)) was tried and broke the Transformer export (no output file → no videoID → no replay card). It was reverted. Recording/trim must stay on the stock InAppMp4Muxer.Factory path.
  • Fix (implemented, at the PLAYBACK layer): read the (short) clip into a Blob and play URL.createObjectURL(blob) — fully in-memory, plays instantly regardless of moov position, server-independent.
    • Native EnodeVideo.readVideoBase64({videoId}) → { data: string | null } (EnodeVideoPlugin.kt; Android-only, locates the file like getVideoPath). Typed optional/native-only in capacitor-video.ts.
    • getVideoSrc (packages/core/src/video/index.ts) branches on Capacitor.getPlatform(): Android reads base64 → Blob → object URL; iOS keeps convertFileSrc (already fast via faststart).
    • Both consumers (video-review-player.tsx, set-review.tsx) release the blob: URL via releaseVideoSrc once nothing renders it. set-review.tsx releases on the SWAP, not when the videoID changes: stepping to the prev/next set keeps the current recording on screen until the next one has resolved (§4, "Full-screen set review"), then frees the one it replaced.
  • Remaining — device verification: Android review opens in <1s; iOS unchanged; no memory leak over repeated opens (watch WebView memory while opening/closing a review repeatedly).
  • Known tradeoff: the clip crosses the JS bridge as base64, so a long set's clip costs a transient ~1.3× file-size string. Acceptable for trimmed clips; if it ever hurts, the next lever is streaming the file into the Blob in chunks, not the muxer.

P0 — Android upload → review, end-to-end on backdev​

  • Recording + local replay work; the actual Cloudflare PUT and post-upload flow have not been confirmed on Android (iOS is confirmed).
  • Verify: finish a session → GET /videos/upload_url/{id} → native uploadVideo (bare PUT) returns 200 → outbox entry clears → file deleted (see cleanup invariant, §5 / ADR 0002). Watch EnodeVideoPlugin/chromium logcat.
  • Acceptance: the pendingVideos IndexedDB entry disappears after a real upload on Android; the pill (PendingUploadsIndicator) reflects it.

P1 — Post-upload review (server download path)​

  • Gap created by the cleanup decision: after a successful upload the local file is deleted, so reviewing an already-uploaded video (History / past sessions) is blank — getVideoPath returns null.
  • Build: GET /videos/download_url/{id} binding (mirror getVideoUploadUrl in api/videos.ts), then in getVideoSrc: if no local file, fetch the download URL → (reuse the P0 Blob path) → play. iOS ENVideoService has the .uploaded download concept to mirror.
  • Acceptance: a set whose video already uploaded still replays from History.

P1 — Privilege / setting gate for upload (deferred by decision)​

  • iOS gates upload on writeCloudStorage privilege and an automaticVideoUpload user setting; the web outbox currently always uploads. See memory project_video-upload-privilege-gate.
  • Build: inject a shouldUploadVideos() predicate into getVideoUploadDeps() / the flush + finish-time best-effort flush. Needs a privilege accessor (token-claims) + a user-setting toggle.

P2 — Natural-landscape tablet orientation​

  • The upright-rotation formula in Camera2VideoRecorder.uprightRotationDegrees() assumes a portrait-natural device. Verify on a tablet (natural-landscape); if off by 90°, switch to a natural-orientation-aware mapping.

P2 — Preview transport (only if it janks)​

  • Today: base64 JPEG frames over the Capacitor bridge; the per-frame rotation re-encode was removed (frames go sensor-native + a rotation field; the web CSS-rotates the <img>). User reported perf "ok" — likely closeable.
  • If landscape ever judders, the lever is the transport, not rotation. Most reliable option = a native preview view (TextureView/SurfaceView hole-punch) on Android (the POC's ViewfinderView path, zero encode/bridge); MJPEG-over- localhost is a middle option but adds an HTTP server + mixed-content/cleartext config. See the transport review in the project memory / git history.

P2 — Manual exposure (ISO/shutter)​

  • setExposure is a no-op stub on both platforms. iOS setExposureModeCustom; Android CONTROL_AE_MODE_OFF + SENSOR_EXPOSURE_TIME/SENSOR_SENSITIVITY.

P3 — Local-file reconciliation / pruning​

  • Native listVideos/deleteVideo exist but nothing prunes orphaned active_session/ files or reconciles the outbox against disk. Add a launch-time sweep (drop files with no set + no outbox entry).

Build/run reminders for the next agent​

  • Web→native: npm run build tracking → cd apps/tracking && npx cap sync ios (and npx cap sync android). Kotlin-only changes need only an Android Studio Run; TS changes need the build+sync. Backend: NEXT_PUBLIC_API_SERVER=backdev.
  • Android Gradle needs JDK 21 — macOS default java is 17, so set JAVA_HOME=/Applications/Android Studio.app/Contents/jbr/Contents/Home for CLI ./gradlew :app:compileDebugKotlin / assembleDebug (Android Studio uses the JBR automatically).

9. Key decisions / gotchas​

Saving a set does not wait for the trim​

Tapping ✓ commits the set and closes the overlay as soon as the recorder has finalized its file; the trim re‑encode then runs in the background. Behind BACKGROUND_VIDEO_TRIM in use-video-recording.ts (set it to false to restore the awaited path).

Why: the trim cost scales with clip length, so the old awaited save blocked the UI for seconds on a normal set. Measured on device:

StageiPad Air M4 (1080p60)Galaxy Tab A9+ (720p30)
stopRecording (writer finish)~8 ms~210 ms
trim re‑encode~7% of clip duration~3.3% of clip duration

iOS re‑encodes via AVAssetExportSession; Android transmuxes via Media3 Transformer, hence the ~2× difference. A 60 s set cost ~4 s on iOS and ~2 s on Android, all of it blocking. Now it costs ~8 ms.

What makes it safe: everything the set persists — videoID and firstFrameDate — is known before the trim runs. The id is generated in JS and passed to trimAndSave({ videoId }); the anchor comes from videoStartEpochMs() (packages/core/src/video/index.ts), which duplicates the native trim‑window math. Keep that function in lockstep with both native trimmers — a divergence silently desynchronises the rep overlay in review. Only the file itself lags, and a set whose trim never lands keeps a videoID with no file, which both review surfaces already render as "video unavailable".

Consequences worth knowing:

  • Raw recordings are raw_<uuid>.mp4, one per recording. A fixed filename would let the next set's recording overwrite the file an in‑flight trim is still reading. The raw_ prefix is also excluded from locateVideo / listVideos, which would otherwise read the trailing _<uuid> as a video id.
  • Leftover raw files are swept at plugin load() on both platforms — the trim deletes its own input, so anything remaining is from a process that died mid‑export, and no trim outlives its process.
  • Android runs the trim on Dispatchers.Main (Media3 requires a Looper thread to start Transformer), so extractVideoDuration is explicitly pushed to Dispatchers.IO — under a background trim it would otherwise stutter a UI the user is actively touching.

Other​

  • Native recording was chosen over web getUserMedia+MediaRecorder for the byte‑accurate first‑frame timestamp; the stream preview keeps that while giving a glitch‑free DOM <img> preview (camera is single‑consumer, so you can't run web preview + native record at once — streaming native frames is the bridge).
  • iOS safe‑area: body already pads env(safe-area-inset-top) globally (design-system.css). In‑flow content must not re‑add it (doubles the gap); only fixed/portaled surfaces self‑inset.
  • sticky top-0 is clamped by its containing block — a scroll‑container padding-top pushes the stick‑point down; use a spacer element instead.