Skip to main content

Set video recording

Per‑set video capture, frame‑accurate sync to sensor rep data, technique‑tab review, and deferred upload. Branch: video-recording. Status (2026‑06‑17): iOS recording + review device‑tested and working; upload pipeline built; Android not started. This doc is the source of truth + handoff for all video work.

1. The core problem this feature solves​

Frame‑accurate video ↔ sensor‑rep sync. Each saved video stores a single wall‑clock timestamp of its first frame; every rep already carries a wall‑clock created and concentric‑relative raw‑data time t. Sync is subtraction:

videoTime(rep, t) = (rep.created − concentricDurationMs − firstFrameDate)/1000 + t

firstFrameDate is the trimmed video's first‑frame wall‑clock (epoch ms). It is the one anchor everything rests on — persisted on the set, sent to the backend, and written into the MP4 metadata.

Verified consistent: WorkoutRep.created == event.timeStamp (reps.ts), and the duration metric == event.t — the exact values the trimmer uses.

2. Architecture — EnodeVideo Capacitor plugin​

Native‑only (iOS AVFoundation / Android Camera2). Mirrors EnodeBle: a TS bridge + native implementations; all product logic stays in shared TS. Gate every call on isVideoSupported() (@enode/core/video) — web rejects.

  • TS bridge + types: packages/core/src/video/capacitor-video.ts
  • Barrel + isVideoSupported() + DEFAULT_VIDEO_CONTEXT_BUFFER_MS = 2000: packages/core/src/video/index.ts
  • Upload deps wiring: packages/core/src/video/upload.ts

Bridge methods: checkCapabilities, requestVideoPermissions, setupPreview ({stream?}), updatePreviewRect, switchCamera, setExposure (no‑op stub), setPreviewStreaming, ackPreviewFrame, startRecording, stopRecording, cancelRecording, trimAndSave ({contextBufferMs}), teardownPreview, getVideoPath, uploadVideo, moveVideoToUploaded, listVideos, deleteVideo, saveVideoToGallery. Events: previewReady, recordingStarted, firstFrameEncoded, recordingStopping, recordingCompleted, previewFrame (stream mode).

iOS (apps/tracking/ios/App/App/Plugins/)​

  • VideoRecorder.swift — AVCaptureSession + AVCaptureVideoDataOutput → AVAssetWriter/pixel‑buffer adaptor (push model). Skips the first 30 warm‑up frames (iOS drops early frames → t=0 offset; the skip fixes it consistently), anchors writerStartTime/firstFrameEpochMs on the first kept frame. 1080p 60→30 selection; orientation follows the UI interface orientation; auto‑exposure. Stream‑mode tap: downscales (≤720px) + JPEG (q0.5), throttled to ~24 Hz (every 2nd frame at 60 fps ≈ 30 Hz), base64 → onPreviewFrame.
  • Preview backpressure (iOS + Android): at most one preview frame is in flight. The web acks each frame with ackPreviewFrame once the <img> has loaded it (also for frames it skips, e.g. while saving); native sends the next frame only after the ack, with a 1 s failsafe for a lost ack. The encode runs on its own queue, off the writer's sessionQueue. Without backpressure, a device that falls behind (thermal throttling, a concurrent trim export) queued every frame: the preview lagged further and further, rep feedback was skipped on the starved WebView main thread, and the growing backlog got the app killed by iOS after ~30 min of video sets on an iPad. Android applies the same slot in Camera2VideoRecorder.onPreviewImage, on top of its existing 20 Hz throttle.
  • Ack on load, never via img.decode(): decode() starts an asynchronous decode of the frame, and WebKit paints an <img> whose frame is still being decoded as nothing (it won't decode one frame on two threads — BitmapImageSource::nativeImageAtIndexCacheIfNeeded). Every paint that hit that window showed the card without a picture for one frame: the preview flickered (roughly once a second on an iPad). Acking on the load event leaves the decode to paint time, where WebKit does it synchronously.
  • Only the open overlay acks: every station mounts its own useVideoRecording, and every instance receives every previewFrame. Closed instances ignore the frame; an ack from them would release native's next frame before the open overlay has loaded the current one, which switches the backpressure off. The per-frame decision lives in preview-frame-gate.ts.
  • Preview warm-up (web, both platforms): the camera's first frames after a start or a front/back switch jump in brightness while exposure and white balance settle. The web doesn't paint the first 500 ms of frames after recording starts (the same span the iOS recorder drops from the file), and the overlay keeps its "Starting camera…" cover up until the first frame after that has loaded.
  • VideoTrimmer.swift — crops to [firstRep.created − concentricMs − buffer, lastRep.created + buffer], re‑encodes (frame‑accurate start), writes commonIdentifierCreationDate (ISO‑8601 UTC) = creationTimestampMs = max(firstFrameEpochMs, desiredStart).
  • EnodeVideoPlugin.swift — CAPBridgedPlugin wrapper, preview placement (hole‑punch mode), stream wiring, permissions, upload/file methods.
  • Registration (Cap 8 gotcha): bridge?.registerPluginInstance(EnodeVideoPlugin()) in AppViewController.swift capacitorDidLoad() — without it JS gets "not implemented on ios". Also force‑ref in AppDelegate + 4 project.pbxproj entries per Swift file.

Android — FULL PIPELINE PORTED (not device-tested)​

EnodeVideoPlugin.kt (ai.enode.tracking.plugins, registered in MainActivity.java) now implements the whole TS contract. Files (all in app/src/main/java/ai/enode/tracking/plugins/):

  • Camera2VideoRecorder.kt — Camera2 + MediaCodec + Mp4Muxer, 30/60 fps, first encoded frame's presentationTimeUs → wall-clock via clockOffsetNanos. Ported 1:1 from the POC EXCEPT preview: instead of the POC's native ViewfinderView (hole-punch), it taps a downscaled (~VGA) ImageReader (YUV_420_888 → NV21 → JPEG → base64) and emits previewFrame events — STREAM mode, matching the iOS default + the shipped overlay. The encoder still gets camera frames directly, so the tap never touches recording/sync.
  • VideoTrimmer.kt — Media3 Transformer + InAppMp4Muxer, DateTimeOriginal re-stamp. Trim ±buffer comes from JS (contextBufferMs, default 2000), not the POC's hardcoded 1000. NOTE: output is moov-at-end (not faststart), so the WebView <video> stalls ~60s over the local file server — fix that at the PLAYBACK layer (read the file into a blob), NOT by swapping the muxer (a custom streamable Mp4Muxer.Factory broke the Transformer export).
  • VideoConstants.kt, VideoFileName.kt, AppLogger.kt, VideoTime.kt.
  • Plugin + file/upload methods (getVideoPath/uploadVideo/move/list/delete).

Deviations from the POC: (1) STREAM preview (above); (2) java.time removed — the app is minSdk 24 (POC was 28), so clockOffsetNanos uses System.currentTimeMillis() and the ISO-8601 metadata string uses SimpleDateFormat (VideoTime.kt), avoiding core-library desugaring; (3) media3 1.9.1 deps added to app/build.gradle, CAMERA/RECORD_AUDIO + camera feature in the manifest. Android key is still DateTimeOriginal (MdtaMetadataEntry).

Verified: :app:compileDebugKotlin + :app:assembleDebug BUILD SUCCESSFUL under JDK 21 (Android Studio JBR). Two deprecation warnings (Mp4Muxer.Builder, ExportResult.durationMs) match the POC — harmless. NOT device-tested: camera open, the ImageReader→JPEG preview (color/stride/orientation — preview is emitted unrotated; the saved file is corrected via Mp4OrientationData), and the record→trim→upload→review loop all need a physical Android device.

3. Recording UI​

Entry: camera button bottom‑right of the tracking view (next to insights), ExerciseDetailView.tsx. Opens video-recording-overlay.tsx.

  • Shared rep‑feedback pieces (rep-feedback-shared.tsx) — metric cards, REP/TIMEOUT/Threshold stats row, sensor pill, set‑end/timeout logic — used by both this overlay and PendingRepsOverlay (single origin). The auto rep overlay is suppressed while the video overlay is open.
  • Lifecycle hook use-video-recording.ts — permission → setup → record on open → stop+trim on ✓ / cancel on discard. Accept always autocommits the set's reps and stamps videoID/firstFrameDate in one completeActiveSet (its applySessionChange is controlled — can't chain two calls).
  • Background trim — BACKGROUND_VIDEO_TRIM flag (reversible) in the hook. See §9 "Saving a set does not wait for the trim".
  • Preview strategy — PREVIEW_MODE flag (reversible) in the overlay:
    • "stream" (default): native records full‑fps and emits ~24 Hz downscaled JPEG frames the web paints into a normal <img> (imperative img.src, no React churn). Overlay is an ordinary modal — no hole‑punch, no clip‑path, no WebView transparency → no compositing glitches. Decoupled from recording, so sync is unaffected.
    • "holepunch": native preview behind a transparent WebView, shown through a clip‑path hole in #app-shell. Full‑fidelity but fragile; kept as fallback.
  • Trim window: ±2 s (DEFAULT_VIDEO_CONTEXT_BUFFER_MS). Center‑square is a display concern (CSS object-cover), never baked into the file.

4. Review (technique tab)​

technique-tab.tsx + video-review-player.tsx. Square <video> synced to the shared crosshairTime: videoTime = baseOffset + crosshairTime, with baseOffset = (rep.created − concentricMs − set.firstFrameDate)/1000. Chart scrub seeks the (paused) video; playing the video drives the crosshair.

getVideoSrc(videoId) (@enode/core/video) → native getVideoPath → Capacitor.convertFileSrc. Player keyed by videoID (remounts per video).

Full-screen set review — stepping between sets​

The fullscreen control opens set-review.tsx → SetReviewOverlay (the whole set, one video + the scrub timeline + best reps). Its timeline picks a mode by pointer type, not screen size: (pointer: coarse) — phone and pad — gets the filmstrip (SetTimeline mode="scroll", centred playhead, the strip scrolls under the finger, 15 s of window on a phone / 30 s on wider screens); mouse and trackpad keep the absolute track. The prev/next-set chevrons live in the timeline's leading / trailing slots so they align with the scrub band rather than the video footer.

Landscape (lg+) follows the app's frosted-header model: the header floats over the view (lg:absolute, clamped and centred on the content width) and the best-reps rail reserves HEADER_OVERLAY_PT inside its own scroll box, so the charts travel up behind the bar. The square video is height-driven, so the rail takes all the width it leaves.

Stepping between sets must not remount the overlay: it is portaled and fixed inset-0, so a remount tears the portal down for a frame and the app flashes through behind it. Instead the mounted overlay is re-pointed at the new set — technique-tab.tsx renders <SetReview> unkeyed, SetReviewOverlay takes a sourceKey and resets the playhead/duration when it changes, and set-review.tsx holds the current src on screen until the next one resolves. The step is a startTransition, so the (covered) technique tab's own chart rebuild can't block the press.

The Insights header is a frosted overlay over a full‑height scroll (its measured height is published as --insights-top); the review video + value header ride up in a sticky block (z‑50, above the header z‑30) and pin over the header on scroll, charts blurring beneath. Flags: STICKY_REVIEW_VIDEO.

5. Data model & upload​

  • WorkoutSet.videoID?: UUID + firstFrameDate?: number (epoch ms — sync anchor and wire value; renamed from videoStartEpochMs 2026‑06‑17).
  • Wire: SetCreateDto.video?: VideoCreateDto = { id, firstFrameDate } (mirrors iOS). Deprecated videoID is sent as null. Built by api/mappers/sessions.ts. Backend uses millisecondsSince1970 for all dates → firstFrameDate is epoch ms, no conversion. Backend reads it directly (no longer reliant on parsing the MP4, though the metadata is still written).
  • Upload outbox packages/core/src/offline/video-queue.ts — IndexedDB outbox (idempotent on videoId), enqueueVideos / flushVideos / videoQueueHealth (7‑day retention warn). Bytes go via a presigned Cloudflare URL: getVideoUploadUrl (api/videos.ts) → native uploadVideo (bare PUT, no Content-Type — the URL is pre‑signed) → moveVideoToUploaded. Deps wired in video/upload.ts (getVideoUploadDeps()), drained alongside the JSON upload queue via use-upload-drain.ts / orchestrator.ts / use-finish-training.ts. File lifecycle: active_session/ → pending_upload/ → uploaded (native).

6. Camera config & permissions​

1080p @ 60→30 fallback, auto‑exposure (manual ISO/shutter deferred — setExposure is a no‑op stub), front/back switch, orientation follows the UI. iOS Info.plist: NSCameraUsageDescription + NSMicrophoneUsageDescription.

7. Build / run​

Web changes: npm run build tracking → cd apps/tracking && npx cap sync ios → Xcode ⌘R. Native (Swift) changes also need ⌘R. Verify Swift compiles headless: xcodebuild build -project apps/tracking/ios/App/App.xcodeproj -scheme App -sdk iphonesimulator -destination 'generic/platform=iOS Simulator' CODE_SIGNING_ALLOWED=NO. Camera needs a physical device (no simulator camera). See reference_ios_native_build_loop in memory.

8. Open tasks — plan for next agents​

Status: iOS record→trim→upload→review device-tested working; Android record + orientation + live preview device-tested working; Android replay fix (blob playback, below) implemented in code, not yet device-verified. Tasks are ordered by priority. Each lists where, how, acceptance, and gotchas.

P0 — Android replay loads ~60s (moov-at-end) — FIXED IN CODE, verify on device​

  • Symptom (was confirmed on device): the trimmed clip plays, but the review <video> sits black "loading" ~60s first.
  • Root cause: Android's Media3 trim writes a moov-at-end MP4. Served over Capacitor's local file server (convertFileSrc), the WebView pulls the whole file before it can start. iOS avoids this via faststart (shouldOptimizeForNetworkUse).
  • Do NOT fix it by swapping the trim muxer — a custom streamable Mp4Muxer.Factory (setAttemptStreamableOutputEnabled(true)) was tried and broke the Transformer export (no output file → no videoID → no replay card). It was reverted. Recording/trim must stay on the stock InAppMp4Muxer.Factory path.
  • Fix (implemented, at the PLAYBACK layer): read the (short) clip into a Blob and play URL.createObjectURL(blob) — fully in-memory, plays instantly regardless of moov position, server-independent.
    • Native EnodeVideo.readVideoBase64({videoId}) → { data: string | null } (EnodeVideoPlugin.kt; Android-only, locates the file like getVideoPath). Typed optional/native-only in capacitor-video.ts.
    • getVideoSrc (packages/core/src/video/index.ts) branches on Capacitor.getPlatform(): Android reads base64 → Blob → object URL; iOS keeps convertFileSrc (already fast via faststart).
    • Both consumers (video-review-player.tsx, set-review.tsx) release the blob: URL via releaseVideoSrc once nothing renders it. set-review.tsx releases on the SWAP, not when the videoID changes: stepping to the prev/next set keeps the current recording on screen until the next one has resolved (§4, "Full-screen set review"), then frees the one it replaced.
  • Remaining — device verification: Android review opens in <1s; iOS unchanged; no memory leak over repeated opens (watch WebView memory while opening/closing a review repeatedly).
  • Known tradeoff: the clip crosses the JS bridge as base64, so a long set's clip costs a transient ~1.3× file-size string. Acceptable for trimmed clips; if it ever hurts, the next lever is streaming the file into the Blob in chunks, not the muxer.

P0 — Android upload → review, end-to-end on backdev​

  • Recording + local replay work; the actual Cloudflare PUT and post-upload flow have not been confirmed on Android (iOS is confirmed).
  • Verify: finish a session → GET /videos/upload_url/{id} → native uploadVideo (bare PUT) returns 200 → outbox entry clears → file deleted (see cleanup invariant, §5 / ADR 0002). Watch EnodeVideoPlugin/chromium logcat.
  • Acceptance: the pendingVideos IndexedDB entry disappears after a real upload on Android; the pill (PendingUploadsIndicator) reflects it.

P1 — Post-upload review (server download path)​

  • Gap created by the cleanup decision: after a successful upload the local file is deleted, so reviewing an already-uploaded video (History / past sessions) is blank — getVideoPath returns null.
  • Build: GET /videos/download_url/{id} binding (mirror getVideoUploadUrl in api/videos.ts), then in getVideoSrc: if no local file, fetch the download URL → (reuse the P0 Blob path) → play. iOS ENVideoService has the .uploaded download concept to mirror.
  • Acceptance: a set whose video already uploaded still replays from History.

P1 — Privilege / setting gate for upload (deferred by decision)​

  • iOS gates upload on writeCloudStorage privilege and an automaticVideoUpload user setting; the web outbox always tries.
  • Live-verified 2026-09-08 (backdev, fresh Enode Pro = free licence, totalVideoStorage: 0, no writeCloudStorage): the session POST with a set video is accepted (201, the video record is created as pending); GET /videos/upload_url/{id} answers 403. A free licence therefore never blocks the workout upload — only the video bytes are refused.
  • Shipped instead of the gate: flushVideos drops an outbox entry whose upload-URL request is deterministically rejected (4xx other than 401/408/429), so the pending indicator no longer warns forever about videos the account cannot upload. The local file is kept (cleanup invariant, §5).
  • Still open: a pre-flight gate on writeCloudStorage (useHasPrivilege exists now) to skip the request or hide recording entirely, and the automaticVideoUpload user setting.

P2 — Natural-landscape tablet orientation​

  • The upright-rotation formula in Camera2VideoRecorder.uprightRotationDegrees() assumes a portrait-natural device. Verify on a tablet (natural-landscape); if off by 90°, switch to a natural-orientation-aware mapping.

P2 — Preview transport (only if it janks)​

  • Today: base64 JPEG frames over the Capacitor bridge; the per-frame rotation re-encode was removed (frames go sensor-native + a rotation field; the web CSS-rotates the <img>). User reported perf "ok" — likely closeable.
  • If landscape ever judders, the lever is the transport, not rotation. Most reliable option = a native preview view (TextureView/SurfaceView hole-punch) on Android (the POC's ViewfinderView path, zero encode/bridge); MJPEG-over- localhost is a middle option but adds an HTTP server + mixed-content/cleartext config. See the transport review in the project memory / git history.

P2 — Manual exposure (ISO/shutter)​

  • setExposure is a no-op stub on both platforms. iOS setExposureModeCustom; Android CONTROL_AE_MODE_OFF + SENSOR_EXPOSURE_TIME/SENSOR_SENSITIVITY.

P3 — Local-file reconciliation / pruning​

  • Native listVideos/deleteVideo exist but nothing prunes orphaned active_session/ files or reconciles the outbox against disk. Add a launch-time sweep (drop files with no set + no outbox entry).

Build/run reminders for the next agent​

  • Web→native: npm run build tracking → cd apps/tracking && npx cap sync ios (and npx cap sync android). Kotlin-only changes need only an Android Studio Run; TS changes need the build+sync. Backend: NEXT_PUBLIC_API_SERVER=backdev.
  • Android Gradle needs JDK 21 — macOS default java is 17, so set JAVA_HOME=/Applications/Android Studio.app/Contents/jbr/Contents/Home for CLI ./gradlew :app:compileDebugKotlin / assembleDebug (Android Studio uses the JBR automatically).

9. Key decisions / gotchas​

Saving a set does not wait for the trim​

Tapping ✓ commits the set and closes the overlay as soon as the recorder has finalized its file; the trim re‑encode then runs in the background. Behind BACKGROUND_VIDEO_TRIM in use-video-recording.ts (set it to false to restore the awaited path).

Why: the trim cost scales with clip length, so the old awaited save blocked the UI for seconds on a normal set. Measured on device:

StageiPad Air M4 (1080p60)Galaxy Tab A9+ (720p30)
stopRecording (writer finish)~8 ms~210 ms
trim re‑encode~7% of clip duration~3.3% of clip duration

iOS re‑encodes via AVAssetExportSession; Android transmuxes via Media3 Transformer, hence the ~2× difference. A 60 s set cost ~4 s on iOS and ~2 s on Android, all of it blocking. Now it costs ~8 ms.

What makes it safe: everything the set persists — videoID and firstFrameDate — is known before the trim runs. The id is generated in JS and passed to trimAndSave({ videoId }); the anchor comes from videoStartEpochMs() (packages/core/src/video/index.ts), which duplicates the native trim‑window math. Keep that function in lockstep with both native trimmers — a divergence silently desynchronises the rep overlay in review. Only the file itself lags, and a set whose trim never lands keeps a videoID with no file, which both review surfaces already render as "video unavailable".

Consequences worth knowing:

  • Raw recordings are raw_<uuid>.mp4, one per recording. A fixed filename would let the next set's recording overwrite the file an in‑flight trim is still reading. The raw_ prefix is also excluded from locateVideo / listVideos, which would otherwise read the trailing _<uuid> as a video id.
  • Trims run one at a time (iOS EnodeVideoPlugin.enqueueTrim, Android trimMutex): each trimAndSave awaits the previous one, so back‑to‑back sets don't stack several exports on top of the live recording. uploadVideo waits for the queued trims before resolving the file, and the review's "unavailable" grace window (VIDEO_TRIM_GRACE_MS, 45 s) covers a trim that queued behind others.
  • The ready signal is held while the overlay is open. A trim that finishes during the next set's recording defers notifyVideoReady until the overlay closes (use-video-recording.ts), so the always‑mounted iPad review pane doesn't decode a video underneath the live camera.
  • Leftover raw files are swept at plugin load() on both platforms — the trim deletes its own input, so anything remaining is from a process that died mid‑export, and no trim outlives its process.
  • Android runs the trim on Dispatchers.Main (Media3 requires a Looper thread to start Transformer), so extractVideoDuration is explicitly pushed to Dispatchers.IO — under a background trim it would otherwise stutter a UI the user is actively touching.

Saving to the photo library​

saveVideoToGallery({ videoId }) copies a local video into the device gallery and resolves { status: "saved" | "denied" | "notFound" }. No UI calls it yet.

  • iOS: PHPhotoLibrary with add-only authorization (NSPhotoLibraryAddUsageDescription in Info.plist); the app never reads the library.
  • Android: Android 10+ inserts into Movies/Enode via MediaStore without a permission. Android 8–9 (minSdk 26) needs WRITE_EXTERNAL_STORAGE, declared with maxSdkVersion="28" and requested on the first call; the file is copied to the public Movies folder and media-scanned.
  • Only local files: the upload outbox deletes the local copy after a confirmed upload, so an uploaded video resolves "notFound". Saving it later needs a download via getVideoDownloadUrl first — TODO: not implemented.
  • Both platforms wait for queued trims before resolving the file.

Other​

  • Native recording was chosen over web getUserMedia+MediaRecorder for the byte‑accurate first‑frame timestamp; the stream preview keeps that while giving a glitch‑free DOM <img> preview (camera is single‑consumer, so you can't run web preview + native record at once — streaming native frames is the bridge).
  • iOS safe‑area: body already pads env(safe-area-inset-top) globally (design-system.css). In‑flow content must not re‑add it (doubles the gap); only fixed/portaled surfaces self‑inset.
  • sticky top-0 is clamped by its containing block — a scroll‑container padding-top pushes the stick‑point down; use a spacer element instead.