PHX Continuum: End-to-End Scene Intelligence for Hour-Scale Narrative Video
Abstract
We present PHX Continuum, a contrastive reasoning transformer for scene intelligence that ingests a full-length scene end to end and produces a complete chapter timeline together with scene intelligence: fine-grained segment classes, transitions and per-chapter key frames. Long-form studio scene content is difficult for existing video understanding methods: scenes run to an hour or more, subjects are often occluded or framed in extreme close-up, editorial cuts rarely coincide with changes of segment, and many classes are visually similar. Continuum is trained on a large proprietary corpus annotated by domain experts. Under a strict blind protocol, Continuum reaches 0.942 single-frame accuracy, against 0.29–0.50 for general-purpose vision models and 0.69–0.72 for frontier vision APIs at more than 10x the per-frame cost. The current release produces timelines with 0.77 timeline accuracy and boundary F1 of 0.67 at ±5 s and 0.73 at ±10 s on held-out scenes. All inference runs on our own GPU infrastructure without third-party AI services.
Introduction
Studios hold large back catalogues of long-form scene content whose internal structure is undocumented. Chapter markers, preview generation, search, compliance review and ad placement all require knowing what happens when, at a resolution of seconds, across recordings that commonly run for most of an hour. Manual annotation costs roughly real time per scene for a trained annotator, and quality varies between annotators.
Temporal action segmentation is the established research framing of this problem [1]. The public benchmarks that drive the field, such as Breakfast [2], 50Salads [3] and GTEA [4], consist of short, scripted, mostly static-camera activities. Long-form narrative video differs in ways that, in our experience, dominate error:
Hour-scale context. A scene contains many segments whose durations range from a few seconds to many minutes. Short-window methods see too little context; full-sequence methods must avoid fragmenting the timeline.
Occlusion and close-ups. Production framing favours tight close-ups that exclude much of the visual evidence that defines a class. Many moments are ambiguous in isolation, to human annotators as well.
Cuts that do not coincide with segment changes. Editorial cuts are frequent and mostly change camera angle within a segment, while many segment changes occur within a continuous shot. Shot-boundary detection is therefore neither necessary nor sufficient.
Visually similar classes. The taxonomy contains classes that differ only in subtle spatial configuration and look nearly identical from some camera angles.
Domain shift. General-purpose vision models are trained largely on content that excludes this domain, and their zero-shot behaviour on it is poor (Section 6).
Related Work
Temporal action segmentation. Early approaches combined frame classifiers with generative or grammar-based decoding [5,6]; later neural approaches substantially reduced over-segmentation and extended temporal context. Ding et al. [1] survey the area and argue that segmental metrics, not frame accuracy, best reflect usable output, a view we adopt in our evaluation.
Boundary detection and long-form video. Generic event boundary detection [7] treats boundaries as taxonomy-free change points perceived by humans, and work on long-form video understanding [8] highlights the difficulty of reasoning over extended durations. Continuum addresses both problems within one system, in a domain where public benchmarks and pretrained models offer little direct coverage.
PHX Continuum
Continuum takes an entire scene at full duration as input, with no trimming, truncation or manual pre-segmentation, and returns a complete chapter timeline (start, end and segment class per chapter), the transitions between chapters, and a representative key frame for each chapter. The output is a structured timeline suitable for chapter markers, preview selection, catalogue search and downstream analytics. Figure 1 shows the interface.
Continuum is delivered and evaluated as a single sealed system: its interface is the scene in and the timeline out, and its internal design is not described in this report.
Data and Annotation
Continuum is trained on a large proprietary corpus of studio scene content annotated by domain experts against a fine-grained segment taxonomy. Generic examples of segment classes include dialogue and transition. Each class has a written definition with explicit decision rules for visually similar classes, and every item is reviewed by a human before it enters training. Data is partitioned by scene, so no scene contributes to both training and evaluation.
Evaluation Protocol
Strict blind protocol. Evaluation uses held-out scenes that are disjoint from all training scenes and are never used for training, model selection or tuning. Ground truth is authored by experts under the same rules as the training annotation. Results on this set are computed once per release; no iteration is performed against it.
Metrics. Single-frame accuracy is top-1 accuracy over sampled blind frames. Timeline accuracy is the fraction of scene duration whose predicted class matches the expert timeline. Boundary F1 at tolerance τ matches each predicted boundary to at most one unmatched ground-truth boundary within ±τ seconds and takes the harmonic mean of precision and recall, following boundary evaluation practice in [7]. We report τ = 5 s and 10 s. Boundary F1 penalises fragmentation and missed changes, which timeline accuracy does not, so the two are reported together [1].
Results
Recognition
| System | Blind single-frame accuracy |
|---|---|
| General-purpose vision models, zero-shot | 0.29 – 0.50 |
| Frontier vision APIs (more than 10x the per-frame cost)† | 0.69 – 0.72 |
| PHX Continuum | 0.942 |
Table 1. Single-frame accuracy under the strict blind protocol.
† Indicative pilot: a different sample of 100 blind frames, graded by a human reviewer, so not directly comparable to the other rows. Frontier API pricing is more than 10x the per-frame cost of our own GPU inference.
Chapter timelines
| System (10 blind scenes) | Timeline acc. | Boundary F1 @±5 s | Boundary F1 @±10 s |
|---|---|---|---|
| PHX Continuum, current release | 0.77 | 0.67 | 0.73 |
Table 2. Timeline quality under the strict blind protocol.
The gap between single-frame accuracy and timeline accuracy is expected. Sampled blind frames are drawn where the defining evidence is visible, while a full timeline must also label close-up, occluded and transitional moments, and every moment near a misplaced boundary counts as an error. The rise in boundary F1 from ±5 s to ±10 s shows that a meaningful share of boundary errors are small placement offsets rather than missed or spurious changes.
Error Analysis
Errors concentrate in a small number of situations rather than spreading across the taxonomy: classes that differ mainly in orientation and are hard to separate from some camera angles; extended close-ups in which the cues that separate related classes are out of frame; gradual changes between an introductory segment and the segment it leads into, which are also the main source of disagreement between human annotators; and rare classes, which are more often absorbed into a visually similar, more frequent neighbour. Scenes with more than two participants are handled as a distinct case; the results above concern two-participant scenes.
Conclusion
PHX Continuum turns full-length studio scenes into structured chapter timelines and scene intelligence. Trained on expert-annotated domain data, Continuum reaches 0.942 blind single-frame accuracy, well beyond general-purpose and frontier commercial vision systems, and produces coherent timelines with 0.73 boundary F1 at ±10 s in the current release. Work continues on boundary placement and on the small set of confusable situations identified above.
References
- G. Ding, F. Sener, A. Yao. Temporal action segmentation: an analysis of modern techniques. IEEE TPAMI, 2023.
- H. Kuehne, A. Arslan, T. Serre. The language of actions: recovering the syntax and semantics of goal-directed human activities. CVPR, 2014.
- S. Stein, S. J. McKenna. Combining embedded accelerometers with computer vision for recognizing food preparation activities. UbiComp, 2013.
- A. Fathi, X. Ren, J. M. Rehg. Learning to recognize objects in egocentric activities. CVPR, 2011.
- H. Kuehne, J. Gall, T. Serre. An end-to-end generative framework for video segmentation and recognition. WACV, 2016.
- A. Richard, J. Gall. Temporal action detection using a statistical language model. CVPR, 2016.
- M. Z. Shou, S. W. Lei, W. Wang, D. Ghadiyaram, M. Feiszli. Generic event boundary detection: a benchmark for event segmentation. ICCV, 2021.
- C.-Y. Wu, P. Krähenbühl. Towards long-form video understanding. CVPR, 2021.