Towards Scalable and Interaction-Efficient Video Object Segmentation in Unconstrained Scenarios

Video Object Segmentation (VOS) is a fundamental and challenging problem in Computer Vision (CV), where the goal is to delimit the spatial and temporal presence of a collection of objects in a given video sequence at pixel level.
Several approaches tackle VOS, namely automatic, semi-automatic, and interactive, each tailored to specific use cases and requirements.
Here, semi-automatic (sVOS) and interactive (iVOS) approaches provide the necessary flexibility to segment arbitrary objects by leveraging user cues (to various degrees), balancing automation and adaptability.


However, their applicability is predominantly limited to short, pre-recorded sequences due to their inherent design and the necessary user workload (e.g., reviewing and annotation effort). This bias makes sVOS and iVOS approaches impractical for unconstrained video segmentation (or tracking) applications. We consider a video sequence unconstrained when it is not pre-recorded (e.g., live-streamed), has no fixed length, and shows unpredictable content such as appearance changes, clutter, or occlusions.
In addition, the robustness of these models is inherently constrained by their training data.
... mehr
The data distribution learned during training is unlikely to encompass all possible real-world scenarios, particularly when dealing with unusual applications or challenging conditions.


We propose through this work to extend VOS approaches to robustly track arbitrary objects in unconstrained videos via an efficient human-in-the-loop strategy, where we simultaneously focus on minimizing the associated user effort.
To reduce this user workload, we design a proactive framework that monitors its predictive uncertainty and requests user corrections (i.e., clicks) on-the-fly when the method's uncertainty is high.
Our main idea is to involve the user strategically to resolve challenging or ambiguous situations where the system lacks confidence by leveraging the user's availability already at hand in iVOS and sVOS tasks, but in an active manner.
Furthermore, we develop a diversity-driven memory management module that enables segmentation over unconstrained sequences, overcoming the limitations of previous VOS methods, tailored for short videos.

We validate each design choice across various datasets and display consistent improvement in segmentation robustness and in decreasing the user's workload. Through this research, we provide a foundation for proactive, scalable, and user-efficient human-in-the-loop video segmentation adapted for applications geared toward unconstrained videos.