# MUSTAN: Multi-scale Temporal Context as Attention for Robust Video Foreground Segmentation


Video foreground segmentation (VFS) is an important computer vision task wherein
one aims to segment the objects under motion from the background. Most of the
current methods are image-based, i.e., rely only on spatial cues while ignoring
motion cues. Therefore, they tend to overfit the training data and don't
generalize well to out-of-domain (OOD) distribution. To solve the above problem,
prior works exploited several cues such as optical flow, background subtraction
mask, etc. However, having a video data with annotations like optical flow is a
challenging task. In this paper, we utilize the temporal information and the
spatial cues from the video data to improve OOD performance. However, the
challenge lies in how we model the temporal information given the video data in
an interpretable way creates a very noticeable difference. We therefore devise a
strategy that integrates the temporal context of the video in the development of
VFS. Our approach give rise to deep learning architectures, namely MUSTAN1 and
MUSTAN2 and they are based on the idea of multi-scale temporal context as an
attention, i.e., aids our models to learn better representations that are
beneficial for VFS. Further, we introduce a new video dataset, namely Indoor
Surveillance Dataset (ISD) for VFS. It has multiple annotations on a frame level
such as foreground binary mask, depth map, and instance semantic annotations.
Therefore, ISD can benefit other computer vision tasks. We validate the efficacy
of our architectures and compare the performance with baselines. We demonstrate
that proposed methods significantly outperform the benchmark methods on OOD. In
addition, the performance of MUSTAN2 is significantly improved on certain video
categories on OOD data due to ISD.

[Read Full Article](https://arxiv.org/abs/2402.00918)