EdgeTAM: On-Device Track Anything Model

De Transcrire-Wiki
Révision datée du 27 décembre 2025 à 19:46 par AdaFlorey5 (discussion | contributions) (Page créée avec « <br>On prime of Segment Anything Model (SAM), SAM 2 further extends its functionality from image to video inputs by means of a memory financial institution mechanism a... »)
(diff) ← Version précédente | Voir la version actuelle (diff) | Version suivante → (diff)
Aller à la navigation Aller à la recherche


On prime of Segment Anything Model (SAM), SAM 2 further extends its functionality from image to video inputs by means of a memory financial institution mechanism and obtains a exceptional performance compared with earlier methods, making it a basis model for video segmentation job. In this paper, we intention at making SAM 2 far more environment friendly so that it even runs on cell units while sustaining a comparable performance. Despite several works optimizing SAM for higher effectivity, we discover they are not adequate for SAM 2 because they all concentrate on compressing the picture encoder, while our benchmark reveals that the newly launched reminiscence attention blocks are additionally the latency bottleneck. Given this commentary, we suggest EdgeTAM, which leverages a novel 2D Spatial Perceiver to cut back the computational value. In particular, the proposed 2D Spatial Perceiver encodes the densely stored body-stage recollections with a lightweight Transformer that contains a hard and fast set of learnable queries.



Provided that video segmentation is a dense prediction job, we discover preserving the spatial construction of the memories is important in order that the queries are split into global-level and patch-level groups. We additionally suggest a distillation pipeline that further improves the efficiency without inference overhead. DAVIS 2017, MOSE, SA-V val, and SA-V take a look at, whereas operating at 16 FPS on iPhone 15 Pro Max. SAM to handle each picture and video inputs, with a reminiscence financial institution mechanism, and is educated with a new massive-scale multi-grained video monitoring dataset (SA-V). Despite reaching an astonishing performance in comparison with earlier video object segmentation (VOS) models and allowing extra various user prompts, SAM 2, as a server-facet basis mannequin, will not be efficient for on-system inference. CPU and NPU. Throughout the paper, we interchangeably use iPhone and iPhone 15 Pro Max for simplicity.. SAM for higher effectivity solely consider squeezing its image encoder because the mask decoder is extremely lightweight. SAM 2. Specifically, SAM 2 encodes past frames with a memory encoder, and these frame-degree recollections together with object-level pointers (obtained from the mask decoder) serve as the memory financial institution.



These are then fused with the options of present frame via reminiscence consideration blocks. As these reminiscences are densely encoded, this leads to an enormous matrix multiplication during the cross-attention between present body features and reminiscence features. Therefore, despite containing relatively fewer parameters than the picture encoder, the computational complexity of the reminiscence attention will not be reasonably priced for on-system inference. The speculation is further proved by Fig. 2, iTagPro Item Finder where lowering the number of reminiscence consideration blocks nearly linearly cuts down the overall decoding latency and within each memory attention block, eradicating the cross attention offers the most vital speed-up. To make such a video-based mostly monitoring model run on system, in EdgeTAM, we take a look at exploiting the redundancy in videos. To do this in apply, we suggest to compress the raw frame-level memories before performing reminiscence consideration. We begin with naïve spatial pooling and observe a significant performance degradation, especially when using low-capability backbones.



However, naïvely incorporating a Perceiver also leads to a extreme drop in efficiency. We hypothesize that as a dense prediction process, the video segmentation requires preserving the spatial construction of the reminiscence bank, which a naïve Perceiver discards. Given these observations, we propose a novel lightweight module that compresses frame-level reminiscence feature maps while preserving the 2D spatial structure, named 2D Spatial Perceiver. Specifically, we split the learnable queries into two groups, the place one group functions equally to the original Perceiver, where every question performs world consideration on the input options and outputs a single vector as the body-stage summarization. In the opposite group, the queries have 2D priors, i.e., every query is barely chargeable for compressing a non-overlapping native patch, thus the output maintains the spatial construction whereas lowering the whole number of tokens. Along with the architecture improvement, we further propose a distillation pipeline that transfers the information of the powerful instructor SAM 2 to our student mannequin, which improves the accuracy without charge of inference overhead.



We find that in each stages, aligning the options from image encoders of the unique SAM 2 and our efficient variant benefits the efficiency. Besides, we further align the feature output from the reminiscence consideration between the teacher SAM 2 and our student model within the second stage so that in addition to the image encoder, memory-associated modules may also obtain supervision alerts from the SAM 2 teacher. SA-V val and check by 1.3 and 3.3, respectively. Putting together, we propose EdgeTAM (Track Anything Model for Edge gadgets), that adopts a 2D Spatial Perceiver for effectivity and information distillation for accuracy. Through complete benchmark, we reveal that the latency bottleneck lies in the memory consideration module. Given the latency evaluation, we suggest a 2D Spatial Perceiver that considerably cuts down the reminiscence consideration computational value with comparable efficiency, which may be integrated with any SAM 2 variants. We experiment with a distillation pipeline that performs function-smart alignment with the unique SAM 2 in each the picture and iTagPro Brand video segmentation levels and observe efficiency enhancements with none extra cost during inference.