[Paper] WildLIFT: Lifting monocular drone video to 3D for species-agnostic wildlife monitoring

Source: arXiv - 2604.24718v1

Overview

The paper introduces WildLIFT, a software pipeline that turns ordinary monocular drone video into a full‑3D representation of wildlife scenes. By fusing 3‑D reconstruction with open‑vocabulary instance segmentation, the system can detect, label, and track animals of any species in three dimensions—something that previously required costly multi‑camera rigs or manual 3‑D annotation.

Key Contributions

• Species‑agnostic 3‑D detection: Uses open‑vocabulary 2‑D segmentation (e.g., SAM, Grounding‑DINO) to recognize any animal without species‑specific training data.
• Oriented 3‑D bounding boxes with semantic faces: Each box stores not only position and size but also which side faces the camera, enabling viewpoint‑aware analyses (e.g., occlusion, coverage).
• Keyframe‑based annotation refinement: Reduces manual labeling effort by allowing users to correct a small set of keyframes, after which the system propagates corrections throughout the video.
• Large‑scale validation: Tested on 2,581 frames (≈6,700 3‑D detections) across four large‑mammal species, demonstrating high identity consistency even in dense, multi‑animal scenes.
• Open‑source framework: Designed to plug into existing drone‑based monitoring pipelines, with minimal hardware requirements (just a single RGB camera).

Methodology

1. Video‑to‑Structure‑from‑Motion (SfM): The raw drone footage is processed with an off‑the‑shelf SfM tool (e.g., COLMAP) to recover camera poses and a sparse point cloud of the terrain.
2. Dense 3‑D reconstruction: A multi‑view stereo algorithm densifies the point cloud, yielding a mesh that approximates the ground and vegetation.
3. 2‑D open‑vocabulary segmentation: Each frame is fed to a foundation model (e.g., Segment Anything Model) that produces pixel‑level masks for “animal” objects, regardless of species.
4. 3‑D lifting: The 2‑D masks are back‑projected into the 3‑D space using the known camera pose, generating oriented 3‑D bounding boxes. The box orientation is derived from the camera‑to‑object vector, giving a “front‑face” label.
5. Temporal association: A simple Kalman‑filter‑based tracker links boxes across frames, maintaining consistent IDs even when animals cross paths or become partially occluded.
6. Keyframe refinement UI: Users can edit a subset of frames (e.g., correcting a mis‑detected box). The system propagates these edits to neighboring frames via the tracker, dramatically cutting manual effort.

Results & Findings

• Detection accuracy: WildLIFT achieved >85 % average precision (AP) for 3‑D bounding boxes across all four species, comparable to specialized 2‑D detectors.
• Identity consistency: In multi‑animal sequences, the tracker preserved correct IDs for >90 % of frames, even when animals overlapped or were partially hidden.
• Annotation efficiency: Using the keyframe refinement tool, annotators needed to manually correct only ~5 % of frames to reach the same quality as fully manual 3‑D labeling, cutting labor by roughly 20×.
• Viewpoint metrics: The semantic face information allowed the authors to quantify how often each animal was observed from different angles, a metric previously unavailable in standard 2‑D pipelines.

Practical Implications

• Scalable population surveys: Conservation teams can now extract reliable 3‑D counts and movement paths from a single drone fly‑over, eliminating the need for costly multi‑camera setups.
• Behavioral ecology: Researchers can study posture, inter‑animal spacing, and occlusion patterns in three dimensions, opening new avenues for understanding social dynamics.
• Automated monitoring pipelines: The framework can be integrated into existing drone‑data ingestion systems (e.g., AirMap, DroneDeploy) to automatically generate structured metadata for downstream GIS or statistical analysis.
• Reduced field time: Faster data processing and lower annotation overhead mean fewer on‑site personnel and quicker turnaround from data collection to actionable insights.
• Cross‑domain reuse: Because the segmentation backbone is open‑vocabulary, the same pipeline can be repurposed for other aerial monitoring tasks—such as livestock management, illegal logging detection, or disaster assessment—without retraining.

Limitations & Future Work

• Dependence on good SfM: Low‑texture environments (e.g., snow, water) can degrade camera pose estimation, limiting 3‑D accuracy.
• Resolution constraints: Small animals or those far from the drone may be missed due to pixel‑level segmentation limits.
• Occlusion handling: While the tracker copes with moderate overlap, severe occlusions still cause identity switches.
• Future directions: The authors plan to incorporate neural radiance fields (NeRF) for denser reconstructions, explore self‑supervised domain adaptation to improve detection of smaller species, and add real‑time processing capabilities for on‑board analytics.

Authors

• Vandita Shukla
• Fabio Remondino
• Blair Costelloe
• Benjamin Risse

Paper Information

• arXiv ID: 2604.24718v1
• Categories: cs.CV
• Published: April 27, 2026
• PDF: Download PDF