Robust 3D Hand Tracking with Multi-View Videos

In this paper, we introduce a robust 3D hand tracking framework using multi-view video, designed for the construction of a database. For hand pose estimation, real-world videos present frequent challenging scenarios, where current methods could fail, such as complex finger poses, two-hand interactions, and self-occluded situations. To address these challenges, our focus lies in enhancing the robustness and stability of 3D hand pose tracking. Firstly, we leverage multiple pre-trained 2D hand pose estimation algorithms, and combine them based on confidence and two-hand interactions. Additionally, to consider both cross-frame and cross-view consistency, our framework optimizes the entire joints with three objectives: data, guidance, and feature constraints. Throughout evaluation on real-world data, we show that the proposed method is robust to various challenging cases.