Case Study: Vertical Obstruction (VO) Algorithm Evaluations

We have performed several evaluation studies of automated VO extraction algorithms and procedures and have on-going work in this area. VO detection is a critical capability for safety of navigation, particularly for low-flying military aircraft. For these studies, we provide full lifecycle experiment design, often having both budget and timeline constraints. Our most recent work involving analysis of LIDAR-derived VOs, required derivation of “ground truth” data over approximately 1400 SqKm with approximately 1800 VOs, using both airborne and satellite imagery. While our focus in these studies is on Probability of Detection (Pd) and False Alarm Rate (FAR), we attempt to capture details in the evaluation that lead to enhanced characterization of Pd as a function of various parameters, such as terrain cover, terrain variability, collection geometry, and VO height, as well as a more in-depth understanding of false alarms, such as their causes and variability. During the course of these activities, our analysis has often led to important discussions, such as success criteria, technology insertion requirements, the precise definition of vertical obstructions, interpretation of detection of objects that did not fit the specific VO definition, and the role of analysts in quality assurance.