NASA NeMO-Net

NASA NeMO-Net, the Neural Multimodal Observation and Training Network for global coral reef assessment, is a convolutional neural network (CNN) that generates benthic habitat classification maps for coral reef and other shallow marine ecosystems from 2D satellite and 3D airborne remote sensing imagery. Training CNNs with high-accuracy labels for automated 2D and 3D semantic segmentation is a challenging task in machine learning. To overcome this big data challenge, we present a novel 3D online classification video game for mobile and desktop devices. Leveraging the power of citizen science, the NeMO-Net video game is able to generate high-resolution 3D benthic habitat labels at an unprecedented scale. The NeMO-Net video game trains users to accurately identify coral reef families and semantically segment 3D scenes captured using NASA FluidCam, the first remote sensing system capable of mitigating refractive ocean wave distortion. An active learning framework is used to allow users to rate and edit other user classifications. Data labels from the game are used to train the NeMO-Net CNN to autonomously map shallow marine systems, significantly augmenting satellite habitat mapping accuracy in these regions.We share the NeMO-Net approach to user training and retention, outline the 3D labeling technique developed to accurately label complex coral reef imagery, and present preliminary results from over 70,000 user classifications as well as criteria for evaluating and filtering user data, a vital step in overcoming the inherent variability of citizen science. Finally, we examine how future citizen science and machine learning approaches might benefit from label training in 3D space using an active learning framework.Within 7 months of its launch, NeMO-Net has reached over 300 million people globally and enabled a new generation to directly participate in a scientific campaign, uninhibited by geography, language, or physical ability. As the NeMO-Net video game reaches the needed training threshold for the NeMO-Net CNN, anticipated in early 2021, it will help produce the first cm-scale trained global shallow marine habitat mapping products later in 2021. These multimodal, multidecadal, high-resolution global data products will enable novel conservation applications by the UN (SDG 14), IUCN, federal, state, indigenous, and non-profit organizations.

Recent advances in machine learning and computer vision have enabled increased automation in benthic habitat mapping through airborne and satellite remote sensing. Here, we applied deep learning and neural network architectures in NASA NeMO-Net, a novel neural multimodal observation and training network for global habitat mapping of shallow benthic tropical marine systems. These ecosystems, particularly coral reefs, are undergoing rapid changes as a result of increasing ocean temperatures, acidification, and pollution, among other stressors. Remote sensing from air and space has been the primary method in which changes are assessed within these important, often remote, ecosystems at a global scale. However, such global datasets often suffer from large spectral variances due to the time of observation, atmospheric effects, water column properties, and heterogeneous instruments and calibrations. To address these challenges, we developed an object-based fully convolutional network (FCN) to improve upon the spatial-spectral classification problem inherent in multimodal datasets. We showed that with training upon augmented data in conjunction with classical methods, such as K-nearest neighbors, we were able to achieve better overall classification and segmentation results. This suggests FCNs are able to effectively identify the relative applicable spectral and spatial spaces within an image, whereas pixel-based classical methods excel at classification within those identified spaces. Our spectrally invariant results, based on minimally preprocessed WorldView-2 and Planet satellite imagery, show a total accuracy of approximately 85% and 80%, respectively, over nine classes when trained and tested upon a chain of Fijian islands imaged under highly variable day-to-day spectral inputs.