FathomNet is an open-source image database that can be used to train, test, and validate state-of-the-art artificial intelligence algorithms to help us understand our ocean and its inhabitants. Inspired by annotated image databases such as ImageNet and COCO, FathomNet aims to establish the same kind of reference data set for images of ocean life. The long-term goal of FathomNet is to aggregate >1k fully annotated and localized images per marine species of Animalia (>200k), with the ability to expand and include other underwater concepts (e.g., substrate type, equipment, debris, etc.) for training and validating machine learning models. We hope that contributions from the broader community will realize our goals for FathomNet.