All grains, one scheme (AGOS): Learning multigrain instance representation for aerial scene classification

Aerial scene classification remains challenging as: 1) the size of key objects in determining the scene scheme varies greatly and 2) many objects irrelevant to the scene scheme are often flooded in the image. Hence, how to effectively perceive the region of interests (RoIs) from a variety of sizes and build more discriminative representation from such complicated object distribution is vital to understand an aerial scene. In this article, we propose a novel all grains, one scheme (AGOS) framework to tackle these challenges. To the best of our knowledge , it is the first work to extend the classic multiple instance learning (MIL) into multigrain formulation. Specifically, it consists of a multigrain perception (MGP) module, a multibranch multi-instance representation (MBMIR) module, and a self-aligned semantic fusion (SSF) module. First, our MGP module preserves the differential dilated convolutional features from the backbone, which magnifies the discriminative information from multigrains. Then, our MBMIR module highlights the key instances in the multigrain representation under the MIL formulation. Finally, our SSF module allows our framework to learn the same scene scheme from multigrain instance representations and fuses them, so that the entire framework is optimized as a whole. Notably, our AGOS is flexible and can be easily adapted to existing convolutional neural networks (CNNs) in a plug-and-play manner. Extensive experiments on UCM, aerial image dataset (AID), and Northwestern Polytechnical University (NWPU) benchmarks demonstrate that our AGOS achieves a comparable performance against the state-of-the-art methods.