A Hybrid Approach for Semantic Image Annotation

Abstract

In this study, a framework that generates natural language descriptions of images within a controlled environment is proposed. Previous work on neural networks mostly focused on choosing the right labels and/or increasing the number of related labels to depict an image. However, creating a textual description of an image is a completely different phenomenon, structurally, syntactically, and semantically. The proposed semantic image annotation framework presents a novel combination of deep learning models and aligned annotation results derived from the instances of the ontology classes to generate sentential descriptions of images. Our hybrid approach benefits from the unique combination of deep learning and semantic web technologies. We detect objects from unlabeled sports images using a deep learning model based on a residual network and a feature pyramid network, with the focal loss technique to obtain predictions with high probability. The proposed framework not only produces probabilistically labeled images, but also the contextual results obtained from a knowledge base exploiting the relationship between the objects. The framework's object detection and prediction performances are tested with two datasets where the first one includes individual instances of images containing everyday scenes of common objects and the second custom dataset contains sports images collected from the web. Moreover, a sample image set is created to obtain annotation result data by applying all framework layers. Experimental results show that the framework is effective in this controlled environment and can be used with other applications via web services within the supported sports domain.