by Editor Yan Zhang
Edited and published by Yvette Depaepe, the 25^th of March

AiCUS, an AI photography evaluation system developed by DeepMountain (https://deepmountain.com.au), was officially launched on 1x.com.
From now on, the majority of 1x users will have the opportunity to view AiCUS's AI evaluation of their photography works.

However, the AI evalutation is just an extra feedback for our photographes and not at all part of the actual decision-making in curation.

 'Actinotus Forsythii' by Yan Zhang

DeepMountain is the first company to develop an AI photography evaluation system with professional and practical performance, and 1x.com has also become the first international photography online gallery in the industry to use an AI curator.

In this article, I provide some general information about what an AI curator is and explain how AiCUS works on 1x.com.

 1x.com launched DeepMountain's AI curator system AiCUS.

HOW DOES  AI  EVALUATE A PHOTO?

 
Teaching AI to “see” an image 

How does AI evaluate a photo? Well, to answer this question, we first need to know how AI “sees” (recognises) a photo. The core technology of modern AI image recognition is Deep Convolutional Neural Network (DCNN). In general, a convolutional neural network (CNN) can be depicted by the following diagram.

 A CNN structure (source: cs231n.github.io).

 
The left is a regular 3-layer neural network, which is the basic component of a CNN. The right shows that a convolution net (ConvNet) – a module of layers, arranges its neurons in 3-dimension (width, height, depth), as visualised in one of the modules. Then information of the input image is represented by these 3D ConvNets.

In practice, a CNN may contain hundreds or even thousands of layers of depth, hence the name Deep CNN – DCNN. Such a DCNN can effectively learn a large amount of image data, and finally extract most import feature information from the repeated learning of the same type of images (such as human faces), so as to achieve the purpose of accurately identifying such image information.

These are facial features from simple to complex, generated by deep learning algorithms for face recognition (source: Lee, et al https://dl.acm.org/doi/10.1145/1553374.1553453).

Let us take the above face recognition as an example. Through learning, each module of DCNN gradually generates from simple to complex feature information about the face, such as eyebrows, eyes, nose, etc., and finally generates complete face feature information. With this feature information, DCNN can process a new input image to determine whether the image contains a face.

The training process of AI curator

Although using DCNN for image aesthetics assessment may seem quite different from the task of recognizing human faces, the basic principles are the same. Image Aesthetics Assessment (IAA) has been a hot topic in deep learning research since 2015. The basic idea can be described with the following diagram.

The structure of image aesthetics learning model based on DCNN.

 
In order for the DCNN model to learn our human photography aesthetics, we first need to have a large enough collection of photographs, such a collection contains as few as tens of thousands, as many as hundreds of thousands or even millions of photographs. Such data is then divided into two parts: one part for learning, called "training data", and another part for testing, called "test data". With "training data" and "testing data", we can alternately feed these data into the DCNN model – the training data tells the model which images are good, and which are not; and the test data passes the model's self-testing, to correct the bias and automatically adjust various parameters of the model.

This process is repeated until the DCNN model gradually establishes the photography aesthetic feature information close to human. 

Photo “Bridal Veil Falls” by Yan Zhang. Later we will use this photo to explain some basic features of AiCUS.

About photography dataset 

Intuitively, we need a dataset of photographic images with both quantity and quality to train our DCNN model. At present, in the field of AI based IAA (Image Aesthetics Assessment) research, there is only one large-scale photography dataset, AVA (Aesthetics Visual Analysis Dataset), for researchers to develop IAA systems. AVA was jointly developed by two Spanish and French AI researchers in 2012 and contains 250,000 photographic images. AVA images are classified into "good" and "no good" categories, and this categorization is determined by popular ratings (an average of 100-200 popular votes per photo).

AVA Dataset, which Contains 250,000 photographic images, 33% are "good" images and 67% are "no-good" images.

 
However, the picture quality of AVA is not high. Due to the randomness and inconsistency of public image ratings, a large proportion of "good" photos, in the eyes of our professional photographers, are absolutely average photos, such as the following AVA high-rated images:

 A set of images with high AVA ratings but ordinary aesthetic quality.

Obviously, with a common saying in machine learning: "Garbage in, garbage out", such a dataset cannot train a truly useful model.
In the past few years, altrhough many IAA systems have been developed, because they are all based on AVA, these systems and models cannot be used in practice, let alone for professional photography evaluation.

FEATURES OF AiCUS

To put is simply, AiCUS establishes its own photography aesthetic criteria through deep learning, so as to evaluate each input photo based on there criteria.

The AiCUS (Landscape) system has two major components - DCNN learning model and logical reasoning mechanism.

The DCNN model has a depth of more than 600 layers and contains 90 million parameters, which can be said to be a quite complex neural network architecture.

In our design, the DCNN learning model of AiCUS can independently evaluate the four aesthetic characteristics of a landscape image: global view, local view, compostions and color highlight, and then the logical reasoning mechanism of AiCUS gives the overall rating of the input image based on the four indicators.  It should be noted that the current AI evaluation result showing on 1x.com is only a global view score.  The complete AiCUS evaluation subdivision mechanism will be updated and launched in a few weeks.

 AiCUS photography dataset

The dataset used by AiCUS is called CPD (Curated Photography Dataset), independently established by DeepMountains.
This Dataset is the most comprehensive and professional photographic image collection in the field of IAA research so far.
The current CPD image volume is close to 500.000 images, including 5 different photography categories. In addition to 1x.com's photos, the dataset also includes many other photos that have been professionally evaluated (such as international competitions, photography exhibitions, etc.), as well as representative works of hundreds of the best photographers in the contemporary world. 

CPD is the most professional and biggest collection of photographic data in the field of IAA (Image Aesthetics Assessment) research so far.

Some high quality CPD images.

AiCus learns four distinct aesthetic characteristics.

AiCUS uses independent DCNN models to learn four major aesthetic characteristics: global view, local view, composition and color highlight, respectively, so as to obtain the degree of influence of these factors on the aesthetics of a photo. Interestingly, after expensive experiments, AiCUS found that all these four characteristics play an important aesthetic role in a succesful (landscape) photo, with different degrees.  Therefore, base on the underlying DCNN learning model, we also designed an AI logical reasoning mechanism, so that AiCUS can effectively and rationally reason about these four aesthetic standards it has learned when making a final evaluation of a photo.

 AI curator on 1x.com.

AiCUS LEARNING NEVER ENDS

AI photo evalutation is a new concept for the photography industry. It is the fusion of AI technology and photography art.  Everything is being explored, and DeepMountain and 1x.com are at the forefront of this exploration.

Example of 1x.com AI evaluation result -  an image with mountain, sunset and close-up foreground flowers.

 

AiCUS, currently running on 1x.com, has no substitute for the work of the human curators, and the award, publication or rejection of a photo is not tied to its AI score. As mentioned earlier, AiCUS currently only includes global view assessment in landscape photography. However, the upcoming AiCUS update will add evaluation functions for four new categories of photographs: portrait, architecture, wildlife and abstraction.

1x.com AI evaluation result – an image with dramatic sky and a giant rocky stack.

1x.com AI evaluation result – an aerial image.

1x.com AI evaluation result – an image with misty theme.

1x.com AI evaluation result – an image of beautiful lake.

1x.com AI evaluation result – a seascape image.

1x.com AI evaluation result – a waterfall image.