It is difficult to provide absolute scores in many subjective evaluation tasks. For example, scoring the degree of smiles is a difficult task, and the scores can vary greatly depending on the question. Because estimating such a wide range of subjective scores is difficult, sidestep methods have been widely considered. Approaches based on learning-to-rank are now recognised as a promising solution. They provide a learning framework for simply ordering scores among target samples rather than absolute scores. Returning to our previous example, sorting the images by degree of smile is simpler than assigning smile scores to each image.

Metrics for Image Quality Assessment (IQA) take an arbitrary image as input and produce a quality score. The goal of these metrics is to quantitatively represent the human perception of quality. These metrics are commonly used to assess the performance of algorithms in computer vision fields such as image compression, image transmission, and image processing.

Image processing algorithms are fine-tuned to achieve a high level of detail, increased enhancement and realism, and a sense of depth in images. These algorithms were previously configured by hand.

Automatic image quality assessment methods have evolved over time, greatly simplifying the work of algorithm designers. Two such methods are the Structural similarity index (SSIM) and the Blind/Referenceless Image Spatial Quality Evaluator (BRISQUE). SSIM is a full-reference IQA method while BRISQUE is a no-reference image quality assessment – meaning the algorithm only receives the distorted image whose quality is being evaluated, rather than a base image to evaluate image quality. It remains a challenge to develop metrics that capture perceived image quality despite these methods.

Automatic image quality assessment is an issue for many image processing applications, such as image/video compression or reconstruction. These applications fuel the development of computational quality metrics, which forecast the level of impairment as perceived by a human observer. Such metrics must be trained using ground truth data gathered during subjective quality assessment experiments. However, it is not always widely recognised that data from different quality assessment experiments may be scaled differently, resulting in significantly different quality scores.

Supporting authenticity experiments raised a new challenge:

We don’t teach or instruct labelers how to evaluate the authenticity of an image. But this leads to inconsistency in the rating given by the different labelers on the same image. We can have one user rank a specific image for 9 and one as 7. Although both seemed correct to the respective labelers, the discrepancy in the rating raises different questions.

But again, clients generally use this process to measure the quality of the last version. Therefore, we expect similar results from different groups of labelers. We solve this by normalizing those results.