Subjective quality evaluation VIA paired comparison: Application to scalable video coding

Jong Seok Lee; Francesca De Simone; Touradj Ebrahimi

doi:10.1109/TMM.2011.2157333

Subjective quality evaluation VIA paired comparison: Application to scalable video coding

Jong Seok Lee, Francesca De Simone, Touradj Ebrahimi

Research output: Contribution to journal › Article › peer-review

85 Citations (Scopus)

Abstract

Scalable video coding is a powerful solution for content delivery in many interactive multimedia services due to its adaptability to varying terminal and network constraints. In order to successfully exploit such adaptability, it is necessary to understand users' preference among various scalability options and consequently develop an optimal bit rate adaptation strategy. In this paper, we present a study of subjective quality assessment of scalable video coding, which investigates the influence of the combination of scalability options on perceived quality with the goal of providing guidelines for an adaptive strategy that selects the optimal combination for a given bandwidth constraint. In particular, the study is based on paired comparison of stimuli that is suitable for our goal due to its simplicity and easiness. We propose a new method, called Paired Evaluation via Analysis of Reliability (PEAR), which analyzes paired comparison results and produces not only quality scores but also intuitive measures of confidence of the scores for significance analysis. Results and analysis of extensive subjective tests for two different scalable video codecs and high definition contents are described, from which general consistent conclusions are drawn. The video and subjective data used in the paper are publicly available to the research community.

Original language	English
Article number	5772009
Pages (from-to)	882-893
Number of pages	12
Journal	IEEE Transactions on Multimedia
Volume	13
Issue number	5
DOIs	https://doi.org/10.1109/TMM.2011.2157333
Publication status	Published - 2011 Oct

Bibliographical note

Funding Information:
Manuscript received September 01, 2010; revised February 28, 2011; accepted May 09, 2011. Date of publication May 23, 2011; date of current version September 16, 2011. This work was supported in part by the European Community’s Seventh Framework Programme (FP7/2007-2011) under grant agreement no. 216444 (PetaMedia), and in part by the Swiss NCCR Interactive Multimodal Information Management (IM2). The associate editor coordinating the review of this manuscript and approving it for publication was Dr. Jin Li.

All Science Journal Classification (ASJC) codes

Signal Processing
Media Technology
Computer Science Applications
Electrical and Electronic Engineering

Access to Document

10.1109/TMM.2011.2157333

Cite this

@article{959281d1d21a4e0b8556f2d5190617a3,

title = "Subjective quality evaluation VIA paired comparison: Application to scalable video coding",

abstract = "Scalable video coding is a powerful solution for content delivery in many interactive multimedia services due to its adaptability to varying terminal and network constraints. In order to successfully exploit such adaptability, it is necessary to understand users' preference among various scalability options and consequently develop an optimal bit rate adaptation strategy. In this paper, we present a study of subjective quality assessment of scalable video coding, which investigates the influence of the combination of scalability options on perceived quality with the goal of providing guidelines for an adaptive strategy that selects the optimal combination for a given bandwidth constraint. In particular, the study is based on paired comparison of stimuli that is suitable for our goal due to its simplicity and easiness. We propose a new method, called Paired Evaluation via Analysis of Reliability (PEAR), which analyzes paired comparison results and produces not only quality scores but also intuitive measures of confidence of the scores for significance analysis. Results and analysis of extensive subjective tests for two different scalable video codecs and high definition contents are described, from which general consistent conclusions are drawn. The video and subjective data used in the paper are publicly available to the research community.",

keywords = "Bradley-Terry model, content distribution, multimedia quality assessment, paired comparison, scalable video coding, subjective test",

author = "Lee, {Jong Seok} and {De Simone}, Francesca and Touradj Ebrahimi",

note = "Funding Information: Manuscript received September 01, 2010; revised February 28, 2011; accepted May 09, 2011. Date of publication May 23, 2011; date of current version September 16, 2011. This work was supported in part by the European Community{\textquoteright}s Seventh Framework Programme (FP7/2007-2011) under grant agreement no. 216444 (PetaMedia), and in part by the Swiss NCCR Interactive Multimodal Information Management (IM2). The associate editor coordinating the review of this manuscript and approving it for publication was Dr. Jin Li.",

year = "2011",

month = oct,

doi = "10.1109/TMM.2011.2157333",

language = "English",

volume = "13",

pages = "882--893",

journal = "IEEE Transactions on Multimedia",

issn = "1520-9210",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

number = "5",

}

TY - JOUR

T1 - Subjective quality evaluation VIA paired comparison

T2 - Application to scalable video coding

AU - Lee, Jong Seok

AU - De Simone, Francesca

AU - Ebrahimi, Touradj

N1 - Funding Information: Manuscript received September 01, 2010; revised February 28, 2011; accepted May 09, 2011. Date of publication May 23, 2011; date of current version September 16, 2011. This work was supported in part by the European Community’s Seventh Framework Programme (FP7/2007-2011) under grant agreement no. 216444 (PetaMedia), and in part by the Swiss NCCR Interactive Multimodal Information Management (IM2). The associate editor coordinating the review of this manuscript and approving it for publication was Dr. Jin Li.

PY - 2011/10

Y1 - 2011/10

N2 - Scalable video coding is a powerful solution for content delivery in many interactive multimedia services due to its adaptability to varying terminal and network constraints. In order to successfully exploit such adaptability, it is necessary to understand users' preference among various scalability options and consequently develop an optimal bit rate adaptation strategy. In this paper, we present a study of subjective quality assessment of scalable video coding, which investigates the influence of the combination of scalability options on perceived quality with the goal of providing guidelines for an adaptive strategy that selects the optimal combination for a given bandwidth constraint. In particular, the study is based on paired comparison of stimuli that is suitable for our goal due to its simplicity and easiness. We propose a new method, called Paired Evaluation via Analysis of Reliability (PEAR), which analyzes paired comparison results and produces not only quality scores but also intuitive measures of confidence of the scores for significance analysis. Results and analysis of extensive subjective tests for two different scalable video codecs and high definition contents are described, from which general consistent conclusions are drawn. The video and subjective data used in the paper are publicly available to the research community.

AB - Scalable video coding is a powerful solution for content delivery in many interactive multimedia services due to its adaptability to varying terminal and network constraints. In order to successfully exploit such adaptability, it is necessary to understand users' preference among various scalability options and consequently develop an optimal bit rate adaptation strategy. In this paper, we present a study of subjective quality assessment of scalable video coding, which investigates the influence of the combination of scalability options on perceived quality with the goal of providing guidelines for an adaptive strategy that selects the optimal combination for a given bandwidth constraint. In particular, the study is based on paired comparison of stimuli that is suitable for our goal due to its simplicity and easiness. We propose a new method, called Paired Evaluation via Analysis of Reliability (PEAR), which analyzes paired comparison results and produces not only quality scores but also intuitive measures of confidence of the scores for significance analysis. Results and analysis of extensive subjective tests for two different scalable video codecs and high definition contents are described, from which general consistent conclusions are drawn. The video and subjective data used in the paper are publicly available to the research community.

KW - Bradley-Terry model

KW - content distribution

KW - multimedia quality assessment

KW - paired comparison

KW - scalable video coding

KW - subjective test

UR - http://www.scopus.com/inward/record.url?scp=80052932019&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=80052932019&partnerID=8YFLogxK

U2 - 10.1109/TMM.2011.2157333

DO - 10.1109/TMM.2011.2157333

M3 - Article

AN - SCOPUS:80052932019

SN - 1520-9210

VL - 13

SP - 882

EP - 893

JO - IEEE Transactions on Multimedia

JF - IEEE Transactions on Multimedia

IS - 5

M1 - 5772009

ER -

Subjective quality evaluation VIA paired comparison: Application to scalable video coding

Abstract

Bibliographical note

All Science Journal Classification (ASJC) codes

Access to Document

Other files and links

Fingerprint

Cite this