Enhancing network I/o performance for a virtualized Hadoop cluster

Jinkyu Jeong, Dong Hoon Choi, Heeseung Jo

Research output: Contribution to journalArticlepeer-review

2 Citations (Scopus)

Abstract

A MapReduce programming model is proposed to process big data using Hadoop, one of the major cloud computing frameworks. With the increasing adoption of cloud computing, running a Hadoop framework on a virtualized cluster is a compelling approach to reducing costs and increasing efficiency. In this paper, we measure the performance of a virtualized network and analyze the impact of network performance on Hadoop workloads running on a virtualized cluster. Then, we propose a virtualized network I/O architecture as a novel optimization for a virtualized Hadoop cluster for a public/private cloud provider. The proposed network architecture combines traditional network configurations and achieves better performance for Hadoop workloads. We also show a better way to utilize the rack awareness feature of the Hadoop framework in the proposed computing environment. The evaluation demonstrates that the proposed network architecture and mechanisms improve performance by up to 4.1 times compared with a bridge network architecture. This novel architecture can even virtually match the performance of the expensive, hardware-based single root I/O virtualization network architecture.

Original languageEnglish
Article numbere3974
JournalConcurrency Computation Practice and Experience
Volume29
Issue number8
DOIs
Publication statusPublished - 2017 Apr 25

Bibliographical note

Publisher Copyright:
Copyright © 2016 John Wiley & Sons, Ltd.

All Science Journal Classification (ASJC) codes

  • Theoretical Computer Science
  • Software
  • Computer Science Applications
  • Computer Networks and Communications
  • Computational Theory and Mathematics

Fingerprint

Dive into the research topics of 'Enhancing network I/o performance for a virtualized Hadoop cluster'. Together they form a unique fingerprint.

Cite this