Take me to SSD: A hybrid block-selection method on HDFS based on storage type

Minkyung Kim; Mincheol Shin; Sanghyun Park

doi:10.1145/2851613.2851658

Take me to SSD: A hybrid block-selection method on HDFS based on storage type

Minkyung Kim, Mincheol Shin, Sanghyun Park

College of Computing

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

3 Citations (Scopus)

Abstract

As the era of Big-data has risen, the importance of big data technologies is also increasing day by day. Especially, Hadoop has become a critical part of the overall Big-data system because of its ability to store, process, and analyze thousands of terabytes of data. A major issue for supporting high performance on Hadoop is managing the growth of data while satisfying high storage I/O request. Hadoop's overall performance is largely influenced by the storage input/output(I/O). However, storage I/O technologies are still very limited. Therefore, now more than ever, studies on improving storage I/O on a distributed file system of Hadoop(HDFS) have been gaining popularity. To this end, latest trend in storage systems is to utilize hybrid storage devices. However, it is not easy to use the information of heterogeneous storage devices in HDFS. This is because, when reading data, HDFS is unable to exploit such heterogeneous storage type information yet. In this paper, we propose a hybrid block-selection method on the HDFS, we consider the storage type such as SSD and HDD when reading data. Using this method, the Hadoop Eco System utilizes the high SSD bandwidth by priority. As a result, we certainly improve the Hadoop Eco System overall performance. In the experiments, we demonstrated that our new method efficiently reduced the execution time of select count(∗) query and TPCH benchmark up to 22% and 30% on average.1

Original language	English
Title of host publication	2016 Symposium on Applied Computing, SAC 2016
Publisher	Association for Computing Machinery
Pages	965-971
Number of pages	7
ISBN (Electronic)	9781450337397
DOIs	https://doi.org/10.1145/2851613.2851658
Publication status	Published - 2016 Apr 4
Event	31st Annual ACM Symposium on Applied Computing, SAC 2016 - Pisa, Italy Duration: 2016 Apr 4 → 2016 Apr 8

Publication series

Name	Proceedings of the ACM Symposium on Applied Computing
Volume	04-08-April-2016

Other

Other	31st Annual ACM Symposium on Applied Computing, SAC 2016
Country/Territory	Italy
City	Pisa
Period	16/4/4 → 16/4/8

Bibliographical note

Funding Information:
This work was supported by the National Research Foundation of Korea(NRF) grant funded by the Korea government(MSIP) (NRF-2015R1A2A1A05001845).

Publisher Copyright:
© 2016 ACM.

All Science Journal Classification (ASJC) codes

Software

Access to Document

10.1145/2851613.2851658

Cite this

@inproceedings{6e98f7dac6834d2b9ed7a71575a6c807,

title = "Take me to SSD: A hybrid block-selection method on HDFS based on storage type",

abstract = "As the era of Big-data has risen, the importance of big data technologies is also increasing day by day. Especially, Hadoop has become a critical part of the overall Big-data system because of its ability to store, process, and analyze thousands of terabytes of data. A major issue for supporting high performance on Hadoop is managing the growth of data while satisfying high storage I/O request. Hadoop's overall performance is largely influenced by the storage input/output(I/O). However, storage I/O technologies are still very limited. Therefore, now more than ever, studies on improving storage I/O on a distributed file system of Hadoop(HDFS) have been gaining popularity. To this end, latest trend in storage systems is to utilize hybrid storage devices. However, it is not easy to use the information of heterogeneous storage devices in HDFS. This is because, when reading data, HDFS is unable to exploit such heterogeneous storage type information yet. In this paper, we propose a hybrid block-selection method on the HDFS, we consider the storage type such as SSD and HDD when reading data. Using this method, the Hadoop Eco System utilizes the high SSD bandwidth by priority. As a result, we certainly improve the Hadoop Eco System overall performance. In the experiments, we demonstrated that our new method efficiently reduced the execution time of select count(∗) query and TPCH benchmark up to 22% and 30% on average.1",

author = "Minkyung Kim and Mincheol Shin and Sanghyun Park",

note = "Funding Information: This work was supported by the National Research Foundation of Korea(NRF) grant funded by the Korea government(MSIP) (NRF-2015R1A2A1A05001845). Publisher Copyright: {\textcopyright} 2016 ACM.; 31st Annual ACM Symposium on Applied Computing, SAC 2016 ; Conference date: 04-04-2016 Through 08-04-2016",

year = "2016",

month = apr,

day = "4",

doi = "10.1145/2851613.2851658",

language = "English",

series = "Proceedings of the ACM Symposium on Applied Computing",

publisher = "Association for Computing Machinery",

pages = "965--971",

booktitle = "2016 Symposium on Applied Computing, SAC 2016",

}

Kim, M, Shin, M & Park, S 2016, Take me to SSD: A hybrid block-selection method on HDFS based on storage type. in 2016 Symposium on Applied Computing, SAC 2016. Proceedings of the ACM Symposium on Applied Computing, vol. 04-08-April-2016, Association for Computing Machinery, pp. 965-971, 31st Annual ACM Symposium on Applied Computing, SAC 2016, Pisa, Italy, 16/4/4. https://doi.org/10.1145/2851613.2851658

Take me to SSD: A hybrid block-selection method on HDFS based on storage type. / Kim, Minkyung; Shin, Mincheol; Park, Sanghyun.
2016 Symposium on Applied Computing, SAC 2016. Association for Computing Machinery, 2016. p. 965-971 (Proceedings of the ACM Symposium on Applied Computing; Vol. 04-08-April-2016).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Take me to SSD

T2 - 31st Annual ACM Symposium on Applied Computing, SAC 2016

AU - Kim, Minkyung

AU - Shin, Mincheol

AU - Park, Sanghyun

N1 - Funding Information: This work was supported by the National Research Foundation of Korea(NRF) grant funded by the Korea government(MSIP) (NRF-2015R1A2A1A05001845). Publisher Copyright: © 2016 ACM.

PY - 2016/4/4

Y1 - 2016/4/4

N2 - As the era of Big-data has risen, the importance of big data technologies is also increasing day by day. Especially, Hadoop has become a critical part of the overall Big-data system because of its ability to store, process, and analyze thousands of terabytes of data. A major issue for supporting high performance on Hadoop is managing the growth of data while satisfying high storage I/O request. Hadoop's overall performance is largely influenced by the storage input/output(I/O). However, storage I/O technologies are still very limited. Therefore, now more than ever, studies on improving storage I/O on a distributed file system of Hadoop(HDFS) have been gaining popularity. To this end, latest trend in storage systems is to utilize hybrid storage devices. However, it is not easy to use the information of heterogeneous storage devices in HDFS. This is because, when reading data, HDFS is unable to exploit such heterogeneous storage type information yet. In this paper, we propose a hybrid block-selection method on the HDFS, we consider the storage type such as SSD and HDD when reading data. Using this method, the Hadoop Eco System utilizes the high SSD bandwidth by priority. As a result, we certainly improve the Hadoop Eco System overall performance. In the experiments, we demonstrated that our new method efficiently reduced the execution time of select count(∗) query and TPCH benchmark up to 22% and 30% on average.1

AB - As the era of Big-data has risen, the importance of big data technologies is also increasing day by day. Especially, Hadoop has become a critical part of the overall Big-data system because of its ability to store, process, and analyze thousands of terabytes of data. A major issue for supporting high performance on Hadoop is managing the growth of data while satisfying high storage I/O request. Hadoop's overall performance is largely influenced by the storage input/output(I/O). However, storage I/O technologies are still very limited. Therefore, now more than ever, studies on improving storage I/O on a distributed file system of Hadoop(HDFS) have been gaining popularity. To this end, latest trend in storage systems is to utilize hybrid storage devices. However, it is not easy to use the information of heterogeneous storage devices in HDFS. This is because, when reading data, HDFS is unable to exploit such heterogeneous storage type information yet. In this paper, we propose a hybrid block-selection method on the HDFS, we consider the storage type such as SSD and HDD when reading data. Using this method, the Hadoop Eco System utilizes the high SSD bandwidth by priority. As a result, we certainly improve the Hadoop Eco System overall performance. In the experiments, we demonstrated that our new method efficiently reduced the execution time of select count(∗) query and TPCH benchmark up to 22% and 30% on average.1

UR - http://www.scopus.com/inward/record.url?scp=84975801961&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84975801961&partnerID=8YFLogxK

U2 - 10.1145/2851613.2851658

DO - 10.1145/2851613.2851658

M3 - Conference contribution

AN - SCOPUS:84975801961

T3 - Proceedings of the ACM Symposium on Applied Computing

SP - 965

EP - 971

BT - 2016 Symposium on Applied Computing, SAC 2016

PB - Association for Computing Machinery

Y2 - 4 April 2016 through 8 April 2016

ER -

Take me to SSD: A hybrid block-selection method on HDFS based on storage type

Abstract

Publication series

Other

Bibliographical note

All Science Journal Classification (ASJC) codes

Access to Document

Other files and links

Fingerprint

Cite this