Towards performance optimization for Hadoop MapReduce applications

Name: Htay, Than Than

Organization : University of Computer Studies. Ministry of Education

Email : thanthanhtay87@gmail.com

Name: Phyu, Sabai

Organization : University of Computer Studies. Ministry of Education

Email : sabaiphyu72@gmail.com

keyword: Apache Hadoop

LCSH: Big data -- Management

; MapReduce (Computer file)

LCSH: Parallel processing (Electronic computers)

LCSH: Resource allocation

LCSH: Mathematical optimization

Abstract: Apache Hadoop is a widely used open-source distributed platform for big data processing and provides a YARN-based distributed parallel processing framework on low-cost commodity machines. However, YARN adopts static resource management (that is, the number of containers available per node and the size of each container are static in nature), depending on pre-configured default resource units called containers, which leads to poor performance when dealing with various sorts of MapReduce applications. In addition, during the last wave of a job, many available resources frequently become idle because YARN does not consider the wave behavior in MapReduce tasks. To take advantage of idle resources resulting in performance improvement, an important parameter-the number of map tasks-needs to be optimized based on the available resources and governed by split size. Therefore, this parameter is optimized through split size tuning based on available resources. To address the drawback of static resource management in Hadoop YARN, the number of concurrent containers per machine is tuned to optimize node performance for running each MapReduce application. According to the experimental results, the proposed system, which optimizes the selected parameter on optimized concurrent containers, can achieve performance gains for MapReduce applications while reducing optimization overheads.

King Mongkut's University of Technology North Bangkok. Central Library

Address: BANGKOK

Email: library@kmutnb.ac.th

Created: 2020

Modified: 2026-04-03

Issued: 2026-04-03

บทความ/Article

application/pdf

BibliograpyCitation : In Prince of Songkla University. College of Computing. The 17th International Conference on Electrical Engineering/Electronics, Computer, Telecommunications and Information Technology (ECTI-CON 2020) (pp.698-701). Pattani : Prince of Songkla University

eng

©copyrights King Mongkut's University of Technology North Bangkok

ลำดับที่.	ชื่อแฟ้มข้อมูล	ขนาดแฟ้มข้อมูล	จำนวนเข้าถึง	วัน-เวลาเข้าถึงล่าสุด
1	ECTI-CON 2020pp.698-701.pdf	937.45 KB

ใช้เวลา

0.035105 วินาที