Dynamic resource management in a MapReduce-style platform for fast data processing

Kasper Grud Skat Madsen; Yongluan Zhou

doi:10.1109/ICDEW.2015.7129537

Dynamic resource management in a MapReduce-style platform for fast data processing

Kasper Grud Skat Madsen, Yongluan Zhou

1 Citation (Scopus)

Abstract

There is a recent interest in building MapReduce-style platforms for fast data processing, such as MapReduce online [2] and Muppet [5]. In this paper, we highlight the need for dynamic load management in a distributed data stream processing system and present Enorm, a MapReduce-style data stream processing platform with the focus on techniques to achieve dynamic resource management, i.e. the ability to dynamically balance the workload among the running instances and scale the resource usage according to the runtime workload fluctuations. The original MapReduce framework is designed for batched processing and dynamic scaling can only be achieved between batches. To address this problem, we propose a MapReduce-style computation framework and a set of corresponding adaptation strategies that can perform dynamic scaling on the fly with low processing latency.

Original language	English
Title of host publication	Proceedings of the 31st IEEE International Conference on Data Engineering Workshops
Number of pages	4
Publisher	IEEE
Publication date	19 Jun 2015
Pages	10-13
ISBN (Electronic)	978-1-4799-8442-8
DOIs	https://doi.org/10.1109/ICDEW.2015.7129537
Publication status	Published - 19 Jun 2015
Externally published	Yes
Event	31st IEEE International Conference on Data Engineering Workshops - Seoul, Korea, Republic of Duration: 13 Apr 2015 → 17 Apr 2015

Conference

Conference	31st IEEE International Conference on Data Engineering Workshops
Country/Territory	Korea, Republic of
City	Seoul
Period	13/04/2015 → 17/04/2015

Access to Document

10.1109/ICDEW.2015.7129537

Cite this

@inproceedings{0d2c9b41528646659bac7649b6f450ea,

title = "Dynamic resource management in a MapReduce-style platform for fast data processing",

abstract = "There is a recent interest in building MapReduce-style platforms for fast data processing, such as MapReduce online [2] and Muppet [5]. In this paper, we highlight the need for dynamic load management in a distributed data stream processing system and present Enorm, a MapReduce-style data stream processing platform with the focus on techniques to achieve dynamic resource management, i.e. the ability to dynamically balance the workload among the running instances and scale the resource usage according to the runtime workload fluctuations. The original MapReduce framework is designed for batched processing and dynamic scaling can only be achieved between batches. To address this problem, we propose a MapReduce-style computation framework and a set of corresponding adaptation strategies that can perform dynamic scaling on the fly with low processing latency.",

author = "Madsen, {Kasper Grud Skat} and Yongluan Zhou",

year = "2015",

month = jun,

day = "19",

doi = "10.1109/ICDEW.2015.7129537",

language = "English",

pages = "10--13",

booktitle = "Proceedings of the 31st IEEE International Conference on Data Engineering Workshops",

publisher = "IEEE",

note = "31st IEEE International Conference on Data Engineering Workshops, ICDEW ; Conference date: 13-04-2015 Through 17-04-2015",

}

TY - GEN

T1 - Dynamic resource management in a MapReduce-style platform for fast data processing

AU - Madsen, Kasper Grud Skat

AU - Zhou, Yongluan

PY - 2015/6/19

Y1 - 2015/6/19

N2 - There is a recent interest in building MapReduce-style platforms for fast data processing, such as MapReduce online [2] and Muppet [5]. In this paper, we highlight the need for dynamic load management in a distributed data stream processing system and present Enorm, a MapReduce-style data stream processing platform with the focus on techniques to achieve dynamic resource management, i.e. the ability to dynamically balance the workload among the running instances and scale the resource usage according to the runtime workload fluctuations. The original MapReduce framework is designed for batched processing and dynamic scaling can only be achieved between batches. To address this problem, we propose a MapReduce-style computation framework and a set of corresponding adaptation strategies that can perform dynamic scaling on the fly with low processing latency.

AB - There is a recent interest in building MapReduce-style platforms for fast data processing, such as MapReduce online [2] and Muppet [5]. In this paper, we highlight the need for dynamic load management in a distributed data stream processing system and present Enorm, a MapReduce-style data stream processing platform with the focus on techniques to achieve dynamic resource management, i.e. the ability to dynamically balance the workload among the running instances and scale the resource usage according to the runtime workload fluctuations. The original MapReduce framework is designed for batched processing and dynamic scaling can only be achieved between batches. To address this problem, we propose a MapReduce-style computation framework and a set of corresponding adaptation strategies that can perform dynamic scaling on the fly with low processing latency.

U2 - 10.1109/ICDEW.2015.7129537

DO - 10.1109/ICDEW.2015.7129537

M3 - Article in proceedings

SP - 10

EP - 13

BT - Proceedings of the 31st IEEE International Conference on Data Engineering Workshops

PB - IEEE

T2 - 31st IEEE International Conference on Data Engineering Workshops

Y2 - 13 April 2015 through 17 April 2015

ER -

Dynamic resource management in a MapReduce-style platform for fast data processing

Abstract

Conference

Access to Document

Fingerprint

Cite this