Dynamic resource management in a MapReduce-style platform for fast data processing

Kasper Grud Skat Madsen; Yongluan Zhou

doi:10.1109/ICDEW.2015.7129537

Dynamic resource management in a MapReduce-style platform for fast data processing

Kasper Grud Skat Madsen, Yongluan Zhou

1 Citationer (Scopus)

Abstract

There is a recent interest in building MapReduce-style platforms for fast data processing, such as MapReduce online [2] and Muppet [5]. In this paper, we highlight the need for dynamic load management in a distributed data stream processing system and present Enorm, a MapReduce-style data stream processing platform with the focus on techniques to achieve dynamic resource management, i.e. the ability to dynamically balance the workload among the running instances and scale the resource usage according to the runtime workload fluctuations. The original MapReduce framework is designed for batched processing and dynamic scaling can only be achieved between batches. To address this problem, we propose a MapReduce-style computation framework and a set of corresponding adaptation strategies that can perform dynamic scaling on the fly with low processing latency.

Originalsprog	Engelsk
Titel	Proceedings of the 31st IEEE International Conference on Data Engineering Workshops
Antal sider	4
Forlag	IEEE
Publikationsdato	19 jun. 2015
Sider	10-13
ISBN (Elektronisk)	978-1-4799-8442-8
DOI	https://doi.org/10.1109/ICDEW.2015.7129537
Status	Udgivet - 19 jun. 2015
Udgivet eksternt	Ja
Begivenhed	31st IEEE International Conference on Data Engineering Workshops - Seoul, Sydkorea Varighed: 13 apr. 2015 → 17 apr. 2015

Konference

Konference	31st IEEE International Conference on Data Engineering Workshops
Land/Område	Sydkorea
By	Seoul
Periode	13/04/2015 → 17/04/2015

Adgang til dokumentet

10.1109/ICDEW.2015.7129537

Citationsformater

@inproceedings{0d2c9b41528646659bac7649b6f450ea,

title = "Dynamic resource management in a MapReduce-style platform for fast data processing",

abstract = "There is a recent interest in building MapReduce-style platforms for fast data processing, such as MapReduce online [2] and Muppet [5]. In this paper, we highlight the need for dynamic load management in a distributed data stream processing system and present Enorm, a MapReduce-style data stream processing platform with the focus on techniques to achieve dynamic resource management, i.e. the ability to dynamically balance the workload among the running instances and scale the resource usage according to the runtime workload fluctuations. The original MapReduce framework is designed for batched processing and dynamic scaling can only be achieved between batches. To address this problem, we propose a MapReduce-style computation framework and a set of corresponding adaptation strategies that can perform dynamic scaling on the fly with low processing latency.",

author = "Madsen, {Kasper Grud Skat} and Yongluan Zhou",

year = "2015",

month = jun,

day = "19",

doi = "10.1109/ICDEW.2015.7129537",

language = "English",

pages = "10--13",

booktitle = "Proceedings of the 31st IEEE International Conference on Data Engineering Workshops",

publisher = "IEEE",

note = "31st IEEE International Conference on Data Engineering Workshops, ICDEW ; Conference date: 13-04-2015 Through 17-04-2015",

}

TY - GEN

T1 - Dynamic resource management in a MapReduce-style platform for fast data processing

AU - Madsen, Kasper Grud Skat

AU - Zhou, Yongluan

PY - 2015/6/19

Y1 - 2015/6/19

N2 - There is a recent interest in building MapReduce-style platforms for fast data processing, such as MapReduce online [2] and Muppet [5]. In this paper, we highlight the need for dynamic load management in a distributed data stream processing system and present Enorm, a MapReduce-style data stream processing platform with the focus on techniques to achieve dynamic resource management, i.e. the ability to dynamically balance the workload among the running instances and scale the resource usage according to the runtime workload fluctuations. The original MapReduce framework is designed for batched processing and dynamic scaling can only be achieved between batches. To address this problem, we propose a MapReduce-style computation framework and a set of corresponding adaptation strategies that can perform dynamic scaling on the fly with low processing latency.

AB - There is a recent interest in building MapReduce-style platforms for fast data processing, such as MapReduce online [2] and Muppet [5]. In this paper, we highlight the need for dynamic load management in a distributed data stream processing system and present Enorm, a MapReduce-style data stream processing platform with the focus on techniques to achieve dynamic resource management, i.e. the ability to dynamically balance the workload among the running instances and scale the resource usage according to the runtime workload fluctuations. The original MapReduce framework is designed for batched processing and dynamic scaling can only be achieved between batches. To address this problem, we propose a MapReduce-style computation framework and a set of corresponding adaptation strategies that can perform dynamic scaling on the fly with low processing latency.

U2 - 10.1109/ICDEW.2015.7129537

DO - 10.1109/ICDEW.2015.7129537

M3 - Article in proceedings

SP - 10

EP - 13

BT - Proceedings of the 31st IEEE International Conference on Data Engineering Workshops

PB - IEEE

T2 - 31st IEEE International Conference on Data Engineering Workshops

Y2 - 13 April 2015 through 17 April 2015

ER -

Dynamic resource management in a MapReduce-style platform for fast data processing

Abstract

Konference

Adgang til dokumentet

Fingeraftryk

Citationsformater