Dynamic resource management in a massively parallel stream processing engine

TY - GEN

T1 - Dynamic resource management in a massively parallel stream processing engine

AU - Madsen, Kasper Grud Skat

AU - Zhou, Yongluan

N1 - Conference code: 24

PY - 2015/10/17

Y1 - 2015/10/17

N2 - The emerging interest in Massively Parallel Stream Processing Engines (MPSPEs), which are able to process longstanding computations over data streams with ever-growing velocity at a large-scale cluster, calls for efficient dynamic resource management techniques to avoid any waste of resources and/or excessive processing latency. In this paper, we propose an approach to integrate dynamic resource management with passive fault-tolerance mechanisms in a MP-SPE so that we can harvest the checkpoints prepared for failure recovery to enhance the efficiency of dynamic load migrations. To maximize the opportunity of reusing checkpoints for fast load migration, we formally define a checkpoint allocation problem and provide a pragmatic algorithm to solve it. We implement all the proposed techniques on top of Apache Storm, an open-source MPSPE, and conduct extensive experiments using a real dataset to examine various aspects of our techniques. The results show that our techniques can greatly improve the efficiency of dynamic resource reconfiguration without imposing significant overhead or latency to the normal job execution.

AB - The emerging interest in Massively Parallel Stream Processing Engines (MPSPEs), which are able to process longstanding computations over data streams with ever-growing velocity at a large-scale cluster, calls for efficient dynamic resource management techniques to avoid any waste of resources and/or excessive processing latency. In this paper, we propose an approach to integrate dynamic resource management with passive fault-tolerance mechanisms in a MP-SPE so that we can harvest the checkpoints prepared for failure recovery to enhance the efficiency of dynamic load migrations. To maximize the opportunity of reusing checkpoints for fast load migration, we formally define a checkpoint allocation problem and provide a pragmatic algorithm to solve it. We implement all the proposed techniques on top of Apache Storm, an open-source MPSPE, and conduct extensive experiments using a real dataset to examine various aspects of our techniques. The results show that our techniques can greatly improve the efficiency of dynamic resource reconfiguration without imposing significant overhead or latency to the normal job execution.

U2 - 10.1145/2806416.2806449

DO - 10.1145/2806416.2806449

M3 - Article in proceedings

SP - 13

EP - 22

BT - Proceedings of the 24th ACM International Conference on Information and Knowledge Management

PB - Association for Computing Machinery

T2 - 24th ACM International Conference on Information and Knowledge Management

Y2 - 18 October 2015 through 23 October 2015

ER -