Enorm: efficient window-based computation in large-scale distributed stream processing systems

TY - GEN

T1 - Enorm

T2 - 10th ACM International Conference on Distributed and Event-Based Systems

AU - Madsen, Kasper Grud Skat

AU - Zhou, Yongluan

AU - Su, Li

PY - 2016/6/13

Y1 - 2016/6/13

N2 - Modern distributed stream processing systems (DSPS), such as Storm, typically provide a flexible programming model, where computation is specified as complicated UDFs and data is opaque to the system. While such a programming framework provides very high flexibility to the developers, it does not provide much semantic information to the system and hence it is hard to perform optimizations that has already been proved very effective in conventional stream systems. Examples include sharing computation among overlapping windows, co-partitioning operators to save communication overhead and efficient state migration during load balancing. In lieu of these challenges, we propose a new framework, which is designed to expose sufficient semantic information of the applications to enable the aforementioned effective optimizations, while on the other hand, maintaining the flexibility of Storm's original programming framework. Furthermore, we present new optimization algorithms to minimize the communication cost and state migration overhead for dynamic load balancing. We implement our framework on top of Storm and run an extensive experimental study to verify its effectiveness.

AB - Modern distributed stream processing systems (DSPS), such as Storm, typically provide a flexible programming model, where computation is specified as complicated UDFs and data is opaque to the system. While such a programming framework provides very high flexibility to the developers, it does not provide much semantic information to the system and hence it is hard to perform optimizations that has already been proved very effective in conventional stream systems. Examples include sharing computation among overlapping windows, co-partitioning operators to save communication overhead and efficient state migration during load balancing. In lieu of these challenges, we propose a new framework, which is designed to expose sufficient semantic information of the applications to enable the aforementioned effective optimizations, while on the other hand, maintaining the flexibility of Storm's original programming framework. Furthermore, we present new optimization algorithms to minimize the communication cost and state migration overhead for dynamic load balancing. We implement our framework on top of Storm and run an extensive experimental study to verify its effectiveness.

U2 - 10.1145/2933267.2933315

DO - 10.1145/2933267.2933315

M3 - Article in proceedings

SN - 978-1-4503-4021-2

SP - 37

EP - 48

BT - Proceedings of the 10th ACM International Conference on Distributed and Event-based Systems

PB - Association for Computing Machinery

CY - New York, NY, USA

Y2 - 20 June 2016 through 24 June 2016

ER -