Integrating fault-tolerance and elasticity in a distributed data stream processing system

TY - GEN

T1 - Integrating fault-tolerance and elasticity in a distributed data stream processing system

AU - Madsen, Kasper Grud Skat

AU - Thyssen, Philip Pontoppidan

AU - Zhou, Yongluan

N1 - Conference code: 26

PY - 2014

Y1 - 2014

N2 - Recently there has been an increasing interest in building distributed platforms for processing of fast data streams. In this demonstration, we highlight the need for elasticity in distributed data stream processing systems and present Enorm, a data stream processing platform with focus on elasticity, i.e. the ability to dynamically scale resource usage according to the runtime workload fluctuations. In order to achieve dynamic scaling with minimal overhead and latency, we use an integrated approach for both fault-tolerance and elasticity. The idea is that both fault-tolerance and elasticity essentially require replicating or migrating computation states among different nodes. Integrating and sharing the state management operations between the two modules can not only provide abundant opportunities to reduce the system's runtime overhead but also simplify the system's architecture.

AB - Recently there has been an increasing interest in building distributed platforms for processing of fast data streams. In this demonstration, we highlight the need for elasticity in distributed data stream processing systems and present Enorm, a data stream processing platform with focus on elasticity, i.e. the ability to dynamically scale resource usage according to the runtime workload fluctuations. In order to achieve dynamic scaling with minimal overhead and latency, we use an integrated approach for both fault-tolerance and elasticity. The idea is that both fault-tolerance and elasticity essentially require replicating or migrating computation states among different nodes. Integrating and sharing the state management operations between the two modules can not only provide abundant opportunities to reduce the system's runtime overhead but also simplify the system's architecture.

U2 - 10.1145/2618243.2618288

DO - 10.1145/2618243.2618288

M3 - Article in proceedings

SN - 978-1-4503-2722-0

BT - Proceedings of the 26th International Conference on Scientific and Statistical Database Management

PB - Association for Computing Machinery

T2 - 26th International Conference on Scientific and Statistical Database Management

Y2 - 30 June 2014 through 2 July 2014

ER -