Building modern dataflow systems

Add to your list(s) Download to your calendar using vCal

• Frank McSherry (ETH Zürich)
• Thursday 12 July 2018, 11:00-12:00
• SW00, Computer Laboratory, William Gates Building.

If you have a question about this talk, please contact Marco Caballero.

Abstract: I’ll talk through the design and implementation of “timely dataflow in Rust”, an open-source project that extends and enriches the “timely dataflow” computational model first presented by the Naiad system, and the differential dataflow framework built on top of it. The project’s goal is to provide an near-zero overhead framework for data-parallel dataflow computation, and to this end it simplifies and unifies several of Naiad’s concepts through lossless abstractions that largely compile away. Our experience has been that timely dataflow programs give best-in-class performance, while still providing the experience of a medium-to-high level programming language. To support this, I’ll walk through the example of differential dataflow, an incremental re-computation framework which seems to out-perform the current crop of specialized data processing systems, in part due to its ability to provide general computation abstractions that compile down to sequential scans over carefully managed resources.

https://github.com/frankmcsherry/timely-dataflow
https://github.com/frankmcsherry/differential-dataflow

These projects reflect joint work with a great many people, including what was once the Naiad team at MSR -SV, the Systems Group at ETH Z ürich, and many other collaborators.

Bio: Frank McSherry received his PhD from the University of Washington, working with Anna Karlin on spectral analysis of data. He then spent twelve years at Microsoft Research’s Silicon Valley research center, working on topics ranging from differential privacy to data-parallel computation. He currently works at ETH Z ürich’s Systems Group on scalable stream processing and related topics.

This talk is part of the Computer Laboratory Systems Research Group Seminar series.

This talk is included in these lists:

• All Talks (aka the CURE list)
• CL's SRG seminar
• Cambridge Centre for Data-Driven Discovery (C2D3)
• Cambridge talks
• Chris Davis' list
• Computer Laboratory Systems Research Group Seminar
• Department of Computer Science and Technology talks and seminars
• Interested Talks
• SW00, Computer Laboratory, William Gates Building
• School of Technology
• Trust & Technology Initiative - interesting events
• bld31
• ndk22's list
• ob366-ai4er
• rp587

Note that ex-directory lists are not shown.