University of Cambridge > Talks.cam > Microsoft Research Cambridge, public talks > CIEL: a universal execution engine for distributed data-flow computing

CIEL: a universal execution engine for distributed data-flow computing

Add to your list(s) Download to your calendar using vCal

If you have a question about this talk, please contact Microsoft Research Cambridge Talks Admins.

Distributed data-flow frameworks, such as MapReduce and Dryad, have made it simpler to program a large number of computers at once. However, as many researchers have observed, existing frameworks provide a restricted programming model that inhibits the implementation of many naturally data-parallel algorithms. To address this problem, we have developed CIEL , which is a distributed execution engine that extends previous systems by allowing data-dependent control flow. This enables CIEL to natively execute iterative and recursive algorithms, including k-means clustering, PageRank and many sparse linear algebra methods.

In this talk, I will introduce CIEL , and describe the techniques that we use to make execution reliable. I will also describe Skywriting, which is a Turing-complete scripting language that runs directly on a CIEL cluster, and simplifies the construction of iterative jobs. We have deployed CIEL on a cloud computing platform, and I will present performance results that compare CIEL with Hadoop, the open-source implementation of MapReduce.

This talk is part of the Microsoft Research Cambridge, public talks series.

Tell a friend about this talk:

This talk is included in these lists:

Note that ex-directory lists are not shown.

 

© 2006-2024 Talks.cam, University of Cambridge. Contact Us | Help and Documentation | Privacy and Publicity