Stream fusion, to completeness

Oleg Kiselyov, Aggelos Biboudis, Nick Palladinos, Yannis Smaragdakis

Research output: Contribution to journalArticlepeer-review

1 Citation (Scopus)

Abstract

Stream processing is mainstream (again): Widely-used stream libraries are now available for virtually all modern OO and functional languages, from Java to C# to Scala to OCaml to Haskell. Yet expressivity and performance are still lacking. For instance, the popular, well-optimized Java 8 streams do not support the zip operator and are still an order of magnitude slower than hand-written loops. We present the first approach that represents the full generality of stream processing and eliminates overheads, via the use of staging. It is based on an unusually rich semantic model of stream interaction. We support any combination of zipping, nesting (or flat-mapping), sub-ranging, filtering, mapping - of finite or infinite streams. Our model captures idiosyncrasies that a programmer uses in optimizing stream pipelines, such as rate differences and the choice of a "for"vs. "while"loops. Our approach delivers hand-written-like code, but automatically. It explicitly avoids the reliance on black-box optimizers and sufficiently-smart compilers, offering highest, guaranteed and portable performance. Our approach relies on high-level concepts that are then readily mapped into an implementation. Accordingly, we have two distinct implementations: an OCaml stream library, staged via MetaOCaml, and a Scala library for the JVM, staged via LMS. In both cases, we derive libraries richer and simultaneously many tens of times faster than past work. We greatly exceed in performance the standard stream libraries available in Java, Scala and OCaml, including the well-optimized Java 8 streams.

Original languageEnglish
Pages (from-to)285-299
Number of pages15
JournalACM SIGPLAN Notices
Volume52
Issue number1
DOIs
Publication statusPublished - 2017 Jan

Keywords

  • Code generation
  • multi-stage programming
  • optimization
  • stream fusion
  • streams

ASJC Scopus subject areas

  • Computer Science(all)

Fingerprint

Dive into the research topics of 'Stream fusion, to completeness'. Together they form a unique fingerprint.

Cite this