Stream fusion, to completeness

Oleg Kiselyov, Aggelos Biboudis, Nick Palladinos, Yannis Smaragdakis

Research output: Chapter in Book/Report/Conference proceedingConference contribution

19 Citations (Scopus)


Stream processing is mainstream (again): Widely-used stream libraries are now available for virtually all modern OO and functional languages, from Java to C# to Scala to OCaml to Haskell. Yet expressivity and performance are still lacking. For instance, the popular, well-optimized Java 8 streams do not support the zip operator and are still an order of magnitude slower than hand-written loops. We present the first approach that represents the full generality of stream processing and eliminates overheads, via the use of staging. It is based on an unusually rich semantic model of stream interaction. We support any combination of zipping, nesting (or flat-mapping), sub-ranging, filtering, mapping-of finite or infinite streams. Our model captures idiosyncrasies that a programmer uses in optimizing stream pipelines, such as rate differences and the choice of a "for" vs. "while" loops. Our approach delivers hand-written-like code, but automatically. It explicitly avoids the reliance on black-box optimizers and sufficiently-smart compilers, offering highest, guaranteed and portable performance. Our approach relies on high-level concepts that are then readily mapped into an implementation. Accordingly, we have two distinct implementations: an OCaml stream library, staged via MetaOCaml, and a Scala library for the JVM, staged via LMS. In both cases, we derive libraries richer and simultaneously many tens of times faster than past work. We greatly exceed in performance the standard stream libraries available in Java, Scala and OCaml, including the well-optimized Java 8 streams.

Original languageEnglish
Title of host publicationPOPL 2017 - Proceedings of the 44th ACM SIGPLAN Symposium on Principles of Programming Languages
EditorsAndrew D. Gordon, Giuseppe Castagna
PublisherAssociation for Computing Machinery
Number of pages15
ISBN (Electronic)9781450346603
Publication statusPublished - 2017 Jan 1
Event44th ACM SIGPLAN Symposium on Principles of Programming Languages, POPL 2017 - Paris, France
Duration: 2017 Jan 152017 Jan 21

Publication series

NameConference Record of the Annual ACM Symposium on Principles of Programming Languages
ISSN (Print)0730-8566


Other44th ACM SIGPLAN Symposium on Principles of Programming Languages, POPL 2017


  • Code generation
  • Multi-stage programming
  • Optimization
  • Stream Fusion
  • Streams

ASJC Scopus subject areas

  • Software

Fingerprint Dive into the research topics of 'Stream fusion, to completeness'. Together they form a unique fingerprint.

Cite this