Written mostly in Scala and with over 1000 contributors, Apache Spark has become the de facto standard for big data processing. In this talk, I will review the evolution of Spark for the last seven years and our experience using Scala as the main programming language in a high profile open source project with a distributed team. I will outline language features that we can't live without, and features we wish were designed differently. Last but not least, I will discuss how we at Databricks are leveraging native code to further improve performance.
{"title":"Spark and Scala (keynote)","authors":"Reynold Xin","doi":"10.1145/3136000.3148042","DOIUrl":"https://doi.org/10.1145/3136000.3148042","url":null,"abstract":"Written mostly in Scala and with over 1000 contributors, Apache Spark has become the de facto standard for big data processing. In this talk, I will review the evolution of Spark for the last seven years and our experience using Scala as the main programming language in a high profile open source project with a distributed team. I will outline language features that we can't live without, and features we wish were designed differently. Last but not least, I will discuss how we at Databricks are leveraging native code to further improve performance.","PeriodicalId":158126,"journal":{"name":"Proceedings of the 8th ACM SIGPLAN International Symposium on Scala","volume":"37 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-10-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116441543","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
A programming language is only as good at its tooling. Traditionally, tooling has always been an after-thought of language design since developing good tools take significant development efforts. Dotty is an experimental compiler for what will one day be called Scala 3, and Scala already has established and functional tooling. For Dotty to be seen as a viable alternative to Scala 2, it needs to deliver a developer experience at least as good. In particular, good support for Integrated Development Environments (IDEs) is crucial. In this paper we report our progress on providing IDE support for Dotty.
{"title":"Interactive development using the Dotty compiler (tool paper)","authors":"Guillaume Martres","doi":"10.1145/3136000.3136012","DOIUrl":"https://doi.org/10.1145/3136000.3136012","url":null,"abstract":"A programming language is only as good at its tooling. Traditionally, tooling has always been an after-thought of language design since developing good tools take significant development efforts. Dotty is an experimental compiler for what will one day be called Scala 3, and Scala already has established and functional tooling. For Dotty to be seen as a viable alternative to Scala 2, it needs to deliver a developer experience at least as good. In particular, good support for Integrated Development Environments (IDEs) is crucial. In this paper we report our progress on providing IDE support for Dotty.","PeriodicalId":158126,"journal":{"name":"Proceedings of the 8th ACM SIGPLAN International Symposium on Scala","volume":"10 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-10-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114183721","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
The Rust programming language demonstrates that memory safety can be achieved in a practical systems language, based on a sophisticated type system that controls object lifetimes and aliasing through notions of ownership and borrowing. While Scala has traditionally targeted only managed language runtimes, the ScalaNative effort makes Scala a viable low-level language as well. Thus, memory safety becomes an important concern, and the question bears asking what, if anything, Scala can learn from Rust. In addition, Rust's type system can encode forms of protocols, state machines, and session types, which would also be useful for Scala in general. A key challenge is that Rust's typing rules are inherently flow-sensitive, but Scala's type system is not. In this paper, we sketch one possible method of achieving static guarantees similar to Rust with only mild extensions to Scala's type system. Our solution is based on two components: First, the observation that continuation passing style (CPS) or monadic style can transform a flow-sensitive checking problem into a type-checking problem based on scopes. Second, on a previously presented type system extension with second-class values, which we use to model scope-based lifetimes.
{"title":"Rust-like borrowing with 2nd-class values (short paper)","authors":"Leo Osvald, Tiark Rompf","doi":"10.1145/3136000.3136010","DOIUrl":"https://doi.org/10.1145/3136000.3136010","url":null,"abstract":"The Rust programming language demonstrates that memory safety can be achieved in a practical systems language, based on a sophisticated type system that controls object lifetimes and aliasing through notions of ownership and borrowing. While Scala has traditionally targeted only managed language runtimes, the ScalaNative effort makes Scala a viable low-level language as well. Thus, memory safety becomes an important concern, and the question bears asking what, if anything, Scala can learn from Rust. In addition, Rust's type system can encode forms of protocols, state machines, and session types, which would also be useful for Scala in general. A key challenge is that Rust's typing rules are inherently flow-sensitive, but Scala's type system is not. In this paper, we sketch one possible method of achieving static guarantees similar to Rust with only mild extensions to Scala's type system. Our solution is based on two components: First, the observation that continuation passing style (CPS) or monadic style can transform a flow-sensitive checking problem into a type-checking problem based on scopes. Second, on a previously presented type system extension with second-class values, which we use to model scope-based lifetimes.","PeriodicalId":158126,"journal":{"name":"Proceedings of the 8th ACM SIGPLAN International Symposium on Scala","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-10-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131080767","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
The Dependent Object Types (DOT) calculus formalizes key features of Scala. The D<: calculus is the core of DOT. To date, presentations of D<: have used declarative, as opposed to algorithmic, typing and subtyping rules. Unfortunately, algorithmic typing for full D<: is known to be an undecidable problem. We explore the design space for a restricted version of D<: that has decidable typechecking. Even in this simplified D<:, algorithmic typing and subtyping are tricky, due to the "bad bounds" problem. The Scala compiler bypasses bad bounds at the cost of a loss in expressiveness in its type system. Based on the approach taken in the Scala compiler, we present the Step Typing and Step Subtyping relations for D<:. These relations are sound and decidable. They are not complete with respect to the original D<: typing rules.
{"title":"Towards algorithmic typing for DOT (short paper)","authors":"A. Nieto","doi":"10.1145/3136000.3136003","DOIUrl":"https://doi.org/10.1145/3136000.3136003","url":null,"abstract":"The Dependent Object Types (DOT) calculus formalizes key features of Scala. The D<: calculus is the core of DOT. To date, presentations of D<: have used declarative, as opposed to algorithmic, typing and subtyping rules. Unfortunately, algorithmic typing for full D<: is known to be an undecidable problem. We explore the design space for a restricted version of D<: that has decidable typechecking. Even in this simplified D<:, algorithmic typing and subtyping are tricky, due to the \"bad bounds\" problem. The Scala compiler bypasses bad bounds at the cost of a loss in expressiveness in its type system. Based on the approach taken in the Scala compiler, we present the Step Typing and Step Subtyping relations for D<:. These relations are sound and decidable. They are not complete with respect to the original D<: typing rules.","PeriodicalId":158126,"journal":{"name":"Proceedings of the 8th ACM SIGPLAN International Symposium on Scala","volume":"8 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-10-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"125258569","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Algebraic effects are an interesting way to structure effectful programs and offer new modularity properties. We present the Scala library Effekt, which is implemented in terms of a monad for multi-prompt delimited continuations and centered around capability passing. This makes the newly proposed feature of implicit function types a perfect fit for the syntax of our library. Basing the library design on capability passing and a polymorphic embedding of effect handlers furthermore opens up interesting dimensions of extensibility. Preliminary benchmarks comparing Effekt with an established library suggest significant speedups.
{"title":"Effekt: extensible algebraic effects in Scala (short paper)","authors":"J. Brachthäuser, Philipp Schuster","doi":"10.1145/3136000.3136007","DOIUrl":"https://doi.org/10.1145/3136000.3136007","url":null,"abstract":"Algebraic effects are an interesting way to structure effectful programs and offer new modularity properties. We present the Scala library Effekt, which is implemented in terms of a monad for multi-prompt delimited continuations and centered around capability passing. This makes the newly proposed feature of implicit function types a perfect fit for the syntax of our library. Basing the library design on capability passing and a polymorphic embedding of effect handlers furthermore opens up interesting dimensions of extensibility. Preliminary benchmarks comparing Effekt with an established library suggest significant speedups.","PeriodicalId":158126,"journal":{"name":"Proceedings of the 8th ACM SIGPLAN International Symposium on Scala","volume":"80 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-10-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"115305281","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Supercompilation is a program transformation technique that attempts to evaluate programs as much as possible at compile time. Supercompilation has been used for theorem proving, function inversion, and most notably optimization, especially of functional programs. However, the technique has numerous practical problems that prevent it from being applied in mainstream compilers. In this paper, we describe a framework that can be used for experimenting with supercompilation techniques. Our framework allows supercompilers to be constructed directly from an interpreter. The user specifies the interpreter using rewrite rules and the framework handles termination checking, generalization, and residualization. We demonstrate the approach by implementing a supercompiler for JavaScript.
{"title":"A Scala framework for supercompilation","authors":"Nathaniel Nystrom","doi":"10.1145/3136000.3136011","DOIUrl":"https://doi.org/10.1145/3136000.3136011","url":null,"abstract":"Supercompilation is a program transformation technique that attempts to evaluate programs as much as possible at compile time. Supercompilation has been used for theorem proving, function inversion, and most notably optimization, especially of functional programs. However, the technique has numerous practical problems that prevent it from being applied in mainstream compilers. In this paper, we describe a framework that can be used for experimenting with supercompilation techniques. Our framework allows supercompilers to be constructed directly from an interpreter. The user specifies the interpreter using rewrite rules and the framework handles termination checking, generalization, and residualization. We demonstrate the approach by implementing a supercompiler for JavaScript.","PeriodicalId":158126,"journal":{"name":"Proceedings of the 8th ACM SIGPLAN International Symposium on Scala","volume":"464 2","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-10-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131580474","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Quasiquotes have been shown to greatly simplify the task of metaprogramming. This is in part because they hide the data structures of the intermediate representation (IR), instead allowing metaprogrammers to use the concrete syntax of the language they manipulate. Scala has had "syntactic" quasiquotes for a long time, but still misses a statically-typed version like in MetaOCaml, Haskell and F#. This safer flavor of quasiquotes has been particularly useful for staging and domain-specific languages. In this paper we present Squid, a metaprogramming system for Scala that fills this gap. Squid quasiquotes are novel in three ways: they are the first statically-typed quasiquotes we know that allow code inspection (via pattern matching); they are implemented purely as a macro library, without modifications to the compiler; and they are reusable in the sense that they can manipulate different IRs. Adapting (or binding) a new IR to Squid is done simply by implementing a well-defined interface in the style of object algebras (i.e., tagless-final). We detail how Squid is implemented, leveraging the metaprogramming tools already offered by Scala, and show three application examples: the definition of a binding for a DSL in the style of LMS; a safe ANF conversion; and the introduction of type-safe, hygienic macros as an alternative to the current macro system.
{"title":"Squid: type-safe, hygienic, and reusable quasiquotes","authors":"L. Parreaux, A. Shaikhha, Christoph E. Koch","doi":"10.1145/3136000.3136005","DOIUrl":"https://doi.org/10.1145/3136000.3136005","url":null,"abstract":"Quasiquotes have been shown to greatly simplify the task of metaprogramming. This is in part because they hide the data structures of the intermediate representation (IR), instead allowing metaprogrammers to use the concrete syntax of the language they manipulate. Scala has had \"syntactic\" quasiquotes for a long time, but still misses a statically-typed version like in MetaOCaml, Haskell and F#. This safer flavor of quasiquotes has been particularly useful for staging and domain-specific languages. In this paper we present Squid, a metaprogramming system for Scala that fills this gap. Squid quasiquotes are novel in three ways: they are the first statically-typed quasiquotes we know that allow code inspection (via pattern matching); they are implemented purely as a macro library, without modifications to the compiler; and they are reusable in the sense that they can manipulate different IRs. Adapting (or binding) a new IR to Squid is done simply by implementing a well-defined interface in the style of object algebras (i.e., tagless-final). We detail how Squid is implemented, leveraging the metaprogramming tools already offered by Scala, and show three application examples: the definition of a binding for a DSL in the style of LMS; a safe ANF conversion; and the introduction of type-safe, hygienic macros as an alternative to the current macro system.","PeriodicalId":158126,"journal":{"name":"Proceedings of the 8th ACM SIGPLAN International Symposium on Scala","volume":"62 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-10-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116593349","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Aleksandar Prokopec, David Leopoldseder, Gilles Duboscq, Thomas Würthinger
Functional collection combinators are a neat and widely accepted data processing abstraction. However, their generic nature results in high abstraction overheads -- Scala collections are known to be notoriously slow for typical tasks. We show that proper optimizations in a JIT compiler can widely eliminate overheads imposed by these abstractions. Using the open-source Graal JIT compiler, we achieve speedups of up to 20x on collection workloads compared to the standard HotSpot C2 compiler. Consequently, a sufficiently aggressive JIT compiler allows the language compiler, such as Scalac, to focus on other concerns. In this paper, we show how optimizations, such as inlining, polymorphic inlining, and partial escape analysis, are combined in Graal to produce collections code that is optimal with respect to manually written code, or close to optimal. We argue why some of these optimizations are more effectively done by a JIT compiler. We then identify specific use-cases that most current JIT compilers do not optimize well, warranting special treatment from the language compiler.
{"title":"Making collection operations optimal with aggressive JIT compilation","authors":"Aleksandar Prokopec, David Leopoldseder, Gilles Duboscq, Thomas Würthinger","doi":"10.1145/3136000.3136002","DOIUrl":"https://doi.org/10.1145/3136000.3136002","url":null,"abstract":"Functional collection combinators are a neat and widely accepted data processing abstraction. However, their generic nature results in high abstraction overheads -- Scala collections are known to be notoriously slow for typical tasks. We show that proper optimizations in a JIT compiler can widely eliminate overheads imposed by these abstractions. Using the open-source Graal JIT compiler, we achieve speedups of up to 20x on collection workloads compared to the standard HotSpot C2 compiler. Consequently, a sufficiently aggressive JIT compiler allows the language compiler, such as Scalac, to focus on other concerns. In this paper, we show how optimizations, such as inlining, polymorphic inlining, and partial escape analysis, are combined in Graal to produce collections code that is optimal with respect to manually written code, or close to optimal. We argue why some of these optimizations are more effectively done by a JIT compiler. We then identify specific use-cases that most current JIT compilers do not optimize well, warranting special treatment from the language compiler.","PeriodicalId":158126,"journal":{"name":"Proceedings of the 8th ACM SIGPLAN International Symposium on Scala","volume":"63 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-10-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"123410296","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Type classes enable a powerful form of ad-hoc polymorphism which provide solutions to many programming design problems. Inspired by this, Scala programmers have striven to emulate them in the design of libraries like Scalaz and Cats. The natural encoding of type classes combines subtyping and implicits, both central features of Scala. However, this encoding has limitations. If the type class hierarchy branches, seemingly valid programs can hit implicit resolution failures. These failures must then be solved by explicitly passing the implicit arguments which is cumbersome and negates the advantages of type classes. In this paper we describe instances of this problem and show that they are not merely theoretical but often arise in practice. We also discuss and compare the space of solutions to this problem in Scala today and in the future.
{"title":"The limitations of type classes as subtyped implicits (short paper)","authors":"Adelbert Chang","doi":"10.1145/3136000.3136006","DOIUrl":"https://doi.org/10.1145/3136000.3136006","url":null,"abstract":"Type classes enable a powerful form of ad-hoc polymorphism which provide solutions to many programming design problems. Inspired by this, Scala programmers have striven to emulate them in the design of libraries like Scalaz and Cats. The natural encoding of type classes combines subtyping and implicits, both central features of Scala. However, this encoding has limitations. If the type class hierarchy branches, seemingly valid programs can hit implicit resolution failures. These failures must then be solved by explicitly passing the implicit arguments which is cumbersome and negates the advantages of type classes. In this paper we describe instances of this problem and show that they are not merely theoretical but often arise in practice. We also discuss and compare the space of solutions to this problem in Scala today and in the future.","PeriodicalId":158126,"journal":{"name":"Proceedings of the 8th ACM SIGPLAN International Symposium on Scala","volume":"35 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-10-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124009011","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Satisfiability modulo theory (SMT) consists of determining the satisfiability of logical formulas. It can reason in various formal theories, e.g., in linear integer or real arithmetic, first-order logic, or logics of arrays. An SMT solver is a program that implements the corresponding algorithms to automatically determine whether a logical formula is satisfiable. The SMTLIB initiative provides a common input and output format based on S-expressions for interacting with SMT solvers. We present ScalaSMT, a Scala library that provides an abstraction over the SMTLIB format. The library brings consistency and type safety to the textual and dynamically typed world of SMTLIB solver interaction. ScalaSMT relies on the SMTLIB input/output capabilities of the solvers and consequently provides access to numerous popular SMTLIB-compliant solvers such as Z3, CVC4, Yices, MathSat5 or SMTinterpol. ScalaSMT is easily extendable (SMTLIB commands and theories can be added) and configurable (SMTLIB-compliant solvers can be added). ScalaSMT fills a gap in the Scala library landscape by providing support for powerful logical reasoning algorithms.
{"title":"ScalaSMT: satisfiability modulo theory in Scala (tool paper)","authors":"F. Cassez, A. Sloane","doi":"10.1145/3136000.3136004","DOIUrl":"https://doi.org/10.1145/3136000.3136004","url":null,"abstract":"Satisfiability modulo theory (SMT) consists of determining the satisfiability of logical formulas. It can reason in various formal theories, e.g., in linear integer or real arithmetic, first-order logic, or logics of arrays. An SMT solver is a program that implements the corresponding algorithms to automatically determine whether a logical formula is satisfiable. The SMTLIB initiative provides a common input and output format based on S-expressions for interacting with SMT solvers. We present ScalaSMT, a Scala library that provides an abstraction over the SMTLIB format. The library brings consistency and type safety to the textual and dynamically typed world of SMTLIB solver interaction. ScalaSMT relies on the SMTLIB input/output capabilities of the solvers and consequently provides access to numerous popular SMTLIB-compliant solvers such as Z3, CVC4, Yices, MathSat5 or SMTinterpol. ScalaSMT is easily extendable (SMTLIB commands and theories can be added) and configurable (SMTLIB-compliant solvers can be added). ScalaSMT fills a gap in the Scala library landscape by providing support for powerful logical reasoning algorithms.","PeriodicalId":158126,"journal":{"name":"Proceedings of the 8th ACM SIGPLAN International Symposium on Scala","volume":"58 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-10-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"123444393","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：