Skip to content

Gather real-world feedback about 2.12.3 performance #392

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
retronym opened this issue Jun 5, 2017 · 10 comments
Closed

Gather real-world feedback about 2.12.3 performance #392

retronym opened this issue Jun 5, 2017 · 10 comments
Assignees
@retronym
Labels
help-wanted t:performance
Milestone
2.12.3

Comments

@retronym
Copy link
Member

retronym commented Jun 5, 2017 •
edited
Loading

Here are instructions to run a comparative benchmark of the performance of the 2.12.2 and 2.12.3-SNAPSHOT compilers over your project sources. We're interested in finding out real-world numbers to complement our automatically run benchmarks.

The instructions below assume sbt as your build tool. Users of other build tools can also benchmark their project, providing they first create an "args file" containing the command line to the compiler.

Create an sbt plugin to export the compiler command line

// ~/.sbt/0.13/plugins/ArgsFile.scala
package io.github.retronym

import sbt._
import Keys._

object SbtArgsFilePlugin extends AutoPlugin {
  override def trigger = allRequirements
  override def requires = sbt.plugins.JvmPlugin
  val argsFileContents = taskKey[String]("Contents of file suitable for `scalac @args.txt`")
  val argsFile = taskKey[Unit]("Write compiler command line into an args file suitable for `scalac @target/compile.args`")
  override lazy val projectSettings = List(Compile, Test).flatMap(c => inConfig(c)(Seq(
    argsFileContents := {
      val sourcesValue = sources.value
      val depdependencyClasspathValue = dependencyClasspath.value
      val cp = if (depdependencyClasspathValue.isEmpty) Nil else ("-classpath" :: depdependencyClasspathValue.map(_.data.toString).mkString(":") :: Nil)
      val result = (scalacOptions.value.toList ::: List("-d", classDirectory.value) ::: cp ::: sourcesValue.distinct.toList).mkString("\n")
      if (sourcesValue.isEmpty) "" else result
    },
    argsFile := {
      val f = target.value / (c.name + ".args")
      val contents = argsFileContents.value
      val log = streams.value.log
      if (!contents.isEmpty) {
        IO.write(f, contents)
        log.info("Wrote compiler comand line to: " + f.getAbsolutePath)
      }
    }
  )))
}

Export compiler command line

We'll use the akka-actor subproject of Akka to demonstrate.

% cd /code
% git clone https://github.com/akka/akka.git; cd akka
% sbt ++2.12.2 akka-actor/argsFile
...
[info] Wrote compiler comand line to: /code/akka/akka-actor/target/compile.args
[success] Total time: 0 s, completed 06/06/2017 9:23:35 AM

Clone the compiler-benchmark project

% cd /code
% git clone https://github.com/scala/compiler-benchmark.git; cd compiler-benchmark

Configure a SBT resolver to access nightly builds

// ~/.sbt/0.13/resolver.sbt

resolvers ++= (
  if (scalaVersion.value.contains("-bin"))
     List("scala-integration" at "https://scala-ci.typesafe.com/artifactory/scala-integration/")
  else Nil
)

Quiesce your machine

Find latest nightly build

% curl https://scala-ci.typesafe.com/job/scala-2.12.x-integrate-bootstrap/lastSuccessfulBuild/artifact/jenkins.properties/*view*/
version=2.12.3-bin-bd6294d
sbtDistVersionOverride=-Dproject.version=2.12.3-bin-bd6294d

Use the first version number in the next step.

Execute the benchmark

% for i in 2.12.2 2.12.3-bin-37663b0-SNAPSHOT; do \
  sbt "set scalaVersion in compilation := \"$i\"" 'hot -p source=@/code/akka/akka-actor/target/compile.args' || break; \
done
...
[info] # Run complete. Total time: 00:04:05
[info] 
[info] Benchmark                                   (corpusVersion)  (extraArgs)                            (source)    Mode  Cnt     Score     Error  Units
[info] HotScalacBenchmark.compile                           latest               @/code/akka/akka-actor.args  sample   24  5089.438 ± 265.585  ms/op
[info] HotScalacBenchmark.compile:compile·p0.00             latest               @/code/akka/akka-actor.args  sample       4815.061            ms/op
[info] HotScalacBenchmark.compile:compile·p0.50             latest               @/code/akka/akka-actor.args  sample       4919.919            ms/op
[info] HotScalacBenchmark.compile:compile·p0.90             latest               @/code/akka/akka-actor.args  sample       5511.315            ms/op
[info] HotScalacBenchmark.compile:compile·p0.95             latest               @/code/akka/akka-actor.args  sample       6201.278            ms/op
[info] HotScalacBenchmark.compile:compile·p0.99             latest               @/code/akka/akka-actor.args  sample       6358.565            ms/op
[info] HotScalacBenchmark.compile:compile·p0.999            latest               @/code/akka/akka-actor.args  sample       6358.565            ms/op
[info] HotScalacBenchmark.compile:compile·p0.9999           latest               @/code/akka/akka-actor.args  sample       6358.565            ms/op
[info] HotScalacBenchmark.compile:compile·p1.00             latest               @/code/akka/akka-actor.args  sample       6358.565            ms/op
[success] Total time: 247 s, completed 05/06/2017 6:39:07 PM

[info] # Run complete. Total time: 00:04:14
[info] 
[info] Benchmark                                   (corpusVersion)  (extraArgs)                            (source)    Mode  Cnt     Score     Error  Units
[info] HotScalacBenchmark.compile                           latest               @/code/akka/akka-actor.args  sample   30  4174.870 ± 170.827  ms/op
[info] HotScalacBenchmark.compile:compile·p0.00             latest               @/code/akka/akka-actor.args  sample       4034.920            ms/op
[info] HotScalacBenchmark.compile:compile·p0.50             latest               @/code/akka/akka-actor.args  sample       4099.932            ms/op
[info] HotScalacBenchmark.compile:compile·p0.90             latest               @/code/akka/akka-actor.args  sample       4363.754            ms/op
[info] HotScalacBenchmark.compile:compile·p0.95             latest               @/code/akka/akka-actor.args  sample       5027.293            ms/op
[info] HotScalacBenchmark.compile:compile·p0.99             latest               @/code/akka/akka-actor.args  sample       5276.434            ms/op
[info] HotScalacBenchmark.compile:compile·p0.999            latest               @/code/akka/akka-actor.args  sample       5276.434            ms/op
[info] HotScalacBenchmark.compile:compile·p0.9999           latest               @/code/akka/akka-actor.args  sample       5276.434            ms/op
[info] HotScalacBenchmark.compile:compile·p1.00             latest               @/code/akka/akka-actor.args  sample       5276.434            ms/op
[success] Total time: 277 s, completed 05/06/2017 6:43:50 PM

Let us know your results

In the comments below.

While you are at it
@retronym retronym self-assigned this Jun 5, 2017
@retronym retronym added this to the 2.12.3 milestone Jun 6, 2017
@olafurpg
Copy link

I ran the benchmark on the trees module in the scalameta repo. I chose that module because it synthesizes a lot of code with paradise macro annotations. The speedup factor is x0.83 (~12s down to ~10s), which seems consistent with the reported numbers in http://developer.lightbend.com/blog/2017-06-12-faster-scala-compiler/

2.12.2

[info] Benchmark                                   (corpusVersion)  (extraArgs)                                                                  (source)    Mode  Cnt      Score     Error  Units
[info] HotScalacBenchmark.compile                           latest               @/Users/ollie/dev/code/scalameta/scalameta/trees/jvm/target/compile.args  sample   30  12016.401 ± 317.487  ms/op
[info] HotScalacBenchmark.compile:compile·p0.00             latest               @/Users/ollie/dev/code/scalameta/scalameta/trees/jvm/target/compile.args  sample       11341.398            ms/op
[info] HotScalacBenchmark.compile:compile·p0.50             latest               @/Users/ollie/dev/code/scalameta/scalameta/trees/jvm/target/compile.args  sample       11970.544            ms/op
[info] HotScalacBenchmark.compile:compile·p0.90             latest               @/Users/ollie/dev/code/scalameta/scalameta/trees/jvm/target/compile.args  sample       12683.575            ms/op
[info] HotScalacBenchmark.compile:compile·p0.95             latest               @/Users/ollie/dev/code/scalameta/scalameta/trees/jvm/target/compile.args  sample       12952.011            ms/op
[info] HotScalacBenchmark.compile:compile·p0.99             latest               @/Users/ollie/dev/code/scalameta/scalameta/trees/jvm/target/compile.args  sample       12952.011            ms/op
[info] HotScalacBenchmark.compile:compile·p0.999            latest               @/Users/ollie/dev/code/scalameta/scalameta/trees/jvm/target/compile.args  sample       12952.011            ms/op
[info] HotScalacBenchmark.compile:compile·p0.9999           latest               @/Users/ollie/dev/code/scalameta/scalameta/trees/jvm/target/compile.args  sample       12952.011            ms/op
[info] HotScalacBenchmark.compile:compile·p1.00             latest               @/Users/ollie/dev/code/scalameta/scalameta/trees/jvm/target/compile.args  sample       12952.011            ms/op

2.12.3-bin-0b150b7

[info] Benchmark                                   (corpusVersion)  (extraArgs)                                                                  (source)    Mode  Cnt      Score     Error  Units
[info] HotScalacBenchmark.compile                           latest               @/Users/ollie/dev/code/scalameta/scalameta/trees/jvm/target/compile.args  sample   47  10027.421 ± 254.630  ms/op
[info] HotScalacBenchmark.compile:compile·p0.00             latest               @/Users/ollie/dev/code/scalameta/scalameta/trees/jvm/target/compile.args  sample        9512.681            ms/op
[info] HotScalacBenchmark.compile:compile·p0.50             latest               @/Users/ollie/dev/code/scalameta/scalameta/trees/jvm/target/compile.args  sample        9881.780            ms/op
[info] HotScalacBenchmark.compile:compile·p0.90             latest               @/Users/ollie/dev/code/scalameta/scalameta/trees/jvm/target/compile.args  sample       10583.068            ms/op
[info] HotScalacBenchmark.compile:compile·p0.95             latest               @/Users/ollie/dev/code/scalameta/scalameta/trees/jvm/target/compile.args  sample       11069.607            ms/op
[info] HotScalacBenchmark.compile:compile·p0.99             latest               @/Users/ollie/dev/code/scalameta/scalameta/trees/jvm/target/compile.args  sample       12280.922            ms/op
[info] HotScalacBenchmark.compile:compile·p0.999            latest               @/Users/ollie/dev/code/scalameta/scalameta/trees/jvm/target/compile.args  sample       12280.922            ms/op
[info] HotScalacBenchmark.compile:compile·p0.9999           latest               @/Users/ollie/dev/code/scalameta/scalameta/trees/jvm/target/compile.args  sample       12280.922            ms/op
[info] HotScalacBenchmark.compile:compile·p1.00             latest               @/Users/ollie/dev/code/scalameta/scalameta/trees/jvm/target/compile.args  sample       12280.922            ms/op

@olafurpg olafurpg mentioned this issue Jun 15, 2017
Closed
@espenhw
Copy link

espenhw commented Jun 17, 2017

Results for our proprietary application, ~70 kLoC in a mixture of styles: ~37s down to ~30s again gives a speedup factor of x0.83.

As an additional data point, the cold benchmark shows a x0.87 speedup, ~63s down to ~55s.

2.12.2

[info] Benchmark                                   (corpusVersion)  (extraArgs)        (source)    Mode  Cnt      Score     Error  Units
[info] HotScalacBenchmark.compile                           latest               @/compile.args  sample   30  36833.818 ± 976.917  ms/op
[info] HotScalacBenchmark.compile:compile·p0.00             latest               @/compile.args  sample       35500.589            ms/op
[info] HotScalacBenchmark.compile:compile·p0.50             latest               @/compile.args  sample       36339.450            ms/op
[info] HotScalacBenchmark.compile:compile·p0.90             latest               @/compile.args  sample       38634.573            ms/op
[info] HotScalacBenchmark.compile:compile·p0.95             latest               @/compile.args  sample       41218.264            ms/op
[info] HotScalacBenchmark.compile:compile·p0.99             latest               @/compile.args  sample       41808.822            ms/op
[info] HotScalacBenchmark.compile:compile·p0.999            latest               @/compile.args  sample       41808.822            ms/op
[info] HotScalacBenchmark.compile:compile·p0.9999           latest               @/compile.args  sample       41808.822            ms/op
[info] HotScalacBenchmark.compile:compile·p1.00             latest               @/compile.args  sample       41808.822            ms/op

2.12.3-bin-d1ec01a

[info] Benchmark                                   (corpusVersion)  (extraArgs)        (source)    Mode  Cnt      Score     Error  Units
[info] HotScalacBenchmark.compile                           latest               @/compile.args  sample   30  30131.880 ± 239.360  ms/op
[info] HotScalacBenchmark.compile:compile·p0.00             latest               @/compile.args  sample       29360.128            ms/op
[info] HotScalacBenchmark.compile:compile·p0.50             latest               @/compile.args  sample       30115.103            ms/op
[info] HotScalacBenchmark.compile:compile·p0.90             latest               @/compile.args  sample       30594.931            ms/op
[info] HotScalacBenchmark.compile:compile·p0.95             latest               @/compile.args  sample       30865.044            ms/op
[info] HotScalacBenchmark.compile:compile·p0.99             latest               @/compile.args  sample       31104.958            ms/op
[info] HotScalacBenchmark.compile:compile·p0.999            latest               @/compile.args  sample       31104.958            ms/op
[info] HotScalacBenchmark.compile:compile·p0.9999           latest               @/compile.args  sample       31104.958            ms/op
[info] HotScalacBenchmark.compile:compile·p1.00             latest               @/compile.args  sample       31104.958            ms/op

@xuwei-k
Copy link
Contributor

xuwei-k commented Jun 18, 2017 •
edited
Loading

scalaz-core

series/7.3.x branch 2017/06/18 latest

full log

2.12.2

[info] Benchmark                                   (corpusVersion)  (extraArgs)                                                 (source)    Mode  Cnt      Score     Error  Units
[info] HotScalacBenchmark.compile                           latest               @/Users/kenji/scalaz/scalaz/core/jvm/target/compile.args  sample   30  35142.675 ± 379.597  ms/op
[info] HotScalacBenchmark.compile:compile·p0.00             latest               @/Users/kenji/scalaz/scalaz/core/jvm/target/compile.args  sample       34292.630            ms/op
[info] HotScalacBenchmark.compile:compile·p0.50             latest               @/Users/kenji/scalaz/scalaz/core/jvm/target/compile.args  sample       35097.936            ms/op
[info] HotScalacBenchmark.compile:compile·p0.90             latest               @/Users/kenji/scalaz/scalaz/core/jvm/target/compile.args  sample       35836.133            ms/op
[info] HotScalacBenchmark.compile:compile·p0.95             latest               @/Users/kenji/scalaz/scalaz/core/jvm/target/compile.args  sample       36393.137            ms/op
[info] HotScalacBenchmark.compile:compile·p0.99             latest               @/Users/kenji/scalaz/scalaz/core/jvm/target/compile.args  sample       36909.875            ms/op
[info] HotScalacBenchmark.compile:compile·p0.999            latest               @/Users/kenji/scalaz/scalaz/core/jvm/target/compile.args  sample       36909.875            ms/op
[info] HotScalacBenchmark.compile:compile·p0.9999           latest               @/Users/kenji/scalaz/scalaz/core/jvm/target/compile.args  sample       36909.875            ms/op
[info] HotScalacBenchmark.compile:compile·p1.00             latest               @/Users/kenji/scalaz/scalaz/core/jvm/target/compile.args  sample       36909.875            ms/op

2.12.3-bin-d1ec01a

[info] Benchmark                                   (corpusVersion)  (extraArgs)                                                 (source)    Mode  Cnt      Score     Error  Units
[info] HotScalacBenchmark.compile                           latest               @/Users/kenji/scalaz/scalaz/core/jvm/target/compile.args  sample   30  29501.057 ± 280.027  ms/op
[info] HotScalacBenchmark.compile:compile·p0.00             latest               @/Users/kenji/scalaz/scalaz/core/jvm/target/compile.args  sample       28823.257            ms/op
[info] HotScalacBenchmark.compile:compile·p0.50             latest               @/Users/kenji/scalaz/scalaz/core/jvm/target/compile.args  sample       29527.900            ms/op
[info] HotScalacBenchmark.compile:compile·p0.90             latest               @/Users/kenji/scalaz/scalaz/core/jvm/target/compile.args  sample       30152.013            ms/op
[info] HotScalacBenchmark.compile:compile·p0.95             latest               @/Users/kenji/scalaz/scalaz/core/jvm/target/compile.args  sample       30308.041            ms/op
[info] HotScalacBenchmark.compile:compile·p0.99             latest               @/Users/kenji/scalaz/scalaz/core/jvm/target/compile.args  sample       30400.315            ms/op
[info] HotScalacBenchmark.compile:compile·p0.999            latest               @/Users/kenji/scalaz/scalaz/core/jvm/target/compile.args  sample       30400.315            ms/op
[info] HotScalacBenchmark.compile:compile·p0.9999           latest               @/Users/kenji/scalaz/scalaz/core/jvm/target/compile.args  sample       30400.315            ms/op
[info] HotScalacBenchmark.compile:compile·p1.00             latest               @/Users/kenji/scalaz/scalaz/core/jvm/target/compile.args  sample       30400.315            ms/op

@jvican
Copy link
Member

jvican commented Jun 18, 2017 •
edited
Loading

sbt core-macros module

I wanted to see the results for macro libraries' code. The new 2.12.3 is 48% faster in percentile 0.99. I have cleaned the environment to make sure the benchmark is reproducible. I have fixed the min and max frequency to 2.5GHz in an Intel(R) Core(TM) i7-6600U CPU whose max CPU is 3.4GHz. At the end of the benchmarks I attach my cpuinfo and meminfo.

Scala 2.12.2

[info] HotScalacBenchmark.compile                           latest               @/data/rw/code/scala/sbt/core-macros/target/compile.args  sample  273  1132.547 ± 5.967  ms/op
[info] HotScalacBenchmark.compile:compile·p0.00             latest               @/data/rw/code/scala/sbt/core-macros/target/compile.args  sample       1073.742          ms/op
[info] HotScalacBenchmark.compile:compile·p0.50             latest               @/data/rw/code/scala/sbt/core-macros/target/compile.args  sample       1128.268          ms/op
[info] HotScalacBenchmark.compile:compile·p0.90             latest               @/data/rw/code/scala/sbt/core-macros/target/compile.args  sample       1169.372          ms/op
[info] HotScalacBenchmark.compile:compile·p0.95             latest               @/data/rw/code/scala/sbt/core-macros/target/compile.args  sample       1188.875          ms/op
[info] HotScalacBenchmark.compile:compile·p0.99             latest               @/data/rw/code/scala/sbt/core-macros/target/compile.args  sample       1213.874          ms/op
[info] HotScalacBenchmark.compile:compile·p0.999            latest               @/data/rw/code/scala/sbt/core-macros/target/compile.args  sample       1291.846          ms/op
[info] HotScalacBenchmark.compile:compile·p0.9999           latest               @/data/rw/code/scala/sbt/core-macros/target/compile.args  sample       1291.846          ms/op
[info] HotScalacBenchmark.compile:compile·p1.00             latest               @/data/rw/code/scala/sbt/core-macros/target/compile.args  sample       1291.846          ms/op

Scala 2.12.3-bin-d1ec01a

[info] Benchmark                                   (corpusVersion)  (extraArgs)                                                  (source)    Mode  Cnt    Score   Error  Units
[info] HotScalacBenchmark.compile                           latest               @/data/rw/code/scala/sbt/core-macros/target/compile.args  sample  551  561.365 ± 2.816  ms/op
[info] HotScalacBenchmark.compile:compile·p0.00             latest               @/data/rw/code/scala/sbt/core-macros/target/compile.args  sample       530.579          ms/op
[info] HotScalacBenchmark.compile:compile·p0.50             latest               @/data/rw/code/scala/sbt/core-macros/target/compile.args  sample       557.842          ms/op
[info] HotScalacBenchmark.compile:compile·p0.90             latest               @/data/rw/code/scala/sbt/core-macros/target/compile.args  sample       586.154          ms/op
[info] HotScalacBenchmark.compile:compile·p0.95             latest               @/data/rw/code/scala/sbt/core-macros/target/compile.args  sample       597.898          ms/op
[info] HotScalacBenchmark.compile:compile·p0.99             latest               @/data/rw/code/scala/sbt/core-macros/target/compile.args  sample       631.033          ms/op
[info] HotScalacBenchmark.compile:compile·p0.999            latest               @/data/rw/code/scala/sbt/core-macros/target/compile.args  sample       677.380          ms/op
[info] HotScalacBenchmark.compile:compile·p0.9999           latest               @/data/rw/code/scala/sbt/core-macros/target/compile.args  sample       677.380          ms/op
[info] HotScalacBenchmark.compile:compile·p1.00             latest               @/data/rw/code/scala/sbt/core-macros/target/compile.args  sample       677.380          ms/op
[success] Total time: 653 s, completed Jun 18, 2017 4:07:54 PM

Cpupower

tribox# cpupower frequency-info
analyzing CPU 0:
  driver: intel_pstate
  CPUs which run at the same hardware frequency: 0
  CPUs which need to have their frequency coordinated by software: 0
  maximum transition latency:  Cannot determine or is not supported.
  hardware limits: 400 MHz - 3.40 GHz
  available cpufreq governors: performance powersave
  current policy: frequency should be within 2.50 GHz and 2.50 GHz.
                  The governor "performance" may decide which speed to use
                  within this range.
  current CPU frequency: 2.50 GHz (asserted by call to hardware)
  boost state support:
    Supported: yes
    Active: yes

/proc/cpuinfo

tribox# cat /proc/cpuinfo 
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 78
model name	: Intel(R) Core(TM) i7-6600U CPU @ 2.60GHz
stepping	: 3
microcode	: 0x88
cpu MHz		: 2499.902
cache size	: 4096 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 2
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 22
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp
bugs		:
bogomips	: 5618.00
clflush size	: 64
cache_alignment	: 64
address sizes	: 39 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 78
model name	: Intel(R) Core(TM) i7-6600U CPU @ 2.60GHz
stepping	: 3
microcode	: 0x88
cpu MHz		: 2499.902
cache size	: 4096 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 2
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 22
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp
bugs		:
bogomips	: 5619.35
clflush size	: 64
cache_alignment	: 64
address sizes	: 39 bits physical, 48 bits virtual
power management:

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 78
model name	: Intel(R) Core(TM) i7-6600U CPU @ 2.60GHz
stepping	: 3
microcode	: 0x88
cpu MHz		: 2499.902
cache size	: 4096 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 2
apicid		: 1
initial apicid	: 1
fpu		: yes
fpu_exception	: yes
cpuid level	: 22
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp
bugs		:
bogomips	: 5620.98
clflush size	: 64
cache_alignment	: 64
address sizes	: 39 bits physical, 48 bits virtual
power management:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 78
model name	: Intel(R) Core(TM) i7-6600U CPU @ 2.60GHz
stepping	: 3
microcode	: 0x88
cpu MHz		: 2499.902
cache size	: 4096 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 2
apicid		: 3
initial apicid	: 3
fpu		: yes
fpu_exception	: yes
cpuid level	: 22
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp
bugs		:
bogomips	: 5619.42
clflush size	: 64
cache_alignment	: 64
address sizes	: 39 bits physical, 48 bits virtual
power management:

/proc/meminfo

MemTotal:       20430272 kB
MemFree:        12453344 kB
MemAvailable:   18396672 kB
Buffers:            4228 kB
Cached:          6259632 kB
SwapCached:            0 kB
Active:          5007704 kB
Inactive:        2550424 kB
Active(anon):    1008356 kB
Inactive(anon):   445756 kB
Active(file):    3999348 kB
Inactive(file):  2104668 kB
Unevictable:          32 kB
Mlocked:              32 kB
SwapTotal:      12582908 kB
SwapFree:       12582908 kB
Dirty:              6608 kB
Writeback:             0 kB
AnonPages:       1266372 kB
Mapped:           391608 kB
Shmem:            159844 kB
Slab:             265392 kB
SReclaimable:     205904 kB
SUnreclaim:        59488 kB
KernelStack:        6048 kB
PageTables:        18360 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    22798044 kB
Committed_AS:    3871752 kB
VmallocTotal:   34359738367 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
HardwareCorrupted:     0 kB
AnonHugePages:    227328 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:      204204 kB
DirectMap2M:    14340096 kB
DirectMap1G:     6291456 kB

Computer

Thinkpad t460s, connected to AC.

@DarkDimius
Copy link

DarkDimius commented Jun 18, 2017 via email

@jvican
Copy link
Member

jvican commented Jun 18, 2017 •
edited
Loading

sbt main module

With the same environment as before, I've benchmarked the compilation of the main module in sbt (the one that contains lots of calls to sbt macros and has Defaults.scala). The new version 2.12.3 is 22.5% faster in p0.99. This module has 17KLOC.

Scala 2.12.2

[info] Benchmark                                   (corpusVersion)  (extraArgs)                                           (source)    Mode  Cnt     Score     Error  Units
[info] HotScalacBenchmark.compile                           latest               @/data/rw/code/scala/sbt/main/target/compile.args  sample   40  10089.818 ± 174.993  ms/op
[info] HotScalacBenchmark.compile:compile·p0.00             latest               @/data/rw/code/scala/sbt/main/target/compile.args  sample        9646.899            ms/op
[info] HotScalacBenchmark.compile:compile·p0.50             latest               @/data/rw/code/scala/sbt/main/target/compile.args  sample       10032.775            ms/op
[info] HotScalacBenchmark.compile:compile·p0.90             latest               @/data/rw/code/scala/sbt/main/target/compile.args  sample       10507.570            ms/op
[info] HotScalacBenchmark.compile:compile·p0.95             latest               @/data/rw/code/scala/sbt/main/target/compile.args  sample       10933.712            ms/op
[info] HotScalacBenchmark.compile:compile·p0.99             latest               @/data/rw/code/scala/sbt/main/target/compile.args  sample       11056.185            ms/op
[info] HotScalacBenchmark.compile:compile·p0.999            latest               @/data/rw/code/scala/sbt/main/target/compile.args  sample       11056.185            ms/op
[info] HotScalacBenchmark.compile:compile·p0.9999           latest               @/data/rw/code/scala/sbt/main/target/compile.args  sample       11056.185            ms/op
[info] HotScalacBenchmark.compile:compile·p1.00             latest               @/data/rw/code/scala/sbt/main/target/compile.args  sample       11056.185            ms/op

Scala 2.12.3-bin-d1ec01a

[info] Benchmark                                   (corpusVersion)  (extraArgs)                                           (source)    Mode  Cnt     Score     Error  Units
[info] HotScalacBenchmark.compile                           latest               @/data/rw/code/scala/sbt/main/target/compile.args  sample   60  7825.872 ± 110.925  ms/op
[info] HotScalacBenchmark.compile:compile·p0.00             latest               @/data/rw/code/scala/sbt/main/target/compile.args  sample       7407.141            ms/op
[info] HotScalacBenchmark.compile:compile·p0.50             latest               @/data/rw/code/scala/sbt/main/target/compile.args  sample       7818.183            ms/op
[info] HotScalacBenchmark.compile:compile·p0.90             latest               @/data/rw/code/scala/sbt/main/target/compile.args  sample       8110.106            ms/op
[info] HotScalacBenchmark.compile:compile·p0.95             latest               @/data/rw/code/scala/sbt/main/target/compile.args  sample       8396.577            ms/op
[info] HotScalacBenchmark.compile:compile·p0.99             latest               @/data/rw/code/scala/sbt/main/target/compile.args  sample       8514.437            ms/op
[info] HotScalacBenchmark.compile:compile·p0.999            latest               @/data/rw/code/scala/sbt/main/target/compile.args  sample       8514.437            ms/op
[info] HotScalacBenchmark.compile:compile·p0.9999           latest               @/data/rw/code/scala/sbt/main/target/compile.args  sample       8514.437            ms/op
[info] HotScalacBenchmark.compile:compile·p1.00             latest               @/data/rw/code/scala/sbt/main/target/compile.args  sample       8514.437            ms/op

@retronym
Copy link
Member Author

retronym commented Jun 18, 2017 •
edited
Loading

The new version 2.12.3 is 33% faster in p0.99

@jvican I don't think that's the best way to interpret the results, as you are comparing outliers. I'd say comparing 10089±174. vs 7825.872±110 (the result labelled "HotScalacBenchmark.compile") suggests a 22.5% improvement (compile time reduced by 0.776x)

@jvican
Copy link
Member

jvican commented Jun 18, 2017 •
edited
Loading

Sorry, I got the improvement percentage wrong in my last comment. It's actually 22.5% in p99, the same as the average you report.

On a side note, I prefer to use percentiles rather than average times because percentiles give you more precise results. In computers that are running several applications at the same time (browser, other sbt instances, messaging applications, et cetera) hiccups or outliers are more prone to happen. Gil Tene's talk on measuring latency and several other interviews I've read seem to agree with this point. It is my understanding that it's good practice to use high percentiles to compare results safely and with more precision.

@mpollmeier
Copy link

mpollmeier commented Jun 19, 2017 •
edited
Loading

30% speedup for https://github.com/mpollmeier/gremlin-scala/ which uses shapeless and a macro!

:)

2.12.2

Benchmark                                   (corpusVersion)  (extraArgs)  (source)    Mode  Cnt     Score   Error  Units
HotScalacBenchmark.compile                           latest               @...args  sample  342   912.206 ± 4.854  ms/op
HotScalacBenchmark.compile:compile·p0.00             latest               @...args  sample        869.270          ms/op
HotScalacBenchmark.compile:compile·p0.50             latest               @...args  sample        907.018          ms/op
HotScalacBenchmark.compile:compile·p0.90             latest               @...args  sample        940.258          ms/op
HotScalacBenchmark.compile:compile·p0.95             latest               @...args  sample        960.338          ms/op
HotScalacBenchmark.compile:compile·p0.99             latest               @...args  sample       1011.425          ms/op
HotScalacBenchmark.compile:compile·p0.999            latest               @...args  sample       1105.199          ms/op
HotScalacBenchmark.compile:compile·p0.9999           latest               @...args  sample       1105.199          ms/op
HotScalacBenchmark.compile:compile·p1.00             latest               @...args  sample       1105.199          ms/op

2.12.3-bin-d1ec01a

Benchmark                                   (corpusVersion)  (extraArgs)  (source)    Mode  Cnt    Score   Error  Units
HotScalacBenchmark.compile                           latest               @...args  sample  481  639.767 ± 2.681  ms/op
HotScalacBenchmark.compile:compile·p0.00             latest               @...args  sample       612.368          ms/op
HotScalacBenchmark.compile:compile·p0.50             latest               @...args  sample       635.437          ms/op
HotScalacBenchmark.compile:compile·p0.90             latest               @...args  sample       664.797          ms/op
HotScalacBenchmark.compile:compile·p0.95             latest               @...args  sample       674.130          ms/op
HotScalacBenchmark.compile:compile·p0.99             latest               @...args  sample       698.540          ms/op
HotScalacBenchmark.compile:compile·p0.999            latest               @...args  sample       735.052          ms/op
HotScalacBenchmark.compile:compile·p0.9999           latest               @...args  sample       735.052          ms/op
HotScalacBenchmark.compile:compile·p1.00             latest               @...args  sample       735.052          ms/op

@kmizu
Copy link

kmizu commented Jun 23, 2017 •
edited
Loading

Onion

Onion is my own programming language written in Scala. lines are about 10000.

Scala 2.12.2

[info] Benchmark                                   (corpusVersion)  (extraArgs)                          (source)    Mode  Cnt     Score    Error  Units
[info] HotScalacBenchmark.compile                           latest               @/code/onion/target/compile.args  sample  154  2058.545 ± 18.401  ms/op
[info] HotScalacBenchmark.compile:compile·p0.00             latest               @/code/onion/target/compile.args  sample       1948.254           ms/op
[info] HotScalacBenchmark.compile:compile·p0.50             latest               @/code/onion/target/compile.args  sample       2053.112           ms/op
[info] HotScalacBenchmark.compile:compile·p0.90             latest               @/code/onion/target/compile.args  sample       2151.678           ms/op
[info] HotScalacBenchmark.compile:compile·p0.95             latest               @/code/onion/target/compile.args  sample       2203.058           ms/op
[info] HotScalacBenchmark.compile:compile·p0.99             latest               @/code/onion/target/compile.args  sample       2300.156           ms/op
[info] HotScalacBenchmark.compile:compile·p0.999            latest               @/code/onion/target/compile.args  sample       2327.839           ms/op
[info] HotScalacBenchmark.compile:compile·p0.9999           latest               @/code/onion/target/compile.args  sample       2327.839           ms/op
[info] HotScalacBenchmark.compile:compile·p1.00             latest               @/code/onion/target/compile.args  sample       2327.839           ms/op

Scala 2.12.3-bin-e72ab5a

[info] Benchmark                                   (corpusVersion)  (extraArgs)                          (source)    Mode  Cnt     Score    Error  Units
[info] HotScalacBenchmark.compile                           latest               @/code/onion/target/compile.args  sample  204  1631.502 ± 11.079  ms/op
[info] HotScalacBenchmark.compile:compile·p0.00             latest               @/code/onion/target/compile.args  sample       1556.087           ms/op
[info] HotScalacBenchmark.compile:compile·p0.50             latest               @/code/onion/target/compile.args  sample       1624.244           ms/op
[info] HotScalacBenchmark.compile:compile·p0.90             latest               @/code/onion/target/compile.args  sample       1689.256           ms/op
[info] HotScalacBenchmark.compile:compile·p0.95             latest               @/code/onion/target/compile.args  sample       1722.286           ms/op
[info] HotScalacBenchmark.compile:compile·p0.99             latest               @/code/onion/target/compile.args  sample       1819.384           ms/op
[info] HotScalacBenchmark.compile:compile·p0.999            latest               @/code/onion/target/compile.args  sample       1826.619           ms/op
[info] HotScalacBenchmark.compile:compile·p0.9999           latest               @/code/onion/target/compile.args  sample       1826.619           ms/op
[info] HotScalacBenchmark.compile:compile·p1.00             latest               @/code/onion/target/compile.args  sample       1826.619           ms/op
[success] Total time: 667 s, completed 2017/06/24 2:37:38

The result seems that switching from Scala 2.12.2 to Scala 2.12.3 cause 30% compilation speed up.

@retronym retronym closed this as completed Jul 8, 2017
@retronym retronym mentioned this issue Dec 13, 2017
Merged
@retronym retronym mentioned this issue Jan 21, 2018
Closed
@jvican jvican mentioned this issue Jan 29, 2018
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@retronym retronym

Labels
help-wanted t:performance
Projects
None yet
Milestone
2.12.3
Development

No branches or pull requests
8 participants
@espenhw @retronym @kmizu @xuwei-k @mpollmeier @DarkDimius @olafurpg @jvican