Scala 2.9 Compiler Performance

We have migrated a few projects to Scala 2.9, and have measured an
increase in compilation time. It's something of a pain point.

We are using SBT 0.7.7 and Scala 2.9.0.1.

The total compile time for one project increased from 146s to 307s. I
have published the statistics, gathered with -Ystatistics, for one
module [1]. Time spent in implicits during type checking has
increased.

It's not a perfectly scientific test, as some dependencies (Akka,
Scalaz, SQLRL) were updated during the migration. We are working on a
more isolated test.

As another data point, the Scala build (including tests) is also
slower with 2.9. [2]

There were quite a few changes, some performance related, in
Implicits.scala [3] for 2.9.

Could others share there experience with compiler performance after
the 2.9 upgrade? If you find a big difference, could you post the
output of scalac -Ystatistics for 2.8.1/2.9.0.1?

We do plan to migrate to SBT 0.10 to avoid needless recompilation, but
obviously the clean build performance is still important.

-jason

[1] https://github.com/retronym/scratch/tree/master/20110609
[2] http://jenkins.scala-tools.org/view/scalaz/
[3] https://github.com/scala/scala/commits/master/src/compiler/scala/tools/n...

Hi Jason,

On Thu, Jun 9, 2011 at 9:54 AM, Jason Zaugg wrote:
> We have migrated a few projects to Scala 2.9, and have measured an
> increase in compilation time. It's something of a pain point.

Have you tried trunk? There are a lot of performance improvements
there and they will be backported to a subsequent release as far as I
understand. That's a better starting point for performance profiling
than 2.9.0.1 at this point.

Best,
Ismael

On Thu, Jun 9, 2011 at 11:07 AM, Ismael Juma wrote:
> On Thu, Jun 9, 2011 at 9:54 AM, Jason Zaugg wrote:
>> We have migrated a few projects to Scala 2.9, and have measured an
>> increase in compilation time. It's something of a pain point.
>
> Have you tried trunk? There are a lot of performance improvements
> there and they will be backported to a subsequent release as far as I
> understand. That's a better starting point for performance profiling
> than 2.9.0.1 at this point.

We haven't tried that yet, but will shortly. Those changes were mostly
around classfile loading from JAR files, IIRC. Not sure if that is our
problem, but fingers crossed!

-jason

On Thu, Jun 9, 2011 at 10:20 AM, Jason Zaugg wrote:
> We haven't tried that yet, but will shortly. Those changes were mostly
> around classfile loading from JAR files, IIRC. Not sure if that is our
> problem, but fingers crossed!

Also see this thread:

https://groups.google.com/d/msg/simple-build-tool/uArg26ON1B8/zipr2CwsCTwJ

Best,
Ismael

We know that implicit search can be a large contributor to compile times. To get a handle on things, it would be good to know whether the implicit definitions and parameters have changed between 2.8 and 2.9 (new scalaz, maybe?) If they did, that could explain the increase in compilation times. Also, there are some recent improvements to implicit search in trunk which help in some cases.

Cheers

 -- Martin


On Thu, Jun 9, 2011 at 11:29 AM, Ismael Juma <ismael [at] juma [dot] me [dot] uk> wrote:

On Thu, Jun 9, 2011 at 10:20 AM, Jason Zaugg <jzaugg [at] gmail [dot] com> wrote:
> We haven't tried that yet, but will shortly. Those changes were mostly
> around classfile loading from JAR files, IIRC. Not sure if that is our
> problem, but fingers crossed!

Also see this thread:

https://groups.google.com/d/msg/simple-build-tool/uArg26ON1B8/zipr2CwsCTwJ

Best,
Ismael

--
----------------------------------------------
Martin Odersky
Prof., EPFL and CEO, Typesafe
PSED, 1015 Lausanne, Switzerland
Tel. EPFL: +41 21 693 6863
Tel. Typesafe: +41 21 691 4967

On Thu, Jun 9, 2011 at 12:49 PM, martin odersky wrote:
> We know that implicit search can be a large contributor to compile times. To
> get a handle on things, it would be good to know whether the implicit
> definitions and parameters have changed between 2.8 and 2.9 (new scalaz,
> maybe?) If they did, that could explain the increase in compilation times.
> Also, there are some recent improvements to implicit search in trunk which
> help in some cases.

I think those optimizations to implicit search are already in 2.9.0;
for example the statistics shows "implicit improves cached", which was
added in r24779.

Scalaz is a dependency of the projects where we've measured a
slow-down, although it's API did not change.

Scalaz itself has effectively the same code base on the master and
scala-2.9.x branches, and also shows a clear slowdown.

branch: master
scalac: 2.8.1
time: 127s
output: http://jenkins.scala-tools.org/job/scalaz-2.8.x-compile-only/3/console

branch: scala-2.9.x
scalac: 2.9.0.1
time: 185s
output: http://jenkins.scala-tools.org/job/scalaz-2.9.x-compile-only/2/console

The code is available: git://github.com/scalaz/scalaz.git

-jason

On Jun 9, 2011, at 2:14 PM, Jason Zaugg wrote:

> On Thu, Jun 9, 2011 at 12:49 PM, martin odersky wrote:
>> We know that implicit search can be a large contributor to compile times. To
>> get a handle on things, it would be good to know whether the implicit
>> definitions and parameters have changed between 2.8 and 2.9 (new scalaz,
>> maybe?) If they did, that could explain the increase in compilation times.
>> Also, there are some recent improvements to implicit search in trunk which
>> help in some cases.
>
> I think those optimizations to implicit search are already in 2.9.0;
> for example the statistics shows "implicit improves cached", which was
> added in r24779.

There are more and some of them are not even in trunk yet (got intercepted by general compiler refactorings):

https://github.com/TiarkRompf/scala-virtualized/commits/perfmaster

- Tiark

[Moved to internals]

On 6/9/11 5:14 AM, Jason Zaugg wrote:
> Scalaz itself has effectively the same code base on the master and
> scala-2.9.x branches, and also shows a clear slowdown.

I wanted to compare the performance of 2.9.0-1 with trunk+tiark's patches. First I had to get trunk+sbt working again (r25068) but then I found that the scalaz 2.9 branch no longer builds. Can you give this a look and opine on whether this represents a regression? (It seems likely that it does.) There have been indications that implicit search developed behavioral changes between 2.8.1 and 2.9 (like .exists vs .exists(), and (x: Byte).toHexString) and I hate to see more springing up during this next leg.

Here's how it fails.

% SBT_OPTS="-Dscala.local=/scala/trunk/build/pack" sbt ++2.10.0-local compile
[warn] Credentials file /Users/paulp/.ivy2/.credentials does not exist
[info] Found scala.local: /scala/trunk/build/pack
[info] Building project scalaz 6.0.1-SNAPSHOT against Scala 2.9.0-1
[info] using ScalazProject with sbt 0.7.5.RC0 and Scala 2.7.7
[warn] Credentials file /Users/paulp/.ivy2/.credentials does not exist
[info] Found scala.local: /scala/trunk/build/pack
[info] Building project scalaz 6.0.1-SNAPSHOT against Scala 2.10.0-local
[info] using ScalazProject with sbt 0.7.5.RC0 and Scala 2.7.7
[info]
[info] == scalaz-core / clean src_managed ==
[info] Deleting directory /soft/src/scalaz/core/src_managed/main/scala
[info] == scalaz-core / clean src_managed ==
[info]
[info] == scalaz-core / generate-tuple-w ==
[info] == scalaz-core / generate-tuple-w ==
[info]
[info] == scalaz-core / compile ==
[info] Source analysis: 120 new/modified, 0 indirectly invalidated, 0 removed.
[info] Compiling main sources...
[error] /soft/src/scalaz/core/src/main/scala/scalaz/Each.scala:27: value toList is not a member of scalaz.Scalaz.IndSeq[A]
[error] def each[A](e: IndSeq[A], f: A => Unit) = e.toList foreach f
[error] ^
[error] /soft/src/scalaz/core/src/main/scala/scalaz/Equal.scala:120: value toList is not a member of scalaz.Scalaz.IndSeq[A]
[error] implicit def IndSeqEqual[A: Equal]: Equal[IndSeq[A]] = equalBy(_.toList)
[error] ^
[error] /soft/src/scalaz/core/src/main/scala/scalaz/Equal.scala:120: diverging implicit expansion for type scalaz.Equal[Nothing]
[error] starting with method ZipperEqual in object Equal
[error] implicit def IndSeqEqual[A: Equal]: Equal[IndSeq[A]] = equalBy(_.toList)
[error] ^
[error] /soft/src/scalaz/core/src/main/scala/scalaz/FingerTree.scala:1048: value <++> is not a member of java.lang.Object with OrdSeqs.this.OrdSeq[A]
[error] case (l, r) => ordSeq(l <++> (a +: r))
[error] ^
[error] /soft/src/scalaz/core/src/main/scala/scalaz/FingerTree.scala:1050: value toList is not a member of OrdSeqs.this.OrdSeq[A]
[error] def ++(xs: OrdSeq[A]) = xs.toList.foldLeft(this)(_ insert _)
[error] ^
[error] /soft/src/scalaz/core/src/main/scala/scalaz/Foldable.scala:84: value toList is not a member of scalaz.Scalaz.IndSeq[A]
[error] t.toList.foldLeft(z)(f)
[error] ^
[error] /soft/src/scalaz/core/src/main/scala/scalaz/Foldable.scala:87: value toList is not a member of scalaz.Scalaz.IndSeq[A]
[error] t.toList.foldr(z)(op)
[error] ^
[error] /soft/src/scalaz/core/src/main/scala/scalaz/Order.scala:104: value toList is not a member of scalaz.Scalaz.IndSeq[A]
[error] implicit def IndSeqOrder[A: Order]: Order[IndSeq[A]] = orderBy(_.toList)
[error] ^
[error] /soft/src/scalaz/core/src/main/scala/scalaz/Order.scala:104: diverging implicit expansion for type scalaz.Order[Nothing]
[error] starting with method IterableOrder in object Order
[error] implicit def IndSeqOrder[A: Order]: Order[IndSeq[A]] = orderBy(_.toList)
[error] ^
[error] /soft/src/scalaz/core/src/main/scala/scalaz/Semigroup.scala:56: value * is not a member of scalaz.IntMultiplication
[error] implicit def IntMultiplicationSemigroup: Semigroup[IntMultiplication] = semigroup(_ * _ ∏)
[error] ^
[error] /soft/src/scalaz/core/src/main/scala/scalaz/Semigroup.scala:58: value && is not a member of scalaz.BooleanConjunction
[error] implicit def BooleanConjunctionSemigroup: Semigroup[BooleanConjunction] = semigroup(_ && _ |∧|)
[error] ^
[error] /soft/src/scalaz/core/src/main/scala/scalaz/Semigroup.scala:64: value * is not a member of scalaz.CharMultiplication
[error] implicit def CharMultiplicationSemigroup: Semigroup[CharMultiplication] = semigroup((a, b) => (a * b).toChar ∏)
[error] ^
[error] /soft/src/scalaz/core/src/main/scala/scalaz/Semigroup.scala:68: value * is not a member of scalaz.ByteMultiplication
[error] implicit def ByteMultiplicationSemigroup: Semigroup[ByteMultiplication] = semigroup((a, b) => (a * b).toByte ∏)
[error] ^
[error] /soft/src/scalaz/core/src/main/scala/scalaz/Semigroup.scala:72: value * is not a member of scalaz.LongMultiplication
[error] implicit def LongMultiplicationSemigroup: Semigroup[LongMultiplication] = semigroup((a, b) => (a * b).toLong ∏)
[error] ^
[error] /soft/src/scalaz/core/src/main/scala/scalaz/Semigroup.scala:76: value * is not a member of scalaz.ShortMultiplication
[error] implicit def ShortMultiplicationSemigroup: Semigroup[ShortMultiplication] = semigroup((a, b) => (a * b).toShort ∏)
[error] ^
[error] /soft/src/scalaz/core/src/main/scala/scalaz/Semigroup.scala:105: value orElse is not a member of scalaz.FirstOption[A]
[error] implicit def FirstOptionSemigroup[A]: Semigroup[FirstOption[A]] = semigroup((a, b) => a orElse b)
[error] ^
[error] /soft/src/scalaz/core/src/main/scala/scalaz/Semigroup.scala:107: value orElse is not a member of scalaz.LastOption[A]
[error] implicit def LastOptionSemigroup[A]: Semigroup[LastOption[A]] = semigroup((a, b) => b orElse a)
[error] ^
[error] /soft/src/scalaz/core/src/main/scala/scalaz/Semigroup.scala:113: value orElse is not a member of scalaz.FirstLazyOption[A]
[error] implicit def FirstLazyOptionSemigroup[A]: Semigroup[FirstLazyOption[A]] = semigroup((a, b) => a orElse b)
[error] ^
[error] /soft/src/scalaz/core/src/main/scala/scalaz/Semigroup.scala:115: value orElse is not a member of scalaz.LastLazyOption[A]
[error] implicit def LastLazyOptionSemigroup[A]: Semigroup[LastLazyOption[A]] = semigroup((a, b) => b orElse a)
[error] ^
[error] /soft/src/scalaz/core/src/main/scala/scalaz/Semigroup.scala:138: value compose is not a member of scalaz.Endo[A]
[error] implicit def EndoSemigroup[A]: Semigroup[Endo[A]] = semigroup((x, y) => (EndoTo(x compose y)))
[error] ^
[error] /soft/src/scalaz/core/src/main/scala/scalaz/Show.scala:80: value toList is not a member of scalaz.Scalaz.IndSeq[A]
[error] implicit def IndSeqShow[A: Show]: Show[IndSeq[A]] = showBy(_.toList)
[error] ^
[error] /soft/src/scalaz/core/src/main/scala/scalaz/Show.scala:80: diverging implicit expansion for type scalaz.Show[B]
[error] starting with method IterableShow in object Show
[error] implicit def IndSeqShow[A: Show]: Show[IndSeq[A]] = showBy(_.toList)
[error] ^
[error] 22 errors found
[info] == scalaz-core / compile ==
[error] Error running compile: Compilation failed
[info]
[info] Total time: 39 s, completed Jun 11, 2011 3:33:39 AM
[info]
[info] Total session time: 39 s, completed Jun 11, 2011 3:33:39 AM
[error] Error during build.

On Sat, Jun 11, 2011 at 12:44 PM, Paul Phillips wrote:
> On 6/9/11 5:14 AM, Jason Zaugg wrote:
>> Scalaz itself has effectively the same code base on the master and
>> scala-2.9.x branches, and also shows a clear slowdown.
>
> [info] == scalaz-core / compile ==
> [info]   Source analysis: 120 new/modified, 0 indirectly invalidated, 0 removed.
> [info] Compiling main sources...
> [error] /soft/src/scalaz/core/src/main/scala/scalaz/Each.scala:27: value toList is not a member of scalaz.Scalaz.IndSeq[A]
> [error]     def each[A](e: IndSeq[A], f: A => Unit) = e.toList foreach f

That should use NewType.UnwrapNewType [1] as an implicit view, located
in the implicit scope of IndSeq [2].

-jason

[1] http://scalaz.github.com/scalaz/scalaz-2.9.0-1-6.0/doc.sxr/scalaz/NewTyp...
[2] http://scalaz.github.com/scalaz/scalaz-2.9.0-1-6.0/doc.sxr/scalaz/Finger...

On 6/11/11 3:57 AM, Jason Zaugg wrote:
> That should use NewType.UnwrapNewType [1] as an implicit view, located
> in the implicit scope of IndSeq [2].

Well, at least I knew where to look. (*cough* MOORS *cough*) Here is
what looks like reasonably good news.

2.9.0-1 Total time: 187 s, completed Jun 11, 2011 4:23:43 AM

trunk Total time: 138 s, completed Jun 11, 2011 4:19:50 AM

Not sure about trunk+fix+tiark yet because you don't seem to compile
with that either.

On Sat, Jun 11, 2011 at 1:36 PM, Paul Phillips wrote:
> On 6/11/11 3:57 AM, Jason Zaugg wrote:
> Well, at least I knew where to look.  (*cough* MOORS *cough*) Here is
> what looks like reasonably good news.
>
> 2.9.0-1     Total time: 187 s, completed Jun 11, 2011 4:23:43 AM
>
> trunk       Total time: 138 s, completed Jun 11, 2011 4:19:50 AM

I've reproduced similar results. Times below are for `sbt compile`.
I'll try this out on our work projects on Tuesday.

Any idea which particular changes in 2.9.x and 2.10.x explain the
slow-down and the subsequent speed-up?

-jason

2.10.0.local (trunk@r25068, reverting r25051)

real 2m34.993s
user 4m49.398s
sys 0m11.575s

2.9.0-1

real 3m31.788s
user 5m50.941s
sys 0m14.034s

2.8.1

real 2m28.938s
user 4m47.004s
sys 0m13.760s

One source of slowdown could be parallel collections. The new
collections library has almost twice as many implicits to search for a CanBuildFrom, and also the base type sequences (important for the peed of subtype tests) are alomst 50% longer.  So if your code is using many maps, filters, zips, etc, implicit search has to work much harder.

There might be other sources for the slowdown. Qualified leads are much appreciated!

The speedups are due to improvements to implicit search that we have put in, and are still adding.

Cheers

 -- Martin

On Sat, Jun 11, 2011 at 5:49 PM, Jason Zaugg <jzaugg [at] gmail [dot] com> wrote:

On Sat, Jun 11, 2011 at 1:36 PM, Paul Phillips <paulp [at] improving [dot] org> wrote:
> On 6/11/11 3:57 AM, Jason Zaugg wrote:
> Well, at least I knew where to look.  (*cough* MOORS *cough*) Here is
> what looks like reasonably good news.
>
> 2.9.0-1     Total time: 187 s, completed Jun 11, 2011 4:23:43 AM
>
> trunk       Total time: 138 s, completed Jun 11, 2011 4:19:50 AM

I've reproduced similar results. Times below are for `sbt compile`.
I'll try this out on our work projects on Tuesday.

Any idea which particular changes in 2.9.x and 2.10.x explain the
slow-down and the subsequent speed-up?

-jason

2.10.0.local (trunk@r25068, reverting r25051)

real    2m34.993s
user    4m49.398s
sys     0m11.575s

2.9.0-1

real    3m31.788s
user    5m50.941s
sys     0m14.034s

2.8.1

real    2m28.938s
user    4m47.004s
sys     0m13.760s

--
----------------------------------------------
Martin Odersky
Prof., EPFL and CEO, Typesafe
PSED, 1015 Lausanne, Switzerland
Tel. EPFL: +41 21 693 6863
Tel. Typesafe: +41 21 691 4967

On Sat, Jun 11, 2011 at 5:55 PM, martin odersky <martin [dot] odersky [at] epfl [dot] ch> wrote:

One source of slowdown could be parallel collections. The new
collections library has almost twice as many implicits to search for a CanBuildFrom, and also the base type sequences (important for the peed of subtype tests) are alomst 50% longer.  So if your code is using many maps, filters, zips, etc, implicit search has to work much harder.

There might be other sources for the slowdown. Qualified leads are much appreciated!

The speedups are due to improvements to implicit search that we have put in, and are still adding.

Could implicit search be paralellized (to find candidates) in a fork/join manner?
 

Cheers

 -- Martin

On Sat, Jun 11, 2011 at 5:49 PM, Jason Zaugg <jzaugg [at] gmail [dot] com> wrote:

On Sat, Jun 11, 2011 at 1:36 PM, Paul Phillips <paulp [at] improving [dot] org> wrote:
> On 6/11/11 3:57 AM, Jason Zaugg wrote:
> Well, at least I knew where to look.  (*cough* MOORS *cough*) Here is
> what looks like reasonably good news.
>
> 2.9.0-1     Total time: 187 s, completed Jun 11, 2011 4:23:43 AM
>
> trunk       Total time: 138 s, completed Jun 11, 2011 4:19:50 AM

I've reproduced similar results. Times below are for `sbt compile`.
I'll try this out on our work projects on Tuesday.

Any idea which particular changes in 2.9.x and 2.10.x explain the
slow-down and the subsequent speed-up?

-jason

2.10.0.local (trunk@r25068, reverting r25051)

real    2m34.993s
user    4m49.398s
sys     0m11.575s

2.9.0-1

real    3m31.788s
user    5m50.941s
sys     0m14.034s

2.8.1

real    2m28.938s
user    4m47.004s
sys     0m13.760s

--
----------------------------------------------
Martin Odersky
Prof., EPFL and CEO, Typesafe
PSED, 1015 Lausanne, Switzerland
Tel. EPFL: +41 21 693 6863
Tel. Typesafe: +41 21 691 4967

--
Viktor Klang

Akka Tech LeadTypesafe - Enterprise-Grade Scala from the Experts

Twitter: @viktorklang

On Sat, Jun 11, 2011 at 4:55 PM, martin odersky wrote:
> The speedups are due to improvements to implicit search that we have put in,
> and are still adding.

Is the plan to backport these to the 2.9.x series? That would be a
really nice productivity boost. I find myself waiting for the compiler
more often than I would like these days (and I've already ported our
build to SBT 0.10 with its improved dependency management module).

Best,
Ismael

2011/6/11 √iktor Ҡlang <viktor [dot] klang [at] gmail [dot] com>

On Sat, Jun 11, 2011 at 5:55 PM, martin odersky <martin [dot] odersky [at] epfl [dot] ch> wrote:

One source of slowdown could be parallel collections. The new
collections library has almost twice as many implicits to search for a CanBuildFrom, and also the base type sequences (important for the peed of subtype tests) are alomst 50% longer.  So if your code is using many maps, filters, zips, etc, implicit search has to work much harder.

There might be other sources for the slowdown. Qualified leads are much appreciated!

The speedups are due to improvements to implicit search that we have put in, and are still adding.

Could implicit search be paralellized (to find candidates) in a fork/join manner?

Don't think so. There are too many join points where several implicit searches would force the same attributes. In the end we'd need to synchronize everything, so it would most likely be slower.

 --- Martin

 

Cheers

 -- Martin

On Sat, Jun 11, 2011 at 5:49 PM, Jason Zaugg <jzaugg [at] gmail [dot] com> wrote:

On Sat, Jun 11, 2011 at 1:36 PM, Paul Phillips <paulp [at] improving [dot] org> wrote:
> On 6/11/11 3:57 AM, Jason Zaugg wrote:
> Well, at least I knew where to look.  (*cough* MOORS *cough*) Here is
> what looks like reasonably good news.
>
> 2.9.0-1     Total time: 187 s, completed Jun 11, 2011 4:23:43 AM
>
> trunk       Total time: 138 s, completed Jun 11, 2011 4:19:50 AM

I've reproduced similar results. Times below are for `sbt compile`.
I'll try this out on our work projects on Tuesday.

Any idea which particular changes in 2.9.x and 2.10.x explain the
slow-down and the subsequent speed-up?

-jason

2.10.0.local (trunk@r25068, reverting r25051)

real    2m34.993s
user    4m49.398s
sys     0m11.575s

2.9.0-1

real    3m31.788s
user    5m50.941s
sys     0m14.034s

2.8.1

real    2m28.938s
user    4m47.004s
sys     0m13.760s

--
----------------------------------------------
Martin Odersky
Prof., EPFL and CEO, Typesafe
PSED, 1015 Lausanne, Switzerland
Tel. EPFL: +41 21 693 6863
Tel. Typesafe: +41 21 691 4967

--
Viktor Klang

Akka Tech LeadTypesafe - Enterprise-Grade Scala from the Experts

Twitter: @viktorklang

--
----------------------------------------------
Martin Odersky
Prof., EPFL and CEO, Typesafe
PSED, 1015 Lausanne, Switzerland
Tel. EPFL: +41 21 693 6863
Tel. Typesafe: +41 21 691 4967

And any candidates for object pooling?

On Jun 11, 2011 6:01 PM, "Ismael Juma" <ismael [at] juma [dot] me [dot] uk> wrote:
> On Sat, Jun 11, 2011 at 4:55 PM, martin odersky <martin [dot] odersky [at] epfl [dot] ch> wrote:
>> The speedups are due to improvements to implicit search that we have put in,
>> and are still adding.
>
> Is the plan to backport these to the 2.9.x series? That would be a
> really nice productivity boost. I find myself waiting for the compiler
> more often than I would like these days (and I've already ported our
> build to SBT 0.10 with its improved dependency management module).
>
> Best,
> Ismael

On Sat, Jun 11, 2011 at 5:55 PM, martin odersky wrote:
> The speedups are due to improvements to implicit search that we have put in,
> and are still adding.

I still can't pinpont any ostensibly performance relevant commit(s) to
trunk typechecker [1] that didn't make it to 2.9.0. r24779 made it.
What am I missing?

In case you didn't see, my original mail referenced before/after
compiler statistics [2]. Of note: 2.2 x subtype checks, 1.7x longer in
implicits, similar average base type seq length. The counter for
'tried implicits' shows zero, seemingly spuriously, in 2.9.

I'm hopeful that the trunk improvements will restore the status quo;
but I'm eager to quantify the costs and benefits of individual
changes. Maybe this way we'll find additional areas for improvement.

-jason

[1] http://lampsvn.epfl.ch/trac/scala/log/scala/trunk/src/compiler/scala/too...
[2] https://github.com/retronym/scratch/tree/master/20110609

On 6/11/11 11:25 AM, Jason Zaugg wrote:
> I still can't pinpont any ostensibly performance relevant commit(s) to
> trunk typechecker [1] that didn't make it to 2.9.0. r24779 made it.
> What am I missing?

I don't know yet (I would of course like to believe it is the general
performance goodness of my recent commits, but it's not a very
convincing tale) but to throw something else on the fire: in one of
today's more interesting coincidences, I find that reverting the commit
which was keeping scalaz from building no longer affects anything.
Something in tiark's changes to Infer.scala causes the same regression
from a different angle. Should this sound like programmer error, which
it surely does, I expect it can be confirmed like so:

1) Start with r25072
2) Revert r25051
3) Notice scalaz still doesn't build
4) Check out Infer.scala from r25068, right before tiark's commits
5) scalaz builds

On Sun, Jun 12, 2011 at 12:23 AM, Paul Phillips wrote:
> On 6/11/11 11:25 AM, Jason Zaugg wrote:
> Something in tiark's changes to Infer.scala causes the same regression
> from a different angle.  Should this sound like programmer error, which
> it surely does, I expect it can be confirmed like so:
>
> 1) Start with r25072
> 2) Revert r25051
> 3) Notice scalaz still doesn't build
> 4) Check out Infer.scala from r25068, right before tiark's commits
> 5) scalaz builds

Confirmed.

Even given the revert of Infer.scala, a few more seconds are shaved
off (154s -> 147s).

Times and stats for the four versions:

https://github.com/retronym/scratch/tree/master/20110611

-jason

On 6/11/11 3:46 PM, Jason Zaugg wrote:
> Confirmed.

Compilation is restored by commenting out the RefinedType case in the following. It is searching for a type like { val toList } and declares it impossible. I'm guessing it's supposed to come back looking for a view, but starts musing about clouds and pirates and gets lost at sea.

private def isImpossibleSubType(tp1: Type, tp2: Type) = tp1.normalize.widen match {
case tr1 @ TypeRef(_, sym1, _) =>
tp2.normalize.widen match {
case TypeRef(_, sym2, _) =>
sym1.isClass &&
sym2.isClass &&
!(sym1 isSubClass sym2) &&
!(sym1 isNumericSubClass sym2)
case RefinedType(_, decls) =>
decls.nonEmpty && tp1.member(decls.head.name) == NoSymbol
case _ => false
}
case _ => false
}

// old version

private def isImpossibleSubType(tp1: Type, tp2: Type) = {
(tp1.dealias, tp2.dealias) match {
case (TypeRef(_, sym1, _), TypeRef(_, sym2, _)) =>
sym1.isClass &&
sym2.isClass &&
!(sym1 isSubClass sym2) &&
!(sym1 isNumericSubClass sym2)
case _ =>
false
}
}

On Thu, Jun 9, 2011 at 10:54 AM, Jason Zaugg wrote:
> We have migrated a few projects to Scala 2.9, and have measured an
> increase in compilation time. It's something of a pain point.
> The total compile time for one project increased from 146s to 307s. I
> have published the statistics, gathered with -Ystatistics, for one
> module [1]. Time spent in implicits during type checking has
> increased.

> [1] https://github.com/retronym/scratch/tree/master/20110609

The good news: we've just run that project on r25101 with a 2.9
backwards compatible patch [2] [3], and the total build time is now
116s. Log with compiler stats here [4]. The numbers aren't rigorously
gathered, but they are in line with what I've seen with the Scalaz
build.

We are very keen to see these improvements in a point release of 2.9.

Thanks for the effort invested in this area so far.

-jason

[2] http://dl.dropbox.com/u/106552/scala-2.10.0-SI-4709.zip
[3] https://github.com/retronym/scala/commits/ticket%2FSI-4709
[4] https://gist.github.com/1050736