This page is no longer maintained — Please continue to the home page at www.scala-lang.org

Proguard shrinking and Scala collection library [was 'a kill to a view']

4 replies
Johannes Rudolph 2
Joined: 2010-02-12,
User offline. Last seen 42 years 45 weeks ago.

On Sat, Dec 17, 2011 at 5:45 PM, martin odersky wrote:
>> No, but proguard isn't able to remove view classes in many cases even
>> if they are not used any where in user code. And they are the current
>> road block for using the Scala standard library with Android.
>
> It would be good to dig deeper on this one. Why can't proguard remove
> these classes?

I don't know. You can enable some kind of verbose mode in proguard
where it's supposed to print out a trace why it keeps certain things
but as far as I looked it up in the sources of proguard, the verbose
mode works differently from the normal case, runs much longer and
doesn't finish or provide useful output when it's finished.

In general, it seems to me that shrinking is really hard (probably
even undecidable?) for a language with virtual calls. Especially,
Scala's collections class hierarchy appears to be a particular hard
case because it really needs and uses virtual calls all over the
place. From what I've seen proguard's shrinker isn't sophisticated
enough to get even near an optimal solution in those cases.

Probably we should talk to the Proguard developers, maybe they can say
what's going on exactly.

extempore
Joined: 2008-12-17,
User offline. Last seen 35 weeks 3 days ago.
Re: Proguard shrinking and Scala collection library [was 'a kil


On Sat, Dec 17, 2011 at 9:06 AM, Johannes Rudolph <johannes [dot] rudolph [at] googlemail [dot] com> wrote:
In general, it seems to me that shrinking is really hard (probably
even undecidable?) for a language with virtual calls. Especially,
Scala's collections class hierarchy appears to be a particular hard
case because it really needs and uses virtual calls all over the
place. From what I've seen proguard's shrinker isn't sophisticated
enough to get even near an optimal solution in those cases.

One thing which has been pointed out is that every call to every method in a module is potentially side-effecting, because even if the bodies are pure, the objects are lazily loaded - they are implemented as instance methods of the MODULE$ instance instead of at the static level.  So at the level of sophistication applied by proguard, none of them can ever be eliminated, and then this probably has a bunch of additional spinout in lost cumulative optimizations.
Pavel Pavlov
Joined: 2011-12-01,
User offline. Last seen 42 years 45 weeks ago.
Re: Proguard shrinking and Scala collection library [was 'a kil
Also note that any code which uses structural typing will be silently broken by ProGuard.

For example, innocently looking "times" sugar recently adviced in scala-user:
implicit def liftToTimes(a: Int) = new {
  def times[A](f: => A) = 1 to a foreach { _ => f }
}
10 times println("hello world!") // Oops! reflection call to "times" there
Proguard may freely rename "times" method in synthetic class returned by "liftToTimes" here (and it actually does it).

I fear that with current structural types implementation using Proguard for Scala is just a way to inject subtle bugs into the code.
For now, it would be good at least to have a compiler option to produce the list of such classes/methods to help feeding them manually into proguard projects.

Pavel Pavlov
Joined: 2011-12-01,
User offline. Last seen 42 years 45 weeks ago.
Re: Proguard shrinking and Scala collection library [was 'a kil
Quick grep locates one such place in scala-library.jar nightly build:

class ParHashMapCombiner, method result:
    val table = new HashTable[K, DefaultEntry[K, V]] {
      def insertEntry(e: DefaultEntry[K, V]) = ...
    }
    ...
    for (elem <- buckets(i)) table.insertEntry(elem)

So, mutable.ParHashMap will not work after being processed by ProGuard.

In 2.9.1 library there is one more such place in class scala/reflect/generic/UnPickler.

extempore
Joined: 2008-12-17,
User offline. Last seen 35 weeks 3 days ago.
Re: Proguard shrinking and Scala collection library [was 'a kil

On Tue, Dec 27, 2011 at 12:49 AM, Pavel Pavlov <pavel [dot] e [dot] pavlov [at] gmail [dot] com> wrote:
class ParHashMapCombiner, method result:
    val table = new HashTable[K, DefaultEntry[K, V]] {
      def insertEntry(e: DefaultEntry[K, V]) = ...
    }
    ...
    for (elem <- buckets(i)) table.insertEntry(elem)


% pscalac -Ywarn-reflective-call src/library/scala/collection/parallel/mutable/ParHashMap.scalasrc/library/scala/collection/parallel/mutable/ParHashMap.scala:200: warning: method invocation uses reflection         for (elem <- buckets(i)) table.insertEntry(elem)                                                  ^one warning found
Looks like that's the only one in the library outside of a few in swing.  But hmm, here's something interesting/bad.  This is the buzzkill inflicted by
    type AbstractFileType = AbstractFile
(The irony is that AbstractFile is the concrete file type... concretely abstract.)
% pscalac -Ywarn-reflective-call src/compiler/scala/reflect/internal/Symbols.scala  src/compiler/scala/reflect/internal/Symbols.scala:1505: warning: method invocation uses reflection          if (this.sourceFile.path != that.sourceFile.path) {                              ^ src/compiler/scala/reflect/internal/Symbols.scala:1505: warning: method invocation uses reflection          if (this.sourceFile.path != that.sourceFile.path) {                                                      ^ src/compiler/scala/reflect/internal/Symbols.scala:1508: warning: method invocation uses reflection            if (this.sourceFile.canonicalPath != that.sourceFile.canonicalPath)                                ^ src/compiler/scala/reflect/internal/Symbols.scala:1508: warning: method invocation uses reflection            if (this.sourceFile.canonicalPath != that.sourceFile.canonicalPath)                                                                 ^ src/compiler/scala/reflect/internal/Symbols.scala:2543: warning: method invocation uses reflection    "  Found in " + sym1.sourceFile.canonicalPath + " and " + sym2.sourceFile.canonicalPath                                     ^src/compiler/scala/reflect/internal/Symbols.scala:2543: warning: method invocation uses reflection    "  Found in " + sym1.sourceFile.canonicalPath + " and " + sym2.sourceFile.canonicalPath                                                                               ^6 warnings found

Copyright © 2012 École Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland