This page is no longer maintained — Please continue to the home page at www.scala-lang.org

2.8 copy method needs improvement

19 replies
Archontophoenix
Joined: 2010-02-04,
User offline. Last seen 2 years 35 weeks ago.
The design of the automatically generated copy method, introduced in Scala 2.8
for case classes, is suboptimal.

The main problem with the design of the copy method is that it is incompatible
with inheritance. The copy method is located in the class to be copied, and
provides defaults for its arguments, which are the values for the copied
object's class parameters. Given the way default arguments are implemented, this
design implies that if you generate the copy method for a concrete class A,
there is, in the general case, no consistent way to generate the copy method (as
currently phrased) for a concrete subclass B of A.

There is no shortage of philosophical controversy regarding inheritance in
general, and regarding inheritance from concrete classes in particular; some
people consider the latter not to be good programming practice. Nonetheless,
Scala has so far placed no restrictions on inheriting from concrete classes, and
there are use cases in which the conciseness of concrete inheritance outweighs
any possible difficulty in later refactoring. Scala should continue to support
concrete inheritance without restriction.

There are alternative designs that provide all the benefits of the copy method,
but that are consistent with inheritance, and so would be more useful to Scala
programmers.

Current Design
--------------

Consider the following class hierarchy:
    
    class A (val i: Int)
    class B (val j: Int) extends A(1)
    class C (val s: String, override val i: Int) extends A(i)
    
In the current design, in order to generate a copy method automatically for A,
you declare it as a case class:

    case class A (i: Int)

The compiler transforms this (ignoring case class features not relevant to this
discussion) into something like:

    class A (val i: Int) {
      def copy (i: Int = this.i): A = new A(i)
    }

Given an A:

    val a: A = A(1)

you can invoke this method with:

    a.copy()

to get an unaltered copy, or:

    a.copy(i = 2)

to get a copy with an altered class parameter value.

What if you want to get a copy of a B instead of an A? It is currently legal
(although deprecated) to declare:

    case class B (j: Int) extends A(1)

but if you invoke it on a B:

    val b: B = B(3)
    b.copy()

you get an A instance rather than a B instance, because the copy method is not
generated for a class like B that already inherits a copy method (from A, in
this case).

You might imagine the Scala compiler could automatically generate a copy method
for B that would have the effect of:

    class B (val j: Int) extends A(1) {
      override def copy (j: Int = this.j): B = new B(j)
    }

This looks reasonable enough, but an analogous method for C:

    class C (val s: String, override val i: Int) extends A(i) {
      def copy (s: String = this.s, i: Int = this.i): C = new C(s,i)
    }

runs into trouble. The default-argument mechanism wants to generate this method
in A:

    def copy$default$1: Int = this.i

and this method in C:

    def copy$default$1: String = this.s

Methods with the same parameters cannot have different return types, so it is
not possible to generate a copy of method of this form for C. To avoid such a
situation, the current design refuses to generate any copy method in classes
like B or C, where the superclass already has a copy method.

Alternative 1: Method in Companion Object
-----------------------------------------

An alternative to the current design of the copy method would be to generate a
method not in the class to be copied but in its companion object. For the
classes in the above hierarchy, such methods might look like:

    object A {
      def copy (a: A) (i: Int = a.i): A = new A(i)
    }
    object B {
      def copy (b: B) (j: Int = b.j): B = new B(j)
    }
    object C {
      def copy (c: C) (s: String = c.s, i: Int = c.i): C = new C(s,i)
    }

Given:

    val a: A = A(1)
    val b: B = B(3)
    val c: C = C("hi!",6)

such copy methods might be invoked with:

    A.copy(a)(i = 2)
    B.copy(b)(j = 4)
    C.copy(c)(s = "yo!")

You could even explicitly specify something like the current behavior for
subclasses of case classes, in which only the superclass portion of an object is
copied, if for some reason you considered that useful:

    A.copy(b)(i = 5)        // results in an A instance

Because the copy method is not inherited, there are no issues with inheritance
in this design. You can copy concrete class instances at any level of the class
hierarchy.

You might object that this approach is less elegant than the current one,
because the name of the class being copied appears in the invocation of the
copy method; it is not possible to copy something without specifying the type of
the resulting object. This is a reasonable objection, but it also applies to
the current design of the copy method: even though the name of the destination
class does not appear in the invocation, there is no way to invoke the
automatically generated copy method without knowing the class of the resulting
object, because the copy method is generated for only one class in a given
hierarchy (the resulting object is necessarily of that class).

Another possible objection is that the proposed syntax is more verbose than that
of the current design. One way to remedy this would be to rename "copy" to
"apply":

    object A {
      def apply (a: A) (i: Int = a.i): A = new A(i)
    }
    object B {
      def apply (b: B) (j: Int = b.j): B = new B(j)
    }
    object C {
      def apply (c: C) (s: String = c.s, i: Int = c.i): C = new C(s,i)
    }

The corresponding invocations of these methods would be more succinct:

    A(a)(i = 2)
    B(b)(j = 4)
    C(c)(s = "yo!")

If you are familiar with C++ or similar languages, you might recognize this
syntax (without the second argument list) as being similar to that of a copy
constructor. In fact, a number of people have noted that, in various languages,
copy constructors offer an alternative to an inheritable "clone" or "copy"
method; if you google "clone method vs copy constructor" you will find some
interesting comments on the issue. Ideas vary according to the language being
discussed, of course, but the preponderance of opinion seems to be that a copy
constructor, or something like it, offers less scope for trouble than an
inheritable method.

Alternative 2: Copy/Setter Methods
----------------------------------

In my own pre-2.8 code, I have often needed to make a copy of an immutable
object in which a single value is changed. Lacking an automatically generated
copy method, I have written individual methods to make these copies. These
methods amount to annoying boilerplate, and I would be grateful if a compiler
generated them automatically.

For example, for the above classes, imagine that the compiler generated:

    class A (val i: Int) {
      def i (new_i: Int): A = new A(new_i)
    }
    class B (val j: Int) extends A(1) {
      def j (new_j: Int): B = new B(new_j)
    }
    class C (val s: String, override val i: Int) extends A(i) {
      def s (new_s: String): C = new C(new_s,i)
      override def i (new_i: Int): C = new C(s,new_i)
    }

These setter-like copy methods are less prone to trouble with inheritance, as
illustrated for the class parameter i in C: where a class parameter is
overridden, it is usually of the same type in the subclass as in the superclass,
and the copy/setter method overrides the corresponding superclass method, with
the obvious meaning. (Strictly speaking, the subclass class parameter's type
might be a subtype of the corresponding superclass class parameter's type, in
which case the generated copy/setter method would overload the superclass's
method of the same name rather than overriding it. However, this does not create
an illegal declaration, as in the situation with copy$default$1 described
above.)

Invocations of copy/setter methods read nicely when chained:

    val e =
      Employee(
        firstName = "Bob",lastName = "Johnson",title = Engineer,
        hireDate = "10-Jan-2003",nationality = UnitedStates)
    val e2 =
      e.firstName("Sue").lastName("Ewings").title(Janitor).
        hireDate("7-Jul-2005")
    val e3 =
      e.firstName("Jules").lastName("Fournier").hireDate("8-Sep-2006").
        nationality(France)

although such usage presumably creates intermediate objects that immediately
become garbage (a sufficiently sophisticated VM could optimize the garbage
away).

This type of copy method also avoids requiring the name of the copied class to
appear in the invocation of the method; subclasses that override a superclass
parameter (that is, have a corresponding class parameter of the same name)
appear to have "virtual" copy methods.

Conclusion
----------

If Scala intends to continue offering full support for inheritance from concrete
classes, then either of the above approaches to an automatic copy method would
be less problematic than the current design. The two approaches are not mutually
exclusive, and in combination would allow a variety of succinct and powerful
copying idioms in Scala.

It would be too bad if the current design of the copy method led to the removal
case class inheritance, particularly since (at least with the limited testing
I've done) all pre-2.8 bugs related to case class inheritance appear to be fixed
in 2.8, and there are a number of compelling use cases for case classes that
inherit from one another (see, for example, all the use cases sited in replies
to
http://old.nabble.com/-scala--Do-you-use-case-class-inheritance--to22869...).

A


Hotmail: Powerful Free email with security by Microsoft. Get it now.
milessabin
Joined: 2008-08-11,
User offline. Last seen 33 weeks 3 days ago.
Re: 2.8 copy method needs improvement

On Mon, Feb 22, 2010 at 12:42 AM, adriaN kinG wrote:
> Nonetheless, Scala has so far placed no restrictions on inheriting from
> concrete classes,

Inheritance of *case* classes is deprecated as of 2.8.

Cheers,

Miles

Archontophoenix
Joined: 2010-02-04,
User offline. Last seen 2 years 35 weeks ago.
Re: 2.8 copy method needs improvement

>> Nonetheless, Scala has so far placed no restrictions on inheriting from
>> concrete classes,
>
> Inheritance of *case* classes is deprecated as of 2.8.

By "heretofore" I meant before 2.8.

I'm aware of the status quo with respect to case classes. I was actually hoping
for a defense, or at least a history, of the status quo.

Here is a summary of the technical points I mean to make:

1. Inheritance, both from abstract classes/traits and concrete classes, is
useful and well-established in Scala and other languages.

2. Inheritance of case classes is also useful.

3. In 2.7, there was no inconsistency in the design of case class inheritance
(although there were bugs in the implementation. The bugs in question have
been fixed in 2.8).

4. Case class inheritance has been deprecated, at least in part because of the
poor interaction of inheritance with the copy method, which is new in 2.8.

5. The new copy method is not well designed, because it is desirable to be able
to copy at more than one level of a class hierarchy; but that cannot be
implemented, given the design of the copy method.

6. The new copy method should be replaced with something more useful, and the
deprecation of case class inheritance should be rescinded.

Since I am not part of the compiler team, I realize my argument may be
erroneous. For example, there may be inconsistencies of which I am unaware in
the 2.7 case class design, or lingering case class bugs in 2.8 which I have not
been able to elicit, or virtues in the current copy method design I have not
recognized, or inconsistencies in the alternatives I'm suggesting to the current
copy method. I have been searching past postings to the mailing lists and poking
my nose into the compiler to try to understand this better. I'd welcome any
help.

Assuming I have not made a technical mistake, however, there are some
nontechnical points I am also trying to make:

1. It is often useful to pattern-match at more than one place in a class
hierarchy, so prior to 2.8, case class inheritance was widely used.

2. Not all Scala users follow the mailing lists or use prerelease versions of
Scala.

3. Those who wait to adopt Scala 2.8 until a stable release appears may be
unpleasantly surprised by the large number of deprecation warnings for the
use of what they reasonably assumed was a stable, supported feature.

4. Widespread unpleasant surprise at the deprecation of a commonly used feature
would give the impression that Scala as a whole is too unstable for
production use.

Regarding the last point, if an employer were to ask me today whether I thought
Scala were stable enough for use in a long-lived project, I could not honestly
answer with an unqualified yes.

Finally:

. The Scala team has made many excellent design decisions. Scala is, overall,
the most expressive computer language I've used.

. The changes to the language before 2.8 have not (as far as I can tell) been
gratuitously incompatible with previous versions. Where incompatibilities were
introduced, they were clear improvements (as is the case with the
reorganization of collections in 2.8).

. Changes before 2.8 have generally been in the direction of reducing rather
than increasing the need to write boilerplate code.

For these reasons, the treatment of case class inheritance and the 2.8 copy
method seems particularly incongruous.

A

David Pollak
Joined: 2008-12-16,
User offline. Last seen 42 years 45 weeks ago.
Re: Re: 2.8 copy method needs improvement


On Tue, Feb 23, 2010 at 5:18 PM, Adrian King <ceroxylon [at] hotmail [dot] com> wrote:
>> Nonetheless, Scala has so far placed no restrictions on inheriting from
>> concrete classes,
>
> Inheritance of *case* classes is deprecated as of 2.8.

By "heretofore" I meant before 2.8.

I'm aware of the status quo with respect to case classes. I was actually hoping
for a defense, or at least a history, of the status quo.

Here is a summary of the technical points I mean to make:

1. Inheritance, both from abstract classes/traits and concrete classes, is
  useful and well-established in Scala and other languages.

2. Inheritance of case classes is also useful.

Not really.  I struggled to find an example in any of my code (and I've written more real-world Scala code than almost anyone else) for an example of case class inheritance when I wrote Beginning Scala.  In practice, case classes are like structs in C.  The places where I've used case class inheritance has been a design mistake.
 

3. In 2.7, there was no inconsistency in the design of case class inheritance
  (although there were bugs in the implementation. The bugs in question have
  been fixed in 2.8).

Hmmm.... this is a bold and I think wrong statement.  Yes, case class inheritance was specified, but I think the implementation problems that led to the removal of this feature was based on an incomplete specification.  If you go back and read PaulP's case for removing this feature, you'll have a deeper understand of the corner cases.  While I'm not normally a RTFM kinda guy, I suggest before posting anything else on this subject that you read the case Paul made and his discussion with Martin on the subject.
 

4. Case class inheritance has been deprecated, at least in part because of the
  poor interaction of inheritance with the copy method, which is new in 2.8.

5. The new copy method is not well designed, because it is desirable to be able
  to copy at more than one level of a class hierarchy; but that cannot be
  implemented, given the design of the copy method.

If you've got a better *actual* implementation, then post it.  But barring an implementation, you're just throwing mud.  The copy problem is tremendously difficult in a language like Scala that supports a plethora of different ways of composing classes.
 

6. The new copy method should be replaced with something more useful, and the
  deprecation of case class inheritance should be rescinded.

See #5.  If you've got a better implementation (not just a vague design) that has a proof that it works with all manner of class composition in Scala, great... love to have it... but barring that, I'm going to make the statement that you haven't thought the problem through the way Martin and Paul have.
 

Since I am not part of the compiler team, I realize my argument may be
erroneous. For example, there may be inconsistencies of which I am unaware in
the 2.7 case class design, or lingering case class bugs in 2.8 which I have not
been able to elicit, or virtues in the current copy method design I have not
recognized, or inconsistencies in the alternatives I'm suggesting to the current
copy method. I have been searching past postings to the mailing lists and poking
my nose into the compiler to try to understand this better. I'd welcome any
help.

Assuming I have not made a technical mistake, however, there are some
nontechnical points I am also trying to make:

1. It is often useful to pattern-match at more than one place in a class
  hierarchy, so prior to 2.8, case class inheritance was widely used.

This is incorrect, at least in the code bases that I'm involved with.
 

2. Not all Scala users follow the mailing lists or use prerelease versions of
  Scala.

This is true, but the deprecation of case class inheritance (which, by the way, is a relatively new feature of Scala... circa 2.6 I think) has been widely advertised.
 

3. Those who wait to adopt Scala 2.8 until a stable release appears may be
  unpleasantly surprised by the large number of deprecation warnings for the
  use of what they reasonably assumed was a stable, supported feature.

Yeah.  2.8 is a major break from 2.7.  It should be called 3.0 (but that discussion has been had and Martin said, "it'll be called 2.8) and there will be breaking changes.  I've ported 100K+ lines of code from 2.7.7 to 2.8 and it's not so bad.  In the instant situation, if you want to pattern match against subclasses of a case class, you write an unapply method on a companion object.  Yeah, it's some work.
 

4. Widespread unpleasant surprise at the deprecation of a commonly used feature
  would give the impression that Scala as a whole is too unstable for
  production use.

Hmmm.... slinging some mud... first, the feature is not "commonly used."  Second, it's better to make sure that the compiler can generate valid code vs. the situation of pattern matching against subclassible case classes which leads to undertermined results.  I think that the former (good code) is a better way to boost Scala's reputation than a feature that should never have been introduced.

Regarding the last point, if an employer were to ask me today whether I thought
Scala were stable enough for use in a long-lived project, I could not honestly
answer with an unqualified yes.

Finally:

. The Scala team has made many excellent design decisions. Scala is, overall,
 the most expressive computer language I've used.

. The changes to the language before 2.8 have not (as far as I can tell) been
 gratuitously incompatible with previous versions. Where incompatibilities were
 introduced, they were clear improvements (as is the case with the
 reorganization of collections in 2.8).

The change is not "gratuitous".  The change was discussed and reasoned by some of the smartest people in computing and *they* were unable to come up with a solution.  You spend a day discussing types systems, ADTs, and subclassing with Martin and you'll get a glimpse of the complex intersection of these pieces.
 

. Changes before 2.8 have generally been in the direction of reducing rather
 than increasing the need to write boilerplate code.

For these reasons, the treatment of case class inheritance and the 2.8 copy
method seems particularly incongruous.

First, the copy method is new.  Used in conjunction with named parameters, it reduces boilerplate.  The only place where boilerplate is increased is subclassing case classes (having to write an unapply method in a companion object), which is a bad design decision.

So, I doubt, without some actual implementation and proofs that your implementation is correct, that you will convince anyone who has been around for this argument.

David
 

A





--
Lift, the simply functional web framework http://liftweb.net
Beginning Scala http://www.apress.com/book/view/1430219890
Follow me: http://twitter.com/dpp
Surf the harmonics
Chris Twiner
Joined: 2008-12-17,
User offline. Last seen 42 years 45 weeks ago.
Re: Re: 2.8 copy method needs improvement

I just want to point out that implementing JUST the copy itself is actually quite simple: abstract type member, virtual constructor and factory functions (already in case classes).

The restriction being that its only easy when changed in the compiler (see Kevins answer). I and others have proven that to be the case.

It should not be forgotten that Paul argued for depreciation based on other far more difficult reasons.

On Feb 24, 2010 5:05 AM, "David Pollak" <feeder [dot] of [dot] the [dot] bears [at] gmail [dot] com> wrote:



On Tue, Feb 23, 2010 at 5:18 PM, Adrian King <ceroxylon [at] hotmail [dot] com> wrote: > > >> Nonetheless, Scal...


Not really.  I struggled to find an example in any of my code (and I've written more real-world Scala code than almost anyone else) for an example of case class inheritance when I wrote Beginning Scala.  In practice, case classes are like structs in C.  The places where I've used case class inheritance has been a design mistake.
 

> > > 3. In 2.7, there was no inconsistency in the design of case class inheritance >   (although t...


Hmmm.... this is a bold and I think wrong statement.  Yes, case class inheritance was specified, but I think the implementation problems that led to the removal of this feature was based on an incomplete specification.  If you go back and read PaulP's case for removing this feature, you'll have a deeper understand of the corner cases.  While I'm not normally a RTFM kinda guy, I suggest before posting anything else on this subject that you read the case Paul made and his discussion with Martin on the subject.
 

> > > 4. Case class inheritance has been deprecated, at least in part because of the >   poor inter...


If you've got a better *actual* implementation, then post it.  But barring an implementation, you're just throwing mud.  The copy problem is tremendously difficult in a language like Scala that supports a plethora of different ways of composing classes.
 

> > > 6. The new copy method should be replaced with something more useful, and the >   deprecation...


See #5.  If you've got a better implementation (not just a vague design) that has a proof that it works with all manner of class composition in Scala, great... love to have it... but barring that, I'm going to make the statement that you haven't thought the problem through the way Martin and Paul have.
 

> > > Since I am not part of the compiler team, I realize my argument may be > erroneous. For examp...


This is incorrect, at least in the code bases that I'm involved with.
 

> > > 2. Not all Scala users follow the mailing lists or use prerelease versions of >   Scala.


This is true, but the deprecation of case class inheritance (which, by the way, is a relatively new feature of Scala... circa 2.6 I think) has been widely advertised.
 

> > > 3. Those who wait to adopt Scala 2.8 until a stable release appears may be >   unpleasantly s...


Yeah.  2.8 is a major break from 2.7.  It should be called 3.0 (but that discussion has been had and Martin said, "it'll be called 2.8) and there will be breaking changes.  I've ported 100K+ lines of code from 2.7.7 to 2.8 and it's not so bad.  In the instant situation, if you want to pattern match against subclasses of a case class, you write an unapply method on a companion object.  Yeah, it's some work.
 

> > > 4. Widespread unpleasant surprise at the deprecation of a commonly used feature >   would giv...


Hmmm.... slinging some mud... first, the feature is not "commonly used."  Second, it's better to make sure that the compiler can generate valid code vs. the situation of pattern matching against subclassible case classes which leads to undertermined results.  I think that the former (good code) is a better way to boost Scala's reputation than a feature that should never have been introduced.

> > > Regarding the last point, if an employer were to ask me today whether I thought > Scala were s...


The change is not "gratuitous".  The change was discussed and reasoned by some of the smartest people in computing and *they* were unable to come up with a solution.  You spend a day discussing types systems, ADTs, and subclassing with Martin and you'll get a glimpse of the complex intersection of these pieces.
 

> > > . Changes before 2.8 have generally been in the direction of reducing rather >  than increasi...


First, the copy method is new.  Used in conjunction with named parameters, it reduces boilerplate.  The only place where boilerplate is increased is subclassing case classes (having to write an unapply method in a companion object), which is a bad design decision.

So, I doubt, without some actual implementation and proofs that your implementation is correct, that you will convince anyone who has been around for this argument.

David
 

A





--
Lift, the simply functional web framework http://liftweb.net
Beginning Scala http://www.apress.com/book/view/1430219890
Follow me: http://twitter.com/dpp
Surf the harmonics

extempore
Joined: 2008-12-17,
User offline. Last seen 35 weeks 3 days ago.
Re: Re: 2.8 copy method needs improvement

The copy method has approximately zero to do with the deprecation of
case class inheritance. It's just one more reason were someone looking
to pad the list of reasons. The relevant reason is that there's nobody
to iron out all the hairiness in the pattern matcher, and an unavailable
feature is better than a broken one. It's one of at least a couple
hundred things I might like to improve: in terms of priority, not in the
top half. In terms of time expenditure to payoff, not in the top
9/10ths.

Your words cannot have any effect because it is not a matter of
convincing anyone that it would be better if we could all both have cake
and eat cake. It is a matter of doing the necessary work. If you would
like to hire me to work on it for you I will entertain offers, but if
your hand isn't tired from writing all those zeroes, then the offer
isn't done yet.

Archontophoenix
Joined: 2010-02-04,
User offline. Last seen 2 years 35 weeks ago.
RE: Re: 2.8 copy method needs improvement
> 3. In 2.7, there was no inconsistency in the design of case class inheritance
>   (although there were bugs in the implementation. The bugs in question have
>   been fixed in 2.8).
>
> Hmmm.... this is a bold and I think wrong statement.  Yes, case class inheritance was specified,
> but I think the implementation problems that led to the removal of this feature was based on an
> incomplete specification.  If you go back and read PaulP's case for removing this feature, you'll
> have a deeper understand of the corner cases.  While I'm not normally a RTFM kinda guy, I
> suggest before posting anything else on this subject that you read the case Paul made and his
> discussion with Martin on the subject.

Hi, David --

Thanks for your detailed reply.

I've been looking for exactly that (the case for removing case class inheritance) for the past few
weeks, and been frustrated that my search skills haven't been up to finding it. I'd be much obliged
if you could provide me a link.

I do apologize for overstating my case, and thanks in advance for helping me understand what the
real story is here.

A


Hotmail: Trusted email with powerful SPAM protection. Sign up now.
Archontophoenix
Joined: 2010-02-04,
User offline. Last seen 2 years 35 weeks ago.
RE: Re: 2.8 copy method needs improvement


> The copy method has approximately zero to do with the deprecation of
> case class inheritance. It's just one more reason were someone looking
> to pad the list of reasons. The relevant reason is that there's nobody
> to iron out all the hairiness in the pattern matcher

Hmm, does that mean that the problems in question are not in case classes
per se, but in the pattern matcher?

Would the same problems be triggered by pattern-matching on classes with
hand-written unapply methods?

> Your words cannot have any effect because it is not a matter of
> convincing anyone that it would be better if we could all both have cake
> and eat cake. It is a matter of doing the necessary work. If you would
> like to hire me to work on it for you I will entertain offers, but if
> your hand isn't tired from writing all those zeroes, then the offer
> isn't done yet.

I do understand that in real life, resources are scarce, and I apologize that
my curiosity on this issue has been so insistent as to waste your time.
Frankly, you and the whole Scala team are my heroes, all the more so
because you take the time to reply to my foolish questions. If I had the
zeros in the bank, I'd be writing the check right now.

A



Hotmail: Trusted email with Microsoft’s powerful SPAM protection. Sign up now.
extempore
Joined: 2008-12-17,
User offline. Last seen 35 weeks 3 days ago.
Re: Re: 2.8 copy method needs improvement

On Tue, Feb 23, 2010 at 11:23:46PM -0800, adriaN kinG wrote:
> Hmm, does that mean that the problems in question are not in case
> classes per se, but in the pattern matcher?

Yes, although the two things aren't meaningfully separable. Other than
a few conveniences, it is the pattern matcher behavior that makes case
classes case classes.

> Would the same problems be triggered by pattern-matching on classes
> with hand-written unapply methods?

No. A whole different set of problems instead. In fact the problems
with extractors are worse, but extractors are too useful to give up.
Case class inheritance is exactly the sort of fellow you throw into the
volcano to buy enough time to fix extractors before the lava flow hits.

odersky
Joined: 2008-07-29,
User offline. Last seen 45 weeks 6 days ago.
Re: Re: 2.8 copy method needs improvement

Case class inheritance has a lot of small annoying consequences, for
instance the fact that the Product trait is essentially untyped.
Moreover, it was implemented by turning a match on the sub-case class
into an extractor match, which can effect runtime costs. So we decided
it's not worth it and that it would be clearer if users write the
extractors themselves. I don't think we'll back down on this decision.

Cheers

David Pollak
Joined: 2008-12-16,
User offline. Last seen 42 years 45 weeks ago.
Re: Re: 2.8 copy method needs improvement


On Tue, Feb 23, 2010 at 11:12 PM, adriaN kinG <ceroxylon [at] hotmail [dot] com> wrote:
> 3. In 2.7, there was no inconsistency in the design of case class inheritance
>   (although there were bugs in the implementation. The bugs in question have
>   been fixed in 2.8).
>
> Hmmm.... this is a bold and I think wrong statement.  Yes, case class inheritance was specified,
> but I think the implementation problems that led to the removal of this feature was based on an
> incomplete specification.  If you go back and read PaulP's case for removing this feature, you'll
> have a deeper understand of the corner cases.  While I'm not normally a RTFM kinda guy, I
> suggest before posting anything else on this subject that you read the case Paul made and his
> discussion with Martin on the subject.

Hi, David --

Thanks for your detailed reply.

I've been looking for exactly that (the case for removing case class inheritance) for the past few
weeks, and been frustrated that my search skills haven't been up to finding it. I'd be much obliged
if you could provide me a link.

I do apologize for overstating my case, and thanks in advance for helping me understand what the
real story is here.

Hmmm... the following Google query: case class inheritance

yielded (in order):

http://www.scala-lang.org/node/1582
http://programming-scala.labs.oreilly.com/ch06.html
http://scalatips.tumblr.com/post/101586321/avoid-case-class-inheritance
http://old.nabble.com/-scala--Do-you-use-case-class-inheritance--td22869562.html
http://old.nabble.com/-scala--Another-strike-against-case-class-inheritance-td25418996.html

So, as you can see, a simple query on the issue turns up a fair number of posts related to the issue.  Further, the issue has been debated already (almost a year ago).  I doubt that further discussion will yield a different outcome barring someone stepping up with real, working code.
 

A


Hotmail: Trusted email with powerful SPAM protection. Sign up now.



--
Lift, the simply functional web framework http://liftweb.net
Beginning Scala http://www.apress.com/book/view/1430219890
Follow me: http://twitter.com/dpp
Surf the harmonics
extempore
Joined: 2008-12-17,
User offline. Last seen 35 weeks 3 days ago.
Re: Re: 2.8 copy method needs improvement

On Wed, Feb 24, 2010 at 09:20:44AM -0800, David Pollak wrote:
> http://scalatips.tumblr.com/post/101586321/avoid-case-class-inheritance

Don't miss this one! It gets right to the heart of the issue in a way
few can match. Way to go Jorge.

Archontophoenix
Joined: 2010-02-04,
User offline. Last seen 2 years 35 weeks ago.
Re: 2.8 copy method needs improvement

David Pollak gmail.com> writes:

> Hmmm... the following Google query: case class inheritanceyielded (in
order):http://www.scala-lang.org/node/1582http://programming-scala.labs.oreilly.com/ch06.htmlhttp://scalatips.tumblr.com/post/101586321/avoid-case-class-inheritancehttp://old.nabble.com/-scala--Do-you-use-case-class-inheritance--td22869562.htmlhttp://old.nabble.com/-scala--Another-strike-against-case-class-inheritance-td25418996.html
> So, as you can see, a simple query on the issue turns up a fair number of
posts related to the issue.  Further, the issue has been debated already (almost
a year ago).  I doubt that further discussion will yield a different outcome
barring someone stepping up with real, working code.

I had seen most of the links in question, but some were new to me. Thanks.

Curiously enough, I *can* step up with at least some working code, although I
didn't write it. One of the objections to case class inheritance is that equals
does not work properly, but case class equality changed between 2.7 and 2.8:

Welcome to Scala version 2.7.7.final...
scala> case class A (a: Int)
defined class A
scala> case class B (override val a: Int) extends A(a)
defined class B
scala> val a = A(1)
a: A = A(1)
scala> val b = B(1)
b: B = B(1)
scala> a.equals(b)
res2: Boolean = true
scala> b.equals(a)
res3: Boolean = false

but with the same declarations:

Welcome to Scala version 2.8.0.Beta1-prerelease...
...
scala> a.equals(b)
res2: Boolean = false
scala> b.equals(a)
res3: Boolean = false

The change appears to have been made in Changeset 18696, and demonstrates, I
think, that a reasonable default is possible for inherited equals (even if this
particular implementation is doomed to vanish with case class inheritance).

extempore
Joined: 2008-12-17,
User offline. Last seen 35 weeks 3 days ago.
Re: Re: 2.8 copy method needs improvement

On Fri, Feb 26, 2010 at 06:09:54PM +0000, Adrian King wrote:
> Curiously enough, I *can* step up with at least some working code,
> although I didn't write it.

Yes, I did, which is how you know you can rely upon the remainder of
what I've said on this subject.

Archontophoenix
Joined: 2010-02-04,
User offline. Last seen 2 years 35 weeks ago.
Re: 2.8 copy method needs improvement

martin odersky epfl.ch> writes:

> Case class inheritance has a lot of small annoying consequences, for
> instance the fact that the Product trait is essentially untyped.
> Moreover, it was implemented by turning a match on the sub-case class
> into an extractor match, which can effect runtime costs.

I hadn't realized there were any cases in which the pattern matcher was
able to avoid invoking unapply. I guess I should look at disassembly
listings more often. Thanks!

A

Archontophoenix
Joined: 2010-02-04,
User offline. Last seen 2 years 35 weeks ago.
RE: Re: 2.8 copy method needs improvement
> > Hmm, does that mean that the problems in question are not in case
> > classes per se, but in the pattern matcher?
>
> Yes, although the two things aren't meaningfully separable. Other than
> a few conveniences, it is the pattern matcher behavior that makes case
> classes case classes.

I think I need to back up closer to the root of my confusion regarding the
role of case classes.

First, I've been assuming that some of the methods generated automatically
for case classes are not invoked by the pattern matching mechanism. I've
been trying to gather my courage to look at the pattern matching code to
check that assumption, but frankly, I'm afraid I'll hurt my head, so I'll
just ask:

. I can see that pattern matching requires "unapply", and I can imagine it
  might also use "equals".

. I can also imagine that pattern matching might take advantage of case
  classes inheriting from "ProductN" rather than "Product", if the former
  were the case.

. But does the pattern matching mechanism also need "toString", "apply",
  "copy", or "hashCode" (although it's hardly cricket to omit hashCode if
  you generate equals)?

When I first saw case class methods that looked as if they weren't part of
the pattern-matching mechanism, I jumped to the conclusion that case classes
were actually intended to serve some other purpose. The hierarchy I
envisioned looked like:

                         (mostly-)immutable classes
                        /                          \
                       /                            \
            classes that participate           other immutable
              in pattern matching                  classes

My assumption was that case classes were intended to be the root of this
hierarchy, rather than just the left branch. I thought this was great,
because so many classes in applications I've written over the past few years
(in Java before Scala) are actually located in the right branch of the
diagram.

I expect that I will continue to write far more immutable classes that do
not participate in pattern matching than those that do. Of course, I'd like
to minimize the boilerplate code I write for such classes. Some such classes
are most naturally phrased as parts of inheritance hierarchies, whence my
interest in the relationship between case classes and inheritance.

So now I'm wondering:

. What is the actual relationship between pattern-matching and each of the
  facilities provided by a case class?

. If case classes are not the recommended mechanism for minimizing
  boilerplate in general-purpose immutable classes, then what is the best
  way to do this in Scala?

  Jorge Ortiz suggested the following formulation as a superclass for
  classes that need case-class-like features:

    import scala.runtime.ScalaRunTime
 
    trait CaseLike extends Product {
      override def toString = ScalaRunTime._toString(this)
      override def hashCode = ScalaRunTime._hashCode(this)
      override def equals(x: Any) = x match {
        case that: Product =>
          that.canEqual(this) && ScalaRunTime._equals(this, that)
        case _ => false
      }
      override def canEqual(x: Any) = x match {
        case that: Product => this.productPrefix == that.productPrefix
        case _ => false
      }
 
      override def productPrefix = this.getClass.getName
      override def productArity = parts.size
      override def productElement(n: Int) = parts(n)
  
      def parts: Seq[Any]
    }

  This covers much of what I'd like to do, but it does lack support for
  copy-with-modification, which is a pretty common operation on immutable
  classes. Copy-with-modification really has to be synthesized to be most
  useful, as in the 2.8 copy method. (I gather from other threads that the
  use of compiler plugins for synthesizing method declarations does not
  really work, at least not yet.) But as mentioned previously,
  copy-with-modification cannot look like the 2.8 copy method if it is to be
  compatible with inheritance.

  Without escape analysis, the efficiency of this approach is also inferior
  to that of synthesized methods, as some of the methods in ScalaRunTime
  create short-lived objects (hashCode and equals have been bottlenecks in
  many applications I've written, because they make heavy use of hashed data
  structures). I really should try doing some performance comparisons here
  between hand-"synthesized" methods and those in ScalaRunTime, both with
  and without the JVM -XX:+DoEscapeAnalysis switch introduced in Java 6
  update 14. If I have a chance to do this I'll report my results.

  (And of course, ScalaRunTime._hashCodeJenkins produces hash codes far more
  suitable for actual use than the current ScalaRunTime._hashCode).

A


Hotmail: Trusted email with Microsoft’s powerful SPAM protection. Sign up now.
ijuma
Joined: 2008-08-20,
User offline. Last seen 22 weeks 2 days ago.
Re: 2.8 copy method needs improvement

adriaN kinG hotmail.com> writes:
> I really should try doing some performance comparisons here  between
> hand-"synthesized" methods and those in ScalaRunTime, both with  and
> without the JVM -XX:+DoEscapeAnalysis switch introduced in Java 6
> update 14.

Note that this switch doesn't work in JDK6u18 (check the release notes).

Ismael

Archontophoenix
Joined: 2010-02-04,
User offline. Last seen 2 years 35 weeks ago.
RE: Re: 2.8 copy method needs improvement

> > the JVM -XX:+DoEscapeAnalysis switch introduced in Java 6
> > update 14.
>
> Note that this switch doesn't work in JDK6u18 (check the release notes).

Thanks for the tip!

A


Your E-mail and More On-the-Go. Get Windows Live Hotmail Free. Sign up now.
Archontophoenix
Joined: 2010-02-04,
User offline. Last seen 2 years 35 weeks ago.
RE: Re: 2.8 copy method needs improvement
I've done some benchmarking, as well as collision testing, on various implementations of equals and hashCode.

Preliminary findings:

. The ScalaRunTime._equals runs about as fast as a handwritten equals method.

. ScalaRunTime._hashCodeJenkins runs about 15 - 200 times (!) as slow as ScalaRunTime._hashCode. However, a handwritten implementation of the latest incarnation of the Jenkins hash algorithm is nearly as fast (from a few percent faster to around 30% slower) as ScalaRunTime._hashCode, so there is room for improvement. The Jenkins hash is much better distributed than ScalaRunTime._hashCode.

. -XX:+DoEscapeAnalysis (run under JDK6u17) makes no discernible difference. That may have something to do with why it was withdrawn in update 18.

I'm trying to come up with a version of the Jenkins hash that takes a Product argument, so that it could be a plug-compatible replacement for ScalaRunTime._hashCode, and do the same thing for FNV-1, FNV-1a, and MurmurHash2. Details to follow.

A

> To: scala-debate [at] listes [dot] epfl [dot] ch
> From: mlists [at] juma [dot] me [dot] uk
> Subject: [scala-debate] Re: 2.8 copy method needs improvement
> Date: Sun, 28 Feb 2010 00:16:05 +0000
>
> adriaN kinG <ceroxylon <at> hotmail.com> writes:
> > I really should try doing some performance comparisons here  between
> > hand-"synthesized" methods and those in ScalaRunTime, both with  and
> > without the JVM -XX:+DoEscapeAnalysis switch introduced in Java 6
> > update 14.
>
> Note that this switch doesn't work in JDK6u18 (check the release notes).
>
> Ismael
>

Your E-mail and More On-the-Go. Get Windows Live Hotmail Free. Sign up now.
Archontophoenix
Joined: 2010-02-04,
User offline. Last seen 2 years 35 weeks ago.
RE: Re: 2.8 copy method needs improvement
> I'm trying to come up with a version of the Jenkins hash that takes a Product
> argument, so that it could be a plug-compatible replacement for
> ScalaRunTime._hashCode, and do the same thing for FNV-1, FNV-1a, and
> MurmurHash2. Details to follow.

Austin Appleby, author of MurmurHash2, ran some quality tests on several hash
algorithms: MurmurHash2, Bob Jenkins's lookup3, FNV, and Paul Hsieh's
SuperFastHash (at http://sites.google.com/site/murmurhash/statistics). Of these
four algorithms, only MurmurHash2 and lookup3 performed well on all tests; FNV
and SuperFastHash have distributions that are poor by at least some measures.

On my machine, MurmurHash2 runs faster than lookup3, by about 10% - 40%,
depending on what is being hashed. MurmurHash2 uses lots of multiplies, and
lookup3 lots of rotates, so different relative speeds of those operations might
lead to different results on different machines.

I've coded both algorithms to take a Product argument and to avoid the
allocation of temporary objects, other than those required for boxing when
calling Product.productElement. Judging from the lack of apparent effect from
the -XX:+DoEscapeAnalysis switch in JDK 6 update 17, it may be premature to
expect VMs with escape analysis to eliminate concerns about such allocations in
time-critical code.

I also coded versions of the data structures I hashed to implement a trait that
looks like this:

    trait Hashable {
      def hashableInts: Int
      def hashableInt (n: Int): Int
    }

and coded versions of the hash algorithms to call the methods in it to obtain
the contents of the data structures as a sequence of 32-bit integers, thereby
avoiding any allocations during hashing. As expected, this resulted in a
substantial speedup, of as much as 500%. However, it is not clear that it would
be worth changing Product (for example) to include such methods, particularly
since a superior hash algorithm may one day appear that consumes its input in
something other than 32-bit chunks. (Scala should certainly avoid the mistake of
java.lang.String, which promises a specific, and decidedly low-quality, hash
function.)

I've attached implementations of lookup3 and MurmurHash2 for Product and String
to ticket #2537, which complains about the quality of the default hash code for
case classes.


Hotmail: Trusted email with powerful SPAM protection. Sign up now.

Copyright © 2012 École Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland