Scala 2.9.0.RC1 erasing ParameterizedType details

On 4/7/11 4:07 PM, Wille Faler wrote:
> parameterized.toString used to result in:
> scala.collection.mutable.LinkedList
>
> With Scala 2.9.0.RC1:
> scala.collection.mutable.LinkedList

Unfortunately this is a feature.

> Is this "erased" detail for ParameterizedType's using reflection an
> intended thing in Scala 2.9, and if so is there a workaround?

It is an intended thing. If you parameterize it on java.lang.Integer
instead of Int, you'll see java.lang.Integer, which may or may not count
as a workaround.

You don't want to know how much I've suffered for signatures, and for
now there was no way forward except to lose some precision as viewed via
java reflection.

On 8 Apr 2011, at 00:34, Paul Phillips wrote:

> On 4/7/11 4:07 PM, Wille Faler wrote:
>> parameterized.toString used to result in:
>> scala.collection.mutable.LinkedList
>>
>> With Scala 2.9.0.RC1:
>> scala.collection.mutable.LinkedList
>
> Unfortunately this is a feature.
>
>> Is this "erased" detail for ParameterizedType's using reflection an
>> intended thing in Scala 2.9, and if so is there a workaround?
>
> It is an intended thing. If you parameterize it on java.lang.Integer instead of Int, you'll see java.lang.Integer, which may or may not count as a workaround.
>
> You don't want to know how much I've suffered for signatures, and for now there was no way forward except to lose some precision as viewed via java reflection.

is there no other workaround? (other ways than parameterizedtype are acceptable).
If not, I would imagine this really pulls the rug from under a lot of key features in a lot of frameworks that try to map some data into arbitrary objects.

On 4/7/11 4:41 PM, Wille Faler wrote:
> is there no other workaround? (other ways than parameterizedtype are acceptable).

The information is there, you just can't get it through java reflection.
If we had a decent reflection library I'd just point you at a similar
function in that. Since we don't yet, I can point you at the raw data.

// a program
import scala.collection.mutable.LinkedList

class A {
var x1 = new LinkedList[Int]
var x2 = new LinkedList[java.lang.Integer]
}

// javap
Compiled from "bip.scala"
public class A extends java.lang.Object implements scala.ScalaObject
[...]
const #35 = Asciz Lscala/reflect/ScalaSignature;;
const #36 = Asciz bytes;
const #37 = Asciz 
y2A!

\t\tIC
aT-9usz
aE
\r9
\"a!Q!!\t1gn!!.)
3,15: TERMname 7:
4,24: NONEsym 0:
5,26: CLASSINFOtpe 3: 0 6 14
6,31: TYPEREFtpe 2: 7 12
7,35: THIStpe 1: 8
8,38: EXTMODCLASSref 2: 9(lang) 10
9,42: TERMname 4: lang
10,48: EXTMODCLASSref 1: 11(java)
11,51: TERMname 4: java
12,57: EXTref 2: 13(Object) 8
13,61: TYPEname 6: Object
14,69: TYPEREFtpe 2: 15 18
15,73: THIStpe 1: 16
16,76: EXTMODCLASSref 1: 17(scala)
17,79: TERMname 5: scala
18,86: EXTref 2: 19(ScalaObject) 16
19,90: TYPEname 11: ScalaObject
20,103: VALsym 5: 21() 0 200[] 22
21,110: TERMname 6:
22,118: METHODtpe 1: 23
23,121: TYPEREFtpe 2: 24 0
24,125: THIStpe 1: 2
25,128: VALsym 7: 26(x1) 0 8000200[ ] 27
26,137: TERMname 2: x1
27,141: POLYtpe 1: 28
28,144: TYPEREFtpe 3: 29 34 36
29,149: THIStpe 1: 30
30,152: EXTMODCLASSref 2: 31(mutable) 32
31,156: TERMname 7: mutable
32,165: EXTMODCLASSref 2: 33(collection) 16
33,169: TERMname 10: collection
34,181: EXTref 2: 35(LinkedList) 30
35,185: TYPEname 10: LinkedList
36,197: TYPEREFtpe 2: 15 37
37,201: EXTref 2: 38(Int) 16
38,205: TYPEname 3: Int
39,210: VALsym 7: 40(x1_$eq) 0 8000200[ ] 41
40,219: TERMname 6: x1_$eq
41,227: METHODtpe 2: 42 45
42,231: TYPEREFtpe 2: 15 43
43,235: EXTref 2: 44(Unit) 16
44,239: TYPEname 4: Unit
45,245: VALsym 7: 46(x$1) 39 202000[ ] 28
46,254: TERMname 3: x$1
47,259: VALsym 6: 48(x1 ) 0 81004[private ] 28
48,267: TERMname 3: x1
49,272: VALsym 7: 50(x2) 0 8000200[ ] 51
50,281: TERMname 2: x2
51,285: POLYtpe 1: 52
52,288: TYPEREFtpe 3: 29 34 53
53,293: TYPEREFtpe 2: 7 54
54,297: EXTref 2: 55(Integer) 8
55,301: TYPEname 7: Integer
56,310: VALsym 7: 57(x2_$eq) 0 8000200[ ] 58
57,319: TERMname 6: x2_$eq
58,327: METHODtpe 2: 42 59
59,331: VALsym 7: 46(x$1) 56 202000[ ] 52
60,340: VALsym 6: 61(x2 ) 0 81004[private ] 52
61,348: TERMname 3: x2

On 8 April 2011 00:45, Paul Phillips <paulp [at] improving [dot] org> wrote:

On 4/7/11 4:41 PM, Wille Faler wrote: > is there no other workaround? (other ways than parameterizedtype are acceptable). The information is there, you just can't get it through java reflection.    If we had a decent reflection library I'd just point you at a similar  function in that.  Since we don't yet, I can point you at the raw data. // a program import scala.collection.mutable.LinkedList class A {    var x1 = new LinkedList[Int]    var x2 = new LinkedList[java.lang.Integer] // javap Compiled from "bip.scala" public class A extends java.lang.Object implements scala.ScalaObject [...] const #35 = Asciz Lscala/reflect/ScalaSignature;; const #36 = Asciz bytes; const #37 = Asciz y2A! \t\t 9usz !Q!! PE*,7\r )1oY1mC& G.Y(cU =S:LGO #I5\taD ;bE2,'BA 7fGRLwN\ T5oW $G*[:u!\tyQ% '!\t 0M0%KF$\"AK \t\rA \tA('F 5!\ri\"% YJ!a %sG/Z4fe\"9 +w!9a &A'A See, it's right there! Here's our command line signature unpickler making it somewhat less  opaque.  You can see the Int:    37,201: EXTref 2: 38(Int) 16 % /scala/trunk/tools/showPickled -cp .  A Version 5.0 0,3: CLASSsym 4: 1(A) 2 0[] 5 1,9: TYPEname 1: A 2,12: EXTMODCLASSref 1: 3(<empty>) 3,15: TERMname 7: <empty> 4,24: NONEsym 0: 5,26: CLASSINFOtpe 3: 0 6 14 6,31: TYPEREFtpe 2: 7 12 7,35: THIStpe 1: 8 8,38: EXTMODCLASSref 2: 9(lang) 10 9,42: TERMname 4: lang 10,48: EXTMODCLASSref 1: 11(java) 11,51: TERMname 4: java 12,57: EXTref 2: 13(Object) 8 13,61: TYPEname 6: Object 14,69: TYPEREFtpe 2: 15 18 15,73: THIStpe 1: 16 16,76: EXTMODCLASSref 1: 17(scala) 17,79: TERMname 5: scala 18,86: EXTref 2: 19(ScalaObject) 16 19,90: TYPEname 11: ScalaObject 20,103: VALsym 5: 21(<init>) 0 200[<method>] 22 21,110: TERMname 6: <init> 22,118: METHODtpe 1: 23 23,121: TYPEREFtpe 2: 24 0 24,125: THIStpe 1: 2 25,128: VALsym 7: 26(x1) 0 8000200[<method> <accessor>] 27 26,137: TERMname 2: x1 27,141: POLYtpe 1: 28 28,144: TYPEREFtpe 3: 29 34 36 29,149: THIStpe 1: 30 30,152: EXTMODCLASSref 2: 31(mutable) 32 31,156: TERMname 7: mutable 32,165: EXTMODCLASSref 2: 33(collection) 16 33,169: TERMname 10: collection 34,181: EXTref 2: 35(LinkedList) 30 35,185: TYPEname 10: LinkedList 36,197: TYPEREFtpe 2: 15 37 37,201: EXTref 2: 38(Int) 16 38,205: TYPEname 3: Int 39,210: VALsym 7: 40(x1_$eq) 0 8000200[<method> <accessor>] 41 40,219: TERMname 6: x1_$eq 41,227: METHODtpe 2: 42 45 42,231: TYPEREFtpe 2: 15 43 43,235: EXTref 2: 44(Unit) 16 44,239: TYPEname 4: Unit 45,245: VALsym 7: 46(x$1) 39 202000[<param> <synthetic>] 28 46,254: TERMname 3: x$1 47,259: VALsym 6: 48(x1 ) 0 81004[private <mutable> <local>] 28 48,267: TERMname 3: x1 49,272: VALsym 7: 50(x2) 0 8000200[<method> <accessor>] 51 50,281: TERMname 2: x2 51,285: POLYtpe 1: 52 52,288: TYPEREFtpe 3: 29 34 53 53,293: TYPEREFtpe 2: 7 54 54,297: EXTref 2: 55(Integer) 8 55,301: TYPEname 7: Integer 56,310: VALsym 7: 57(x2_$eq) 0 8000200[<method> <accessor>] 58 57,319: TERMname 6: x2_$eq 58,327: METHODtpe 2: 42 59 59,331: VALsym 7: 46(x$1) 56 202000[<param> <synthetic>] 52 60,340: VALsym 6: 61(x2 ) 0 81004[private <mutable> <local>] 52 61,348: TERMname 3: x2

Oh dear, I actually appear to have understood that.  Should I be concerned?

--
Kevin Wright

gtalk / msn : kev [dot] lee [dot] wright [at] gmail [dot] com kev [dot] lee [dot] wright [at] gmail [dot] commail: kevin [dot] wright [at] scalatechnology [dot] com
vibe / skype: kev.lee.wrightquora: http://www.quora.com/Kevin-Wright
twitter: @thecoda

"My point today is that, if we wish to count lines of code, we should not regard them as "lines produced" but as "lines spent": the current conventional wisdom is so foolish as to book that count on the wrong side of the ledger" ~ Dijkstra

I guess one way, inefficient as it may be, is to check for all primitive types speculatively if the generic type is of java.lang.Object. Not pretty, but may just work, as there are only so many primitive types that exist..


On 8 April 2011 02:45, Kevin Wright <kev [dot] lee [dot] wright [at] gmail [dot] com> wrote:

On 8 April 2011 00:45, Paul Phillips <paulp [at] improving [dot] org> wrote:

On 4/7/11 4:41 PM, Wille Faler wrote: > is there no other workaround? (other ways than parameterizedtype are acceptable). The information is there, you just can't get it through java reflection.    If we had a decent reflection library I'd just point you at a similar  function in that.  Since we don't yet, I can point you at the raw data. // a program import scala.collection.mutable.LinkedList class A {    var x1 = new LinkedList[Int]    var x2 = new LinkedList[java.lang.Integer] // javap Compiled from "bip.scala" public class A extends java.lang.Object implements scala.ScalaObject [...] const #35 = Asciz Lscala/reflect/ScalaSignature;; const #36 = Asciz bytes; const #37 = Asciz y2A! \t\t 9usz !Q!! PE*,7\r )1oY1mC& G.Y(cU =S:LGO #I5\taD ;bE2,'BA 7fGRLwN\ T5oW $G*[:u!\tyQ% '!\t 0M0%KF$\"AK \t\rA \tA('F 5!\ri\"% YJ!a %sG/Z4fe\"9 +w!9a &A'A See, it's right there! Here's our command line signature unpickler making it somewhat less  opaque.  You can see the Int:    37,201: EXTref 2: 38(Int) 16 % /scala/trunk/tools/showPickled -cp .  A Version 5.0 0,3: CLASSsym 4: 1(A) 2 0[] 5 1,9: TYPEname 1: A 2,12: EXTMODCLASSref 1: 3(<empty>) 3,15: TERMname 7: <empty> 4,24: NONEsym 0: 5,26: CLASSINFOtpe 3: 0 6 14 6,31: TYPEREFtpe 2: 7 12 7,35: THIStpe 1: 8 8,38: EXTMODCLASSref 2: 9(lang) 10 9,42: TERMname 4: lang 10,48: EXTMODCLASSref 1: 11(java) 11,51: TERMname 4: java 12,57: EXTref 2: 13(Object) 8 13,61: TYPEname 6: Object 14,69: TYPEREFtpe 2: 15 18 15,73: THIStpe 1: 16 16,76: EXTMODCLASSref 1: 17(scala) 17,79: TERMname 5: scala 18,86: EXTref 2: 19(ScalaObject) 16 19,90: TYPEname 11: ScalaObject 20,103: VALsym 5: 21(<init>) 0 200[<method>] 22 21,110: TERMname 6: <init> 22,118: METHODtpe 1: 23 23,121: TYPEREFtpe 2: 24 0 24,125: THIStpe 1: 2 25,128: VALsym 7: 26(x1) 0 8000200[<method> <accessor>] 27 26,137: TERMname 2: x1 27,141: POLYtpe 1: 28 28,144: TYPEREFtpe 3: 29 34 36 29,149: THIStpe 1: 30 30,152: EXTMODCLASSref 2: 31(mutable) 32 31,156: TERMname 7: mutable 32,165: EXTMODCLASSref 2: 33(collection) 16 33,169: TERMname 10: collection 34,181: EXTref 2: 35(LinkedList) 30 35,185: TYPEname 10: LinkedList 36,197: TYPEREFtpe 2: 15 37 37,201: EXTref 2: 38(Int) 16 38,205: TYPEname 3: Int 39,210: VALsym 7: 40(x1_$eq) 0 8000200[<method> <accessor>] 41 40,219: TERMname 6: x1_$eq 41,227: METHODtpe 2: 42 45 42,231: TYPEREFtpe 2: 15 43 43,235: EXTref 2: 44(Unit) 16 44,239: TYPEname 4: Unit 45,245: VALsym 7: 46(x$1) 39 202000[<param> <synthetic>] 28 46,254: TERMname 3: x$1 47,259: VALsym 6: 48(x1 ) 0 81004[private <mutable> <local>] 28 48,267: TERMname 3: x1 49,272: VALsym 7: 50(x2) 0 8000200[<method> <accessor>] 51 50,281: TERMname 2: x2 51,285: POLYtpe 1: 52 52,288: TYPEREFtpe 3: 29 34 53 53,293: TYPEREFtpe 2: 7 54 54,297: EXTref 2: 55(Integer) 8 55,301: TYPEname 7: Integer 56,310: VALsym 7: 57(x2_$eq) 0 8000200[<method> <accessor>] 58 57,319: TERMname 6: x2_$eq 58,327: METHODtpe 2: 42 59 59,331: VALsym 7: 46(x$1) 56 202000[<param> <synthetic>] 52 60,340: VALsym 6: 61(x2 ) 0 81004[private <mutable> <local>] 52 61,348: TERMname 3: x2

Oh dear, I actually appear to have understood that.  Should I be concerned?

--
Kevin Wright

gtalk / msn : kev [dot] lee [dot] wright [at] gmail [dot] com kev [dot] lee [dot] wright [at] gmail [dot] commail: kevin [dot] wright [at] scalatechnology [dot] com
vibe / skype: kev.lee.wrightquora: http://www.quora.com/Kevin-Wright
twitter: @thecoda

"My point today is that, if we wish to count lines of code, we should not regard them as "lines produced" but as "lines spent": the current conventional wisdom is so foolish as to book that count on the wrong side of the ledger" ~ Dijkstra

This one has me completely stumped right now - is there ANYTHING short of inspecting raw bytecode that will give away the generic type-parameter where the type is a Java primitive?
In my case, setting values reflectively on an object and transforming to the correct type from another format doesn't work with "speculative" setting of primitives if I see a java.lang.Object type as I would have thought. On first access of the field from a type-safe context during runtime, it will throw a ClassCastException (which in theory can be long after the field has been set).
I can see a hole raft of frameworks struggling with this one (JSON and web frameworks), and also especially those still relying on old Java libraries which will be broken.

On 8 April 2011 09:29, Wille Faler <wille [dot] faler [at] gmail [dot] com> wrote:

I guess one way, inefficient as it may be, is to check for all primitive types speculatively if the generic type is of java.lang.Object. Not pretty, but may just work, as there are only so many primitive types that exist..


On 8 April 2011 02:45, Kevin Wright <kev [dot] lee [dot] wright [at] gmail [dot] com> wrote:

On 8 April 2011 00:45, Paul Phillips <paulp [at] improving [dot] org> wrote:

On 4/7/11 4:41 PM, Wille Faler wrote: > is there no other workaround? (other ways than parameterizedtype are acceptable). The information is there, you just can't get it through java reflection.    If we had a decent reflection library I'd just point you at a similar  function in that.  Since we don't yet, I can point you at the raw data. // a program import scala.collection.mutable.LinkedList class A {    var x1 = new LinkedList[Int]    var x2 = new LinkedList[java.lang.Integer] // javap Compiled from "bip.scala" public class A extends java.lang.Object implements scala.ScalaObject [...] const #35 = Asciz Lscala/reflect/ScalaSignature;; const #36 = Asciz bytes; const #37 = Asciz y2A! \t\t 9usz !Q!! PE*,7\r )1oY1mC& G.Y(cU =S:LGO #I5\taD ;bE2,'BA 7fGRLwN\ T5oW $G*[:u!\tyQ% '!\t 0M0%KF$\"AK \t\rA \tA('F 5!\ri\"% YJ!a %sG/Z4fe\"9 +w!9a &A'A See, it's right there! Here's our command line signature unpickler making it somewhat less  opaque.  You can see the Int:    37,201: EXTref 2: 38(Int) 16 % /scala/trunk/tools/showPickled -cp .  A Version 5.0 0,3: CLASSsym 4: 1(A) 2 0[] 5 1,9: TYPEname 1: A 2,12: EXTMODCLASSref 1: 3(<empty>) 3,15: TERMname 7: <empty> 4,24: NONEsym 0: 5,26: CLASSINFOtpe 3: 0 6 14 6,31: TYPEREFtpe 2: 7 12 7,35: THIStpe 1: 8 8,38: EXTMODCLASSref 2: 9(lang) 10 9,42: TERMname 4: lang 10,48: EXTMODCLASSref 1: 11(java) 11,51: TERMname 4: java 12,57: EXTref 2: 13(Object) 8 13,61: TYPEname 6: Object 14,69: TYPEREFtpe 2: 15 18 15,73: THIStpe 1: 16 16,76: EXTMODCLASSref 1: 17(scala) 17,79: TERMname 5: scala 18,86: EXTref 2: 19(ScalaObject) 16 19,90: TYPEname 11: ScalaObject 20,103: VALsym 5: 21(<init>) 0 200[<method>] 22 21,110: TERMname 6: <init> 22,118: METHODtpe 1: 23 23,121: TYPEREFtpe 2: 24 0 24,125: THIStpe 1: 2 25,128: VALsym 7: 26(x1) 0 8000200[<method> <accessor>] 27 26,137: TERMname 2: x1 27,141: POLYtpe 1: 28 28,144: TYPEREFtpe 3: 29 34 36 29,149: THIStpe 1: 30 30,152: EXTMODCLASSref 2: 31(mutable) 32 31,156: TERMname 7: mutable 32,165: EXTMODCLASSref 2: 33(collection) 16 33,169: TERMname 10: collection 34,181: EXTref 2: 35(LinkedList) 30 35,185: TYPEname 10: LinkedList 36,197: TYPEREFtpe 2: 15 37 37,201: EXTref 2: 38(Int) 16 38,205: TYPEname 3: Int 39,210: VALsym 7: 40(x1_$eq) 0 8000200[<method> <accessor>] 41 40,219: TERMname 6: x1_$eq 41,227: METHODtpe 2: 42 45 42,231: TYPEREFtpe 2: 15 43 43,235: EXTref 2: 44(Unit) 16 44,239: TYPEname 4: Unit 45,245: VALsym 7: 46(x$1) 39 202000[<param> <synthetic>] 28 46,254: TERMname 3: x$1 47,259: VALsym 6: 48(x1 ) 0 81004[private <mutable> <local>] 28 48,267: TERMname 3: x1 49,272: VALsym 7: 50(x2) 0 8000200[<method> <accessor>] 51 50,281: TERMname 2: x2 51,285: POLYtpe 1: 52 52,288: TYPEREFtpe 3: 29 34 53 53,293: TYPEREFtpe 2: 7 54 54,297: EXTref 2: 55(Integer) 8 55,301: TYPEname 7: Integer 56,310: VALsym 7: 57(x2_$eq) 0 8000200[<method> <accessor>] 58 57,319: TERMname 6: x2_$eq 58,327: METHODtpe 2: 42 59 59,331: VALsym 7: 46(x$1) 56 202000[<param> <synthetic>] 52 60,340: VALsym 6: 61(x2 ) 0 81004[private <mutable> <local>] 52 61,348: TERMname 3: x2

Oh dear, I actually appear to have understood that.  Should I be concerned?

--
Kevin Wright

gtalk / msn : kev [dot] lee [dot] wright [at] gmail [dot] com kev [dot] lee [dot] wright [at] gmail [dot] commail: kevin [dot] wright [at] scalatechnology [dot] com
vibe / skype: kev.lee.wrightquora: http://www.quora.com/Kevin-Wright
twitter: @thecoda

"My point today is that, if we wish to count lines of code, we should not regard them as "lines produced" but as "lines spent": the current conventional wisdom is so foolish as to book that count on the wrong side of the ledger" ~ Dijkstra

On 4/9/11 4:12 PM, Wille Faler wrote:

> This one has me completely stumped right now - is there ANYTHING
> short of inspecting raw bytecode that will give away the generic
> type-parameter where the type is a Java primitive?

I'm not sure what counts as "inspecting raw bytecode". The mechanisms
in java.lang.reflect.Method aren't anything more than a wrapper around
raw bytecode. Here's another. (Example only, not proposing you start
embedding scalap.)

% echo "class B { var f = List(10) }" > b.scala ; \
scalac b.scala ; scalap B | grep 'var f'

var f : scala.collection.immutable.List[scala.Int] = { /* compiled code */ }

> In my case, setting values reflectively on an object and transforming
> to the correct type from another format doesn't work with
> "speculative" setting of primitives if I see a java.lang.Object type
> as I would have thought. On first access of the field from a
> type-safe context during runtime, it will throw a ClassCastException
> (which in theory can be long after the field has been set).

I need more elaboration to understand the problem. You are reflectively
manipulating a field, assuming it not to be (say) an Int in the mind of
scala based on the reflective signature saying nothing more specific
than "Object" and then getting a CCE later. Is that right? Have you
been getting away with that until now? Is this something people commonly do?

> I can see a hole raft of frameworks struggling with this one (JSON
> and web frameworks), and also especially those still relying on old
> Java libraries which will be broken.

In case it's not clear, the present changes were made trying to unbreak
things. It's possible that the present approach can be relaxed: the
exact boundaries of the signature problem are quite elusive, and because
I personally don't use the tools where things break, I do not have my
usual confidence levels in the solution. But the ball is in your court
to assemble the evidence, because the issue has already burned way too
much time and there's no way to get across how much stuff starts coming
my way at this point. Search trac for "signature" to get a sense of what
you are up against.

To give you the whole scenario behind what I am doing:I am looking to convert a raw HTTP Request into more strongly typed objects without having the API user having to be bothered with manually mapping between HTTP Request and objects.
I would expect that numerous frameworks do similar sort of "magic" on behalf of the API user.I know for a fact that Jerkson, a Scala wrapper around the Jackson JSON library have ended up re-packaging a swathe of scalap stuff to get around the issues in migrating to 2.9.

In the specific instance I am dealing with, I have a method as follows: def instantiateWithProperties[T](cls: Class[T], Map[String, Any])
What then follows is: - an instantiation of the class into an object- reflectively setting all fields that match keys in the provided Map.- The Map may contain Strings or Lists of Strings (it's basically a representation of request parameters from an HTTP Request, without the HTTP). - the properties are set by inspecting the field signatures on the object and trying to apply a conversion from the String or list of Strings to the correct type. In the case of Option or Java- or Scala Collections, the conversion needs to work out the generic type of the Option or Collection.
The process works fine, but problems arise with Scala 2.9 when using Int, Long, Double, Float or Boolean values, as Scala's version of these still seem to be wrapping their corresponding Java primitive types rather than java.lang.Integer and so on.
Losing the type in compiled code with ParameterizedType to only show "java.lang.Object" for these types is where the problem occurs.
If the change causing these issues was matched by a similar move of Scala's Int, Long, Double, Float, Boolean etc to their class equivalents in Java, rather than primitive equivalents, I suspect this would largely be a non-issue.

On 10 April 2011 01:24, Paul Phillips <paulp [at] improving [dot] org> wrote:

On 4/9/11 4:12 PM, Wille Faler wrote:

This one has me completely stumped right now - is there ANYTHING
short of inspecting raw bytecode that will give away the generic
type-parameter where the type is a Java primitive?

I'm not sure what counts as "inspecting raw bytecode".  The mechanisms
in java.lang.reflect.Method aren't anything more than a wrapper around
raw bytecode.  Here's another.  (Example only, not proposing you start
embedding scalap.)

% echo "class B { var f = List(10) }" > b.scala ; \
 scalac b.scala ; scalap B | grep 'var f'

var f : scala.collection.immutable.List[scala.Int] = { /* compiled code */ }

In my case, setting values reflectively on an object and transforming
to the correct type from another format doesn't work with
"speculative" setting of primitives if I see a java.lang.Object type
as I would have thought. On first access of the field from a
type-safe context during runtime, it will throw a ClassCastException
 (which in theory can be long after the field has been set).

I need more elaboration to understand the problem.  You are reflectively
manipulating a field, assuming it not to be (say) an Int in the mind of
scala based on the reflective signature saying nothing more specific
than "Object" and then getting a CCE later.  Is that right? Have you
been getting away with that until now? Is this something people commonly do?

I can see a hole raft of frameworks struggling with this one (JSON
and web frameworks), and also especially those still relying on old
Java libraries which will be broken.

In case it's not clear, the present changes were made trying to unbreak things. It's possible that the present approach can be relaxed: the exact boundaries of the signature problem are quite elusive, and because I personally don't use the tools where things break, I do not have my usual confidence levels in the solution. But the ball is in your court to assemble the evidence, because the issue has already burned way too much time and there's no way to get across how much stuff starts coming my way at this point. Search trac for "signature" to get a sense of what you are up against.

In Squeryl, the library user has to supply a no arg constructor when there are Option[] fields in a "plain old scala object".
Some()'s are stuffed with the appropriate values, so that types can be deduced :

class Student(val name: String, val lastName: String, val age: Option[Int], val gender: Int, val addressId: Option[Int]) {
  def this() = this("","",Some(0),0, Some(0), Some(false))
}

Recently someone contributed a patch to "guess" the type inside the Option, so the user only has to declare :

class Student(val name: String, val lastName: String, val age: Option[Int], val gender: Int, val addressId: Option[Int])

This uses the reflection trick that Wille is talking about.

The trick is no longer working.
This new feature of Squeryl is not yet documented, so no one will complain about the loss of this possibility in 2.9.0-RC1,
but it's s still unfortunate that people will be declaring things like :

  def this() = this("","",Some(0),0, Some(0), Some(false))

or even more verbose :

  class Student(val name: String, val lastName: String, val age: @Column(optionType=classOf[Int]) Option[Int], val gender: Int, @Column(optionType=classOf[Int]) val addressId: Option[Int])

for the forseable future.

ML

On Sat, Apr 9, 2011 at 9:29 PM, Wille Faler <wille [dot] faler [at] gmail [dot] com> wrote:

To give you the whole scenario behind what I am doing:I am looking to convert a raw HTTP Request into more strongly typed objects without having the API user having to be bothered with manually mapping between HTTP Request and objects.
I would expect that numerous frameworks do similar sort of "magic" on behalf of the API user.I know for a fact that Jerkson, a Scala wrapper around the Jackson JSON library have ended up re-packaging a swathe of scalap stuff to get around the issues in migrating to 2.9.

In the specific instance I am dealing with, I have a method as follows: def instantiateWithProperties[T](cls: Class[T], Map[String, Any])
What then follows is: - an instantiation of the class into an object- reflectively setting all fields that match keys in the provided Map.- The Map may contain Strings or Lists of Strings (it's basically a representation of request parameters from an HTTP Request, without the HTTP). - the properties are set by inspecting the field signatures on the object and trying to apply a conversion from the String or list of Strings to the correct type. In the case of Option or Java- or Scala Collections, the conversion needs to work out the generic type of the Option or Collection.
The process works fine, but problems arise with Scala 2.9 when using Int, Long, Double, Float or Boolean values, as Scala's version of these still seem to be wrapping their corresponding Java primitive types rather than java.lang.Integer and so on.
Losing the type in compiled code with ParameterizedType to only show "java.lang.Object" for these types is where the problem occurs.
If the change causing these issues was matched by a similar move of Scala's Int, Long, Double, Float, Boolean etc to their class equivalents in Java, rather than primitive equivalents, I suspect this would largely be a non-issue.

On 10 April 2011 01:24, Paul Phillips <paulp [at] improving [dot] org> wrote:

On 4/9/11 4:12 PM, Wille Faler wrote:

This one has me completely stumped right now - is there ANYTHING
short of inspecting raw bytecode that will give away the generic
type-parameter where the type is a Java primitive?

I'm not sure what counts as "inspecting raw bytecode".  The mechanisms
in java.lang.reflect.Method aren't anything more than a wrapper around
raw bytecode.  Here's another.  (Example only, not proposing you start
embedding scalap.)

% echo "class B { var f = List(10) }" > b.scala ; \
 scalac b.scala ; scalap B | grep 'var f'

var f : scala.collection.immutable.List[scala.Int] = { /* compiled code */ }

In my case, setting values reflectively on an object and transforming
to the correct type from another format doesn't work with
"speculative" setting of primitives if I see a java.lang.Object type
as I would have thought. On first access of the field from a
type-safe context during runtime, it will throw a ClassCastException
 (which in theory can be long after the field has been set).

I need more elaboration to understand the problem.  You are reflectively
manipulating a field, assuming it not to be (say) an Int in the mind of
scala based on the reflective signature saying nothing more specific
than "Object" and then getting a CCE later.  Is that right? Have you
been getting away with that until now? Is this something people commonly do?

I can see a hole raft of frameworks struggling with this one (JSON
and web frameworks), and also especially those still relying on old
Java libraries which will be broken.

In case it's not clear, the present changes were made trying to unbreak things. It's possible that the present approach can be relaxed: the exact boundaries of the signature problem are quite elusive, and because I personally don't use the tools where things break, I do not have my usual confidence levels in the solution. But the ball is in your court to assemble the evidence, because the issue has already burned way too much time and there's no way to get across how much stuff starts coming my way at this point. Search trac for "signature" to get a sense of what you are up against.

On 4/9/11 6:56 PM, Maxime Lévesque wrote:
> Recently someone contributed a patch to "guess" the type inside the
> Option, so the user only has to declare :
>
> class Student(val name: String, val lastName: String, val age:
> Option[Int], val gender: Int, val addressId: Option[Int])

Oh yeah, I just remembered that's how this works.

https://github.com/paulp/optional

Anyway, you guys don't need to convince me that this sucks. What you
need to do is solve the problem better. I'm not a magician, and if it's
a choice between eclipse working and reflection conveniences working,
eclipse is going to win. I simply cannot table every other facet of the
distribution to dive into this issue again. You guys are
enthusiastically encouraged to research the situation and assemble a
convincing argument that the boxed types can be used without breaking
things.

Thanks for all the input Paul, I might have found a solution that is workable, though at the cost of making scalap.jar a dependency of any project that wants to use this.
Basically, I looked at the Main.scala for the scalap.jar and tried to extract and slightly change the code where needed. I've got a rough gist of it here: https://gist.github.com/912586 (and yes, it is basically Main.scala from scalap.jar slightly modified to be usable from within an application).
This is just a spike to see that it works, so might still be fragile, but from what I can see, it should work as a workaround - just get the signature, parse the resulting String and work out the generic types.
Not sure if this comes with a performance penalty, but time will tell..
Once I'm confident this will work, I'll put the code into my little Scala utilities library here: https://github.com/wfaler/recursivity-commons
Thanks again.Wille Faler

2011/4/10 Paul Phillips <paulp [at] improving [dot] org>

On 4/9/11 6:56 PM, Maxime Lévesque wrote:

Recently someone contributed a patch to "guess" the type inside the
Option, so the user only has to declare :

class Student(val name: String, val lastName: String, val age:
Option[Int], val gender: Int, val addressId: Option[Int])

Oh yeah, I just remembered that's how this works.

 https://github.com/paulp/optional

Anyway, you guys don't need to convince me that this sucks.  What you need to do is solve the problem better.  I'm not a magician, and if it's a choice between eclipse working and reflection conveniences working, eclipse is going to win.  I simply cannot table every other facet of the distribution to dive into this issue again.  You guys are enthusiastically encouraged to research the situation and assemble a convincing argument that the boxed types can be used without breaking things.

Hi Paul,
It seems to me that in most cases when the Java compiler needs to indicate a primitive type during reflection it uses a special Class rather than the boxed type.  For instance, if a method returns int you wouldn't find Class[java.lang.Integer] you would see it as java.lang.Integer.Type which the javadoc explains as "The Class instance representing the primitive type int".  Would this be a possible alternative?  Since these special types are there to indicate a primitive value I would hope that using them wouldn't lead to VerifyErrors.  If anyone else is interested in why the change was made, here is one of the bugs that Paul was referring to: http://lampsvn.epfl.ch/trac/scala/ticket/4214.
Thanks.

On 4/19/11 8:29 AM, Dave Whittaker wrote:
> It seems to me that in most cases when the Java compiler needs to
> indicate a primitive type during reflection it uses a special Class
> rather than the boxed type. For instance, if a method returns int you
> wouldn't find Class[java.lang.Integer] you would see it as
> java.lang.Integer.Type which the javadoc explains as "The Class instance
> representing the primitive type int". Would this be a possible
> alternative?

The class objects have no bearing, except inasmuch as java reflection
will return a class object based on what is in the signature. The issue
is what is the textual content of the generic signatures, which look
something like this:

Lscala/collection/immutable/List;

They can look like this:

Lscala/collection/immutable/List;

But they cannot look like this, both because the generic signature
specification prohibits it and it breaks java software which doesn't
expect it. ("I" is the signature of primitive Int, which corresponds to
java.lang.Integer.TYPE.)

Lscala/collection/immutable/List;

Here's an email I sent somewhere a few weeks ago. It implies a part two
is coming but I haven't finished it.

Scala's Signature Dish, Chapter I

1) This ends with a puzzle. You like puzzles, don't you?
2) A couple days ago retronym took me to the restaurant "Scala" where I
actually ordered their signature dish. No enlightenments yet but I'm
still hopeful.

I am going to attempt to distill the signature problem and why it isn't
fixed yet, but in case I can draw in more interested parties by
uncharacteristically documenting a little bit, here is a quick overview
of signatures. There are two bytecode constructs which communicate the
key characteristics of the blobs of data you find in there. The older,
pre-generics one is called a "descriptor" and the one tacked on later a
"signature", although the latter term is often misused for both. Fields
and methods get both varieties; class definitions only get signatures.
Descriptors have no knowledge of type parameters, but they are
approximately equivalent to a signature after erasing the types.

Why do we care about signatures? Here is how things look to java, and
everything else in the world beyond the scala compiler (which reads its
own, much richer signature format when parsing scala bytecode, but
relies on java generic signatures for reading java since that's all
there is.)

// ...with signatures
class Cell[T](val x: T) // put a T in, get a T out
// ...without signatures
class Cell(val x: Object) // put whatever you want in, take an Object out

To see some sweet signature action, download a 2.9 nightly and:

// define this in the repl
class Bippy[A <: Function0[_ <: Comparable[_]]] extends Mutable {
var x: List[A] = _
def f[B, C <: A](x: Int, y: B) = ((x, y))
}

scala> :javap -verbose Bippy
// trimmed to relevant constant pool entries
const #3 = Asciz x;
const #4 = Asciz Lscala/collection/immutable/List;;
const #5 = Asciz Lscala/collection/immutable/List;;
const #18 = Asciz f;
const #19 = Asciz (ILjava/lang/Object;)Lscala/Tuple2;;
const #38 = Asciz
(ITB;)Lscala/Tuple2;;
const #45 = Asciz
;>;>Ljava/lang/Object;Lscala/Mutable;Lscala/ScalaObject;;

[4] field descriptor // notice the raw List type
[5] field signature // "TA;" means "type parameter A"
[19] method descriptor // "I" is primitive Int; "L...;" is a class.
[38] method signature // "C:TA" means tparam C is bounded by A.
[45] class signature // "+" is covariance; "*" a wildcard.

Notice there is a lot more information in the generic signatures. That
is the point. There are a number of restrictions on exactly how you
create a signature. The most important for our purposes is that you
cannot parameterize a type on primitives. The one-character primitive
type tags cannot appear as type arguments or type bounds unless you
enjoy the music of breakage.

Before we come to the next chapter wherein we examine the problem,
consider the following scala code. The questions for you are:

1) What signatures should go into classes A and B?
1b) "Eclipse crashes." Pay $50 and go back to square 1.
2) Given those signatures, does the java code compile?
2b) "You broke reflection." Lose a turn.
3) Try again. Would you like to buy some bridge methods?
3b) The troll grabbed you! Pay $100 or be eaten and lose infinity turns.
4) Do you like apples? How do you like these apples?

// it's some scala code
trait A[T] {
def f(): T = Predef.error("")
def g(x: Int) = ()
}
class B extends A[Int] { }

// and some java code
public class J {
public void sigs() {
B b = new B();
b.g(b.f());
}
}

Paul,
Thanks for forwarding that on, I think I understand the problem now.  Could you possibly clarify where the signature that does contain the primitive type is stored within the class?  I'm thinking about using a library like ASM to extract the un-cripled signature and any hint you can give me about where to find it would be helpful (I tried to absorb the "unpickled" output but I'm afraid I don't speak JVM byte code very well).  When I've got it worked out I'll try to post an example to Github so it can be used by others with the same problem.
-Dave

On Tue, Apr 19, 2011 at 12:54 PM, Paul Phillips <paulp [at] improving [dot] org> wrote:

On 4/19/11 8:29 AM, Dave Whittaker wrote:
> It seems to me that in most cases when the Java compiler needs to
> indicate a primitive type during reflection it uses a special Class
> rather than the boxed type.  For instance, if a method returns int you
> wouldn't find Class[java.lang.Integer] you would see it as
> java.lang.Integer.Type which the javadoc explains as "The Class instance
> representing the primitive type int".  Would this be a possible
> alternative?

The class objects have no bearing, except inasmuch as java reflection
will return a class object based on what is in the signature.  The issue
is what is the textual content of the generic signatures, which look
something like this:

 Lscala/collection/immutable/List<TA;>;

They can look like this:

 Lscala/collection/immutable/List<Ljava/lang/Integer;>;

But they cannot look like this, both because the generic signature
specification prohibits it and it breaks java software which doesn't
expect it.  ("I" is the signature of primitive Int, which corresponds to
java.lang.Integer.TYPE.)

 Lscala/collection/immutable/List<I>;

Here's an email I sent somewhere a few weeks ago.  It implies a part two
is coming but I haven't finished it.



Scala's Signature Dish, Chapter I

1) This ends with a puzzle.  You like puzzles, don't you?
2) A couple days ago retronym took me to the restaurant "Scala" where I
actually ordered their signature dish.  No enlightenments yet but I'm
still hopeful.

I am going to attempt to distill the signature problem and why it isn't
fixed yet, but in case I can draw in more interested parties by
uncharacteristically documenting a little bit, here is a quick overview
of signatures.  There are two bytecode constructs which communicate the
key characteristics of the blobs of data you find in there.  The older,
pre-generics one is called a "descriptor" and the one tacked on later a
"signature", although the latter term is often misused for both.  Fields
and methods get both varieties; class definitions only get signatures.
Descriptors have no knowledge of type parameters, but they are
approximately equivalent to a signature after erasing the types.

Why do we care about signatures? Here is how things look to java, and
everything else in the world beyond the scala compiler (which reads its
own, much richer signature format when parsing scala bytecode, but
relies on java generic signatures for reading java since that's all
there is.)

 // ...with signatures
 class Cell[T](val x: T)   // put a T in, get a T out
 // ...without signatures
 class Cell(val x: Object) // put whatever you want in, take an Object out

To see some sweet signature action, download a 2.9 nightly and:

 // define this in the repl
 class Bippy[A <: Function0[_ <: Comparable[_]]] extends Mutable {
   var x: List[A] = _
   def f[B, C <: A](x: Int, y: B) = ((x, y))
 }

 scala> :javap -verbose Bippy
 // trimmed to relevant constant pool entries
 const #3 = Asciz    x;
 const #4 = Asciz    Lscala/collection/immutable/List;;
 const #5 = Asciz    Lscala/collection/immutable/List<TA;>;;
 const #18 = Asciz    f;
 const #19 = Asciz    (ILjava/lang/Object;)Lscala/Tuple2;;
 const #38 = Asciz
<B:Ljava/lang/Object;C:TA;>(ITB;)Lscala/Tuple2<Ljava/lang/Object;TB;>;;
 const #45 = Asciz
<A::Lscala/Function0<+Ljava/lang/Comparable<*>;>;>Ljava/lang/Object;Lscala/Mutable;Lscala/ScalaObject;;

 [4] field descriptor    // notice the raw List type
 [5] field signature     // "TA;" means "type parameter A"
 [19] method descriptor  // "I" is primitive Int; "L...;" is a class.
 [38] method signature   // "C:TA" means tparam C is bounded by A.
 [45] class signature    // "+" is covariance; "*" a wildcard.

Notice there is a lot more information in the generic signatures.  That
is the point.  There are a number of restrictions on exactly how you
create a signature.  The most important for our purposes is that you
cannot parameterize a type on primitives.  The one-character primitive
type tags cannot appear as type arguments or type bounds unless you
enjoy the music of breakage.

Before we come to the next chapter wherein we examine the problem,
consider the following scala code.  The questions for you are:

1) What signatures should go into classes A and B?
1b) "Eclipse crashes."  Pay $50 and go back to square 1.
2) Given those signatures, does the java code compile?
2b) "You broke reflection."  Lose a turn.
3) Try again.  Would you like to buy some bridge methods?
3b) The troll grabbed you! Pay $100 or be eaten and lose infinity turns.
4) Do you like apples? How do you like these apples?

 // it's some scala code
 trait A[T] {
   def f(): T = Predef.error("")
   def g(x: Int) = ()
 }
 class B extends A[Int] { }

 // and some java code
 public class J {
   public void sigs() {
     B b = new B();
     b.g(b.f());
   }
 }

On Tue, Apr 19, 2011 at 8:50 PM, David Whittaker wrote:
> Thanks for forwarding that on, I think I understand the problem now.  Could
> you possibly clarify where the signature that does contain the primitive
> type is stored within the class?  I'm thinking about using a library like
> ASM to extract the un-cripled signature and any hint you can give me about
> where to find it would be helpful (I tried to absorb the "unpickled" output
> but I'm afraid I don't speak JVM byte code very well).  When I've got it
> worked out I'll try to post an example to Github so it can be used by others
> with the same problem.

A few people have already done this. An example:

https://github.com/codahale/jerkson/commit/7069e58283d17b96d76f632a38e0d...

Best,
Ismael