type matching in for comprehensions

If your translation can be expressed in plain Scala, which I think is
the case, I'd write such an example by hand and use -Ybrowse: at some
phase to understand how a tree that works is supposed to look. The
patch looks fine to me.

iulian

On Feb 26, 2009, at 10:09 PM, Paul Phillips wrote:

> I have ticket 900 (and 1089 and 1673) implemented, but with a catch.
> I've been stuck for a while trying to get to the finish line. Using
> the
> example in ticket #1673:
>
> *** current scala ***
>
> scala> val a : Any = "foo"
> a: Any = foo
>
> scala> for (s : String <- Some(a)) yield s
> :6: error: type mismatch;
> found : (String) => String
> required: (Any) => ?
> for (s : String <- Some(a)) yield s
> ^
>
> *** paulp scala ***
>
> scala> for (s : String <- Some(a)) yield s
> res0: Option[String] = Some(foo)
>
> ------------
>
> Great! Simple types work fine. The breakdown is parameterized
> types, as
> illustrated by the example in #2d2d2d.
>
> *** current scala ***
>
> scala> for (x : Some[_] <- List(None, Some(1))) yield x
> :5: error: type mismatch;
> found : (Some[_]) => Some[Any]
> required: (Option[Int]) => ?
> for (x : Some[_] <- List(None, Some(1))) yield x
> ^
>
> *** paulp scala ***
>
> scala> for (x : Some[_] <- List(None, Some(1))) yield x
> :5: error: type mismatch;
> found : (Some[_]) => Some[Any]
> required: (Some[Any]) => ?
> for (x : Some[_] <- List(None, Some(1))) yield x
> ^
>
> ------------
>
> I tried many different ways to make scalac happy and hit roadblocks
> everywhere. If anyone who is hipper to the internals than I am could
> possibly take a look at the current patch (it's not large) and give
> me a
> hint or two, I'd be a happy camper. Here it is:
>
> http://www.improving.org/scala/900.diff
>

I am not sure we *should* fix ticket 900 and the rest literally. In my
understanding, if I write

for (x: String <- xs)

that looks like a type annotation, not a pattern match.
So it was fully intentional that it should not filter xs for Strings,
and I believe we should just spec it that way.
However, putting parentheses around x: String should change the semantics.

for ((x: String) <- xs)

should do the filtering. It seems this would change the problem somewhat.

Furthermore, if you have something like

for ((x: Some[_]) <- xs) yield exp

I would translate this to

xs filter { case Some[_] => true case _ => false } map exp

That should take care of it.

Cheers

On Thu, Feb 26, 2009 at 10:54:04PM +0100, martin odersky wrote:
> that looks like a type annotation, not a pattern match.
> So it was fully intentional that it should not filter xs for Strings,
> and I believe we should just spec it that way.
> However, putting parentheses around x: String should change the semantics.

It looks like those parens survive the parser so I can make this
distinction (I hadn't really thought about the parens as yet.)

> for ((x: Some[_]) <- xs) yield exp
>
> I would translate this to
>
> xs filter { case Some[_] => true case _ => false } map exp

That formulation looks like it would be typed the same as xs. Shoudln't
it be typed according to the annotation? I.e.:

val xs = List(a, "abc", 5.5f)
val res = for ((x: String) <- xs) yield x

The filter will reduce that to List("abc"), but it'll still be List[Any]
and I would anticipate as a user that this would result in List[String].

On Thu, Feb 26, 2009 at 9:54 PM, martin odersky <martin [dot] odersky [at] epfl [dot] ch> wrote:

I am not sure we *should* fix ticket 900 and the rest literally. In my
understanding, if I write

for (x: String <- xs)

that looks like a type annotation, not a pattern match.
So it was fully intentional that it should not filter xs for Strings,
and I believe we should just spec it that way.
However, putting parentheses around x: String should change the semantics.

for ((x: String) <- xs)

should do the filtering. It seems this would change the problem somewhat.

Furthermore, if you have something like

for ((x: Some[_]) <- xs) yield exp

I would translate this to

xs filter { case Some[_] => true case _ => false } map exp

That should take care of it.

Eek.

I'm mildly in favour of Paul's change, but the idea of having the behaviour depend so significantly on the presence or absence of brackets is pretty terrifying. I'd rather not have type based filtering on for comprehensions than have something like that in - it seems like it would introduce all sorts of careless errors.

On Thu, Feb 26, 2009 at 11:11:35PM +0000, David MacIver wrote:
> I'm mildly in favour of Paul's change, but the idea of having the behaviour
> depend so significantly on the presence or absence of brackets is pretty
> terrifying. I'd rather not have type based filtering on for comprehensions
> than have something like that in - it seems like it would introduce all
> sorts of careless errors.

Spitballing on the IRC channel generates the proposal to break all
existing scala code by deprecating any uses of ":" which do filtering
and replace them by an operator such as <: thus enabling us to
distinguish whether the intent is to annotate the type (tell) or to
filter by it (ask).

// new undeprecated version
foo match {
case x <: String => x
case y <: Int => y
}

And now we have...

for (x: String <- xs) // compiler, those are strings
for (x <: String <- xs) // compiler, give me the stringy ones

I realize such a huge change is unlikely at best, but it seemed like a
good idea to me in the abstract so I thought I'd throw it out there.

Parenthetically (ha!) it turns out I was wrong and the parser
does swallow the parens which would allow distinguishing between
for (x: String <- xs) and for ((x: String) <- xs). This could
be addressed but I'm acquiring drmaciver's skepticism about the
wisdom of these being so different.

On Thu, Feb 26, 2009 at 3:36 PM, Paul Phillips <paulp [at] improving [dot] org> wrote:

// new undeprecated version
foo match {
 case x <: String => x
 case y <: Int => y
}

And now we have...

 for (x: String <- xs)   // compiler, those are strings
 for (x <: String <- xs) // compiler, give me the stringy ones

for (case x: String <- xs) ?

On Thu, Feb 26, 2009 at 03:42:59PM -0800, Alex Boisvert wrote:
> for (case x: String <- xs) ?

From a parsing and unambiguity perspective this is very appealing... I
already implemented it just in "case" this is deemed attractive.

On Thu, Feb 26, 2009 at 3:11 PM, David MacIver <david [dot] maciver [at] gmail [dot] com> wrote:

On Thu, Feb 26, 2009 at 9:54 PM, martin odersky <martin [dot] odersky [at] epfl [dot] ch> wrote:

I am not sure we *should* fix ticket 900 and the rest literally. In my
understanding, if I write

for (x: String <- xs)

that looks like a type annotation, not a pattern match.
So it was fully intentional that it should not filter xs for Strings,
and I believe we should just spec it that way.
However, putting parentheses around x: String should change the semantics.

for ((x: String) <- xs)

should do the filtering. It seems this would change the problem somewhat.

Furthermore, if you have something like

for ((x: Some[_]) <- xs) yield exp

I would translate this to

xs filter { case Some[_] => true case _ => false } map exp

That should take care of it.

Eek.

I'm mildly in favour of Paul's change, but the idea of having the behaviour depend so significantly on the presence or absence of brackets is pretty terrifying. I'd rather not have type based filtering on for comprehensions than have something like that in - it seems like it would introduce all sorts of careless errors.

+1



--
Lift, the simply functional web framework http://liftweb.net
Beginning Scala http://www.apress.com/book/view/1430219890
Follow me: http://twitter.com/dpp
Git some: http://github.com/dpp

Paul Phillips wrote:
> On Thu, Feb 26, 2009 at 03:42:59PM -0800, Alex Boisvert wrote:
>
>> for (case x: String <- xs) ?
>>
> From a parsing and unambiguity perspective this is very appealing... I
> already implemented it just in "case" this is deemed attractive.
>
I quite agree, and furthermore:

1. the existing "for (Some(x) <- List[Option[_]])" is unambiguous but
perhaps should be deprecated in favour of #2
2. "for (case Some(x) <- List[Option[_]])" is nice and explicit
3. "for (x: String <- List[Any])" is confusing and should be at least a
warning
4. "for (x: String <- List[String])" might be OK if it can be safely
distinguished from #3

And, while we're on the subject changing everything just for fun (and
making pattern matching more explicit) we didn't manage to reach any
kind of consensus on whether val (foo, bar) = exprThatProducesATuple2
should be changed; "val case" seems wrong.

-0xe1a

On Feb 27, 2009, at 11:13 AM, David Pollak wrote:

On Thu, Feb 26, 2009 at 3:11 PM, David MacIver <david [dot] maciver [at] gmail [dot] com> wrote:

On Thu, Feb 26, 2009 at 9:54 PM, martin odersky <martin [dot] odersky [at] epfl [dot] ch> wrote:

I am not sure we *should* fix ticket 900 and the rest literally. In my
understanding, if I write

for (x: String <- xs)

that looks like a type annotation, not a pattern match.

I'm mildly in favour of Paul's change, but the idea of having the behaviour depend so significantly on the presence or absence of brackets is pretty terrifying. I'd rather not have type based filtering on for comprehensions than have something like that in - it seems like it would introduce all sorts of careless errors.

+1

FWIW, it looks like a type filter to me, too.  Even though : is usually a type annotation, it means type filter when it's used in a pattern context.  This is a pattern context, isn't it?
If it's really supposed to be a type annotation, then perhaps we should come up with a different syntax for a type filter, and apply that consistently to pattern matching as well as to for comprehensions.  The new syntax could be added immediately and the old syntax deprecated out.
Lex

On Fri, Feb 27, 2009 at 11:39:30AM +1100, Lex Spoon wrote:
> FWIW, it looks like a type filter to me, too. Even though : is usually a
> type annotation, it means type filter when it's used in a pattern context.
> This is a pattern context, isn't it?

To the spec I go:

In a first step, every generator p <- e , where
p is not irrefutable (§8.1) for the type of e is replaced by

p <- e .filter { case p => true; case _ => false }

So that looks unambiguous that if p is not irrefutable, it is a filter.

A pattern p is irrefutable for a type T , if one of the following applies:
1. p is a variable pattern,
2. p is a typed pattern x : T ′ , and T <: T ′ ,
3. p is a constructor pattern c (p 1 , . . . , p n ), the type T is
an instance of class c , the primary constructor (§5.3) of type T has
argument types T1 , . . . , Tn , and each p i is irrefutable for Ti .

Only 2 could apply here, but doesn't.

So I'm not sure how this can be anything but a filter according to the
spec. I'm also not sure in what circumstance a type annotation there
would be useful.

Also of note is that as specified this filters the list down to Strings
but leaves them Anys, and it seems like it'd be quite a bit more useful
to deliver Strings. This is what I'm understanding would be required
without the map:

// the cast will always succeed!
for (x: String <- xs) yield x.asInstanceOf[String]

...which is a little off-putting.

On Thu, Feb 26, 2009 at 9:54 PM, martin odersky wrote:
> I am not sure we *should* fix ticket 900 and the rest literally. In my
> understanding, if I write
>
> for (x: String <- xs)
>
> that looks like a type annotation, not a pattern match.
> So it was fully intentional that it should not filter xs for Strings,
> and I believe we should just spec it that way.
> However, putting parentheses around x: String should change the semantics.
>
> for ((x: String) <- xs)

That's not really with way I've understood the syntax as working. I've
always read the generator as a pattern with the leading "case" implied
by the context. In that light, adding in enclosing braces seems as
redundant and unnatural as the braces here,

e match {
case (x : String) => ...
}

Alex's proposal to add the implied "case" makes sense, but it's
verbose and really doesn't feel like it should be necessary.

So Paul's original patch gets my vote.

Cheers,

Miles

On Thu, Feb 26, 2009 at 04:25:02PM -0800, Alex Cruise wrote:
> And, while we're on the subject changing everything just for fun (and
> making pattern matching more explicit) we didn't manage to reach any kind
> of consensus on whether val (foo, bar) = exprThatProducesATuple2 should be
> changed; "val case" seems wrong.

For the record I have no issue with the above being a pattern match,
except for the issue described in this longstanding ticket:

https://lampsvn.epfl.ch/trac/scala/ticket/140

I also have been bitten a number of times by it and it's quite
unintuitive that the semantics of multiple assignment are so different
from single assignment. If you write this variation on #2d2d2d,

val is: Option[String] = "a" match {
case "b" => Some("abc")
case _ => Some((1,2))
}

...of course it won't compile. It seems like even if the general
matching situation is beyond reach, the specific case of multiple
assignment and TupleN is worth addressing directly.

On Fri, 2009-02-27 at 09:26 -0800, Paul Phillips wrote:
> For the record I have no issue with the above being a pattern match,
> except for the issue described in this longstanding ticket:
>
> https://lampsvn.epfl.ch/trac/scala/ticket/140
>
> I also have been bitten a number of times by it and it's quite
> unintuitive that the semantics of multiple assignment are so different
> from single assignment.

I feel the same way.

Ismael

My initial reaction to the for comprehension issue was essentally
similar to #2d2d2d, and was caused by the old style for syntax, whic is
now abandoned. Essentially, I was worrried that in

for (val x: String <- e1; val y: String = e2)

the two annotations of ": String" would have different meanings. The
new for syntax removes this concern to some degree, as

for (x: String <- e1; val y: String = e2)

distinguishes the two bindings better. So I retract my
earlier proposition and agree that all bindings in generators should
be treated as patterns.

The alternative, of prefixing with `case' is cute, but would represent
a special case, and I am not sure the gain in expressiveness is worth
that.

For ticket #2d2d2d, I am still not sure what to do.

Cheers

On Feb 27, 2009, at 2:52 PM, martin odersky wrote:
> My initial reaction to the for comprehension issue was essentally
> similar to #2d2d2d, and was caused by the old style for syntax, whic is
> now abandoned. Essentially, I was worrried that in
>
> for (val x: String <- e1; val y: String = e2)
>
> the two annotations of ": String" would have different meanings. The
> new for syntax removes this concern to some degree, as

Oh, right, of course. -Lex

On Mar 3, 2009, at 10:44 AM, Lex Spoon wrote:
> On Feb 27, 2009, at 2:52 PM, martin odersky wrote:
>> My initial reaction to the for comprehension issue was essentally
>> similar to #2d2d2d, and was caused by the old style for syntax, whic is
>> now abandoned. Essentially, I was worrried that in
>>
>> for (val x: String <- e1; val y: String = e2)
>>
>> the two annotations of ": String" would have different meanings. The
>> new for syntax removes this concern to some degree, as
>
> Oh, right, of course. -Lex
>

You know, the same problem still exists for val when used to define a
variable. Would it make sense to use the "case" keyword when a
pattern is desired?

val x: String = foo // variable def
case x: String = foo // pattern match

Similarly:

case (a,b) = somethingReturningATuple()

-Lex