This page is no longer maintained — Please continue to the home page at www.scala-lang.org

Example Issue with Scala API Documentation and Scala Complexity

61 replies
dcsobral
Joined: 2009-04-23,
User offline. Last seen 38 weeks 5 days ago.
Re: Re: Example Issue with Scala API Documentation and Scala C

I'll simplify this for you. If I do even this:

"ls".!

It will leak file descriptors. That is not something to be documented,
it has to be FIXED, because there's no way to close them. I'm trying
to get in touch with Mark Harrah to get a better understanding of it
before I open a ticket, but this is definitely a bug.

On Wed, Nov 30, 2011 at 13:14, Eric Kolotyluk wrote:
> On 2011-11-30 1:54 AM, Alan Burlison wrote:
>>
>> On 30/11/2011 02:07, Daniel Sobral wrote:
>>
>>>> Aren't those methods on ProcessBuilder rather than ProcessIO?  I did
>>>> look at
>>>> ProcessBuilder but I needed to feed the output into the XML parser
>>>> classes
>>>> so I ended up just using Process and ProcessIO directly.  Plus I don't
>>>> really like the ProcessBuilder interface, although that's purely a
>>>> personal
>>>> taste thing - I can accept that other people think it is neat.
>>>
>>> They are not on ProcessIO, that's for sure.  They appear on classes
>>> that are actually package-private, so one can't browse them from
>>> Scaladoc. And, on that note, BasicIO was originally private, but
>>> became public when the library was transposed to Scala, for some
>>> reason.
>>
>> Ah, that explains it - thanks for the detail, although I think my thoughts
>> were something along the lines of "Ugh..." :-)
>>
>
> OK, I've been following this subthread, but I cannot figure out the
> conclusion.
>
> So, is there anything about Process, ProcessIO or ProcessLogger that needs
> to be documented as as warning to to the user?
>
>
>

dcsobral
Joined: 2009-04-23,
User offline. Last seen 38 weeks 5 days ago.
Re: Re: Example Issue with Scala API Documentation and Scala C

I'd go with ResultType. Yes, it is a bit redundant in that the
namespace for it *is* the type namespace, but I think it makes reading
it easier.

On Wed, Nov 30, 2011 at 13:45, Rex Kerr wrote:
> I would go with something shorter than ResultCollection; the signatures are
> long enough already to be hard to parse.  "Result" at most, I'd say.
>
>   --Rex
>
> On Wed, Nov 30, 2011 at 10:38 AM, Josh Suereth
> wrote:
>>
>> "That" is used an awful lot in the collection library, and I agree that
>> better name (like ResultCollection) could be used.  It also helps understand
>> CanBuildFrom ->
>> CanBuildFrom[List[A], B, ResultCollection] // OH, right, building from
>> List to some other collection, not sure about B yet...
>> - Josh
>>
>> On Wed, Nov 30, 2011 at 10:33 AM, Rex Kerr wrote:
>>>
>>> Related to the API documentation discussion--Ittay just pointed out a
>>> confusing-looking method signature:
>>>
>>> def ++ [B >: A, That] (that: TraversableOnce[B])(implicit bf:
>>> CanBuildFrom[List[A], B, That]) : That
>>>
>>> Although he was focused on a different aspect, I noticed a potential
>>> problem with this one. Is there any chance that we can avoid having both a
>>> "That" type and a "that" argument of a completely different type?  This is a
>>> source change, not a documentation change, but it might make the method
>>> signature less impenetrable.
>>>
>>>   --Rex
>>>
>>> P.S. Sorry--don't have time to make the changes myself, just point out
>>> the issue.
>>>
>>
>
>

dcsobral
Joined: 2009-04-23,
User offline. Last seen 38 weeks 5 days ago.
Re: Re: Example Issue with Scala API Documentation and Scala C

On Wed, Nov 30, 2011 at 14:02, Nate Nystrom wrote:
> A related issue I have is that it would be nice if one could use the methods
> in the same way as described in the scaladoc use cases, leaving off the
> "That" entirely.  For instance, List.map has the use case:
> def map [B] (f: (A) => B): List[B]
> But, then I try this:
> scala> List(1,2,3).map[String](_.toString)
> :8: error: wrong number of type parameters for method map: [B,
> That](f: Int => B)(implicit bf:
> scala.collection.generic.CanBuildFrom[List[Int],B,That])That
>               List(1,2,3).map[String](_.toString)
>                              ^
> The cleanest way I can think of to do this is to overload _ yet again, so
> that you'd write:
> List(1,2,3).map[String, _](_.toString)
> and change the scaladoc for the use case to:
> def map [B, _] (f: (A) => B): List[B]

It could also be done like this:

List(1, 2, 3).map[B = String](_.toString)

In other words, in the presence of equal sign, do it strictly by type
parameter name, and infer all other parameters. Perhaps even this
could work:

List(1, 2, 3).map[B <: String](_.toString)

Which would not infer B completely, but add a further refinement.

Of course, if that happens, I'll *personally* review every single type
argument in the library to avoid getting wedded to a bad name choice.

>
> Nate
>
> On 30 Nov 2011, at 16:45, Rex Kerr wrote:
>
> I would go with something shorter than ResultCollection; the signatures are
> long enough already to be hard to parse.  "Result" at most, I'd say.
>
>   --Rex
>
> On Wed, Nov 30, 2011 at 10:38 AM, Josh Suereth
> wrote:
>>
>> "That" is used an awful lot in the collection library, and I agree that
>> better name (like ResultCollection) could be used.  It also helps understand
>> CanBuildFrom ->
>> CanBuildFrom[List[A], B, ResultCollection] // OH, right, building from
>> List to some other collection, not sure about B yet...
>> - Josh
>>
>> On Wed, Nov 30, 2011 at 10:33 AM, Rex Kerr wrote:
>>>
>>> Related to the API documentation discussion--Ittay just pointed out a
>>> confusing-looking method signature:
>>>
>>> def ++ [B >: A, That] (that: TraversableOnce[B])(implicit bf:
>>> CanBuildFrom[List[A], B, That]) : That
>>>
>>> Although he was focused on a different aspect, I noticed a potential
>>> problem with this one. Is there any chance that we can avoid having both a
>>> "That" type and a "that" argument of a completely different type?  This is a
>>> source change, not a documentation change, but it might make the method
>>> signature less impenetrable.
>>>
>>>   --Rex
>>>
>>> P.S. Sorry--don't have time to make the changes myself, just point out
>>> the issue.
>>>
>>
>
>
>

ScottC 2
Joined: 2011-01-30,
User offline. Last seen 42 years 45 weeks ago.
Re: Re: Example Issue with Scala API Documentation and Scala Co

On Tue, Nov 29, 2011 at 6:04 PM, Daniel Sobral wrote:
> On Tue, Nov 29, 2011 at 22:52, Alan Burlison wrote:
>> On 30/11/2011 00:02, Daniel Sobral wrote:
>>
>>> But the Source that is used by ProcessIO is *NOT* scala.io.Source. It
>>> is scala.sys.process.ProcessBuilder.Source -- an altogether different
>>> beast.
>>
>> That just happen to share the same name.  How ... helpful.
>
> Tell me about it. :-/
>
>>> Besides, Source doesn't "leak" file descriptors -- it just doesn't
>>> close them automatically.
>>
>> It doesn't document that behaviour, so unless you happen to know about it
>> it's pretty certain you are going to end up with a FD leak.  Sure it's easy
>> enough to add a close() but it should be in the docs, and it isn't.
>
> Yes, sure. And we bloggers are also guilty with our cool one-liners
> that don't close the Source.
>
> I'm just making the distinction of a situation where the user _can't_
> do anything about it from one where he _should_ be doing something.
>

I am pointing out that anywhere framework code holds on to system
resources, it _should_ have a finalize() or equivalent WeakReference
trigger to allow GC to clean up leaks eventually, and
non-deterministically, or rely on the underlying Java class to do the
same. That way, even with user error, the OS resources are not leaked
permanently.

Most likely, this case is using a low level Java class that already
handles cleanup that way when users don't call close(). If so I
suspect a gc call or two would clear the OS resources when the user
does not manually close the Source if the source was out of scope and
able to be GC'd. In this case, there is no permanent leak and it
behaves identically to most Java framework objects that hold on to
system files/sockets/pipes/etc.

> --
> Daniel C. Sobral
>
> I travel to the future all the time.
>

dcsobral
Joined: 2009-04-23,
User offline. Last seen 38 weeks 5 days ago.
Re: Re: Example Issue with Scala API Documentation and Scala Co

On Wed, Nov 30, 2011 at 15:27, Scott Carey wrote:
> On Tue, Nov 29, 2011 at 6:04 PM, Daniel Sobral wrote:
>> On Tue, Nov 29, 2011 at 22:52, Alan Burlison wrote:
>>> On 30/11/2011 00:02, Daniel Sobral wrote:
>>>
>>>> But the Source that is used by ProcessIO is *NOT* scala.io.Source. It
>>>> is scala.sys.process.ProcessBuilder.Source -- an altogether different
>>>> beast.
>>>
>>> That just happen to share the same name.  How ... helpful.
>>
>> Tell me about it. :-/
>>
>>>> Besides, Source doesn't "leak" file descriptors -- it just doesn't
>>>> close them automatically.
>>>
>>> It doesn't document that behaviour, so unless you happen to know about it
>>> it's pretty certain you are going to end up with a FD leak.  Sure it's easy
>>> enough to add a close() but it should be in the docs, and it isn't.
>>
>> Yes, sure. And we bloggers are also guilty with our cool one-liners
>> that don't close the Source.
>>
>> I'm just making the distinction of a situation where the user _can't_
>> do anything about it from one where he _should_ be doing something.
>>
>
> I am pointing out that anywhere framework code holds on to system
> resources, it _should_ have a finalize() or equivalent WeakReference
> trigger to allow GC to clean up leaks eventually, and
> non-deterministically, or rely on the underlying Java class to do the
> same.  That way, even with user error, the OS resources are not leaked
> permanently.

JVM is allowed, whatever it may actually do, to delay finalize() until
the program finishes execution, so relying on it to free resources is
a bad practice.

> Most likely, this case is using a low level Java class that already
> handles cleanup that way when users don't call close().  If so I
> suspect a gc call or two would clear the OS resources when the user
> does not manually close the Source if the source was out of scope and
> able to be GC'd.  In this case, there is no permanent leak and it
> behaves identically to most Java framework objects that hold on to
> system files/sockets/pipes/etc.

If I do "ls" #| "cat" !, it will leak a single file descriptor more
than just "ls".!. Put another pipe, one more leak... I'm betting these
leaks are stderr leaks. The point, though, is that 2 other file
descriptors are not leaked. They are not leaked because the code
constructed to handle the pipes close the streams! So it can and does
close streams, but only those directly under its control. Streams that
may go to the user are not closed.

Tony Morris
Joined: 2008-12-19,
User offline. Last seen 30 weeks 4 days ago.
Re: Re: Example Issue with Scala API Documentation and Scala C

.NET uses TSource, TResult and TCollection. This greatly reduces complexity.

On Dec 1, 2011 1:38 AM, "Josh Suereth" <joshua [dot] suereth [at] gmail [dot] com> wrote:
"That" is used an awful lot in the collection library, and I agree that better name (like ResultCollection) could be used.  It also helps understand CanBuildFrom ->
CanBuildFrom[List[A], B, ResultCollection] // OH, right, building from List to some other collection, not sure about B yet...
- Josh

On Wed, Nov 30, 2011 at 10:33 AM, Rex Kerr <ichoran [at] gmail [dot] com> wrote:
Related to the API documentation discussion--Ittay just pointed out a confusing-looking method signature:

def ++ [B >: A, That] (that: TraversableOnce[B])(implicit bf: CanBuildFrom[List[A], B, That]) : That


Although he was focused on a different aspect, I noticed a potential problem with this one. Is there any chance that we can avoid having both a "That" type and a "that" argument of a completely different type?  This is a source change, not a documentation change, but it might make the method signature less impenetrable.

  --Rex

P.S. Sorry--don't have time to make the changes myself, just point out the issue.


Alan Burlison
Joined: 2011-08-26,
User offline. Last seen 42 years 45 weeks ago.
Re: Re: Example Issue with Scala API Documentation and Scala Co

On 30/11/2011 17:27, Scott Carey wrote:

> Most likely, this case is using a low level Java class that already
> handles cleanup that way when users don't call close(). If so I
> suspect a gc call or two would clear the OS resources when the user
> does not manually close the Source if the source was out of scope and
> able to be GC'd. In this case, there is no permanent leak and it
> behaves identically to most Java framework objects that hold on to
> system files/sockets/pipes/etc.

What makes you believe that java.io classes autoclose file descriptors?

scala> var i=0
i: Int = 0
scala> while (i < 100) { new java.io.FileInputStream("/dev/zero"); i = i
+ 1}
scala> System.gc

$ pfiles 13752 | grep zero | wc -l
101
$

Naftoli Gugenheim
Joined: 2008-12-17,
User offline. Last seen 42 years 45 weeks ago.
Re: Re: Example Issue with Scala API Documentation and Scala Co


On Wed, Nov 30, 2011 at 1:34 PM, Daniel Sobral <dcsobral [at] gmail [dot] com> wrote:
> I am pointing out that anywhere framework code holds on to system
> resources, it _should_ have a finalize() or equivalent WeakReference
> trigger to allow GC to clean up leaks eventually, and
> non-deterministically, or rely on the underlying Java class to do the
> same.  That way, even with user error, the OS resources are not leaked
> permanently.

JVM is allowed, whatever it may actually do, to delay finalize() until
the program finishes execution, so relying on it to free resources is
a bad practice.

Of course, but it's still better than nothing. (Of course WeakReference doesn't have that problem.)

Matthew Pocock 3
Joined: 2010-07-30,
User offline. Last seen 42 years 45 weeks ago.
Re: Re: Example Issue with Scala API Documentation and Scala C
HI,

On 30 November 2011 17:26, Daniel Sobral <dcsobral [at] gmail [dot] com> wrote:

It could also be done like this:

List(1, 2, 3).map[B = String](_.toString)

I really like this syntax, both in this case and for tutorial-level code. Compare:
var m: Map[String, Int] var n: Map[Key=String, Value=Int]
I personally don't like prefixing or postfixing names. So, running with the Map case, KeyType, KeyT, TKey are all bad from my point of view. Key is good, and K is acceptable. However, I realise that some others have strong opinions about this.
The scala collections seem to consistently use A for the element type, where as the Java collections uses E. I don't think this matters too much as long as it is instantly available in the scaladocs and in the IDE that "A: the element type". In the collection-building methods, replacing That with Result or CollectionResult or some variation on this theme makes the sigs more readable to me. Also, I agree that you shouldn't use an arg name that coincides with a type name if that arg isn't of that type. It's just rude.
Matthew 


--
Daniel C. Sobral

I travel to the future all the time.



--
Dr Matthew PocockIntegrative Bioinformatics Group, School of Computing Science, Newcastle Universitymailto: turingatemyhamster [at] gmail [dot] com gchat: turingatemyhamster [at] gmail [dot] commsn: matthew_pocock [at] yahoo [dot] co [dot] uk irc.freenode.net: drdozerskype: matthew.pococktel: (0191) 2566550mob: +447535664143
John Nilsson
Joined: 2008-12-20,
User offline. Last seen 42 years 45 weeks ago.
Re: Re: Example Issue with Scala API Documentation and Scala C

How about subscript? Use LaTeX for the names so T_key would render as T with a key-subscript. :)

BR,
John

Den 1 dec 2011 11:01 skrev "Matthew Pocock" <turingatemyhamster [at] gmail [dot] com>:
HI,

On 30 November 2011 17:26, Daniel Sobral <dcsobral [at] gmail [dot] com> wrote:

It could also be done like this:

List(1, 2, 3).map[B = String](_.toString)

I really like this syntax, both in this case and for tutorial-level code. Compare:
var m: Map[String, Int] var n: Map[Key=String, Value=Int]
I personally don't like prefixing or postfixing names. So, running with the Map case, KeyType, KeyT, TKey are all bad from my point of view. Key is good, and K is acceptable. However, I realise that some others have strong opinions about this.
The scala collections seem to consistently use A for the element type, where as the Java collections uses E. I don't think this matters too much as long as it is instantly available in the scaladocs and in the IDE that "A: the element type". In the collection-building methods, replacing That with Result or CollectionResult or some variation on this theme makes the sigs more readable to me. Also, I agree that you shouldn't use an arg name that coincides with a type name if that arg isn't of that type. It's just rude.
Matthew 


--
Daniel C. Sobral

I travel to the future all the time.



--
Dr Matthew PocockIntegrative Bioinformatics Group, School of Computing Science, Newcastle Universitymailto: turingatemyhamster [at] gmail [dot] com gchat: turingatemyhamster [at] gmail [dot] commsn: matthew_pocock [at] yahoo [dot] co [dot] uk irc.freenode.net: drdozerskype: matthew.pococktel: (0191) 2566550mob: +447535664143
nilskp
Joined: 2009-01-30,
User offline. Last seen 1 year 27 weeks ago.
Re: Re: Example Issue with Scala API Documentation and Scala Co
On Thu, Dec 1, 2011 at 12:41 AM, Naftoli Gugenheim <naftoligug [at] gmail [dot] com> wrote:
Of course, but it's still better than nothing. (Of course WeakReference doesn't have that problem.)

One should use PhantomReference instead of finalize() for cleanup of resources.  

Copyright © 2012 École Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland