- About Scala
- Documentation
- Code Examples
- Software
- Scala Developers
problems of scala complexity
I am moving this from scala-user to scala-debate upon request. First, a bit of context (copied):
However, as I've thought more about it, I've realized that what enterprises need is _controllable_ complexity. If you have 2,000 software engineers using Scala (which is what we all hope for, right?), then some of them _will_ use the most complex features because they think it's cool, regardless of corporate guidelines.
Next, to put this into my own personal perspective: Until recently, I dismissed arguments that Scala was too complex on the grounds that "you can always use as simply a more productive Java".
More recently, I have seen arguments that convinced me that a significant problems with Scala was that _the possibility of using it an an unduly complex manner restricted its corporate applicability_. I see this as a highly valid point. (Large) corporations need to expect that engineers are, to some degree, interchangeable. I think even Google must have that approach these days. It should be expected that an engineer can be dumped into another engineers' job and be able, int not too long, to pick up on what is going on.
I don't believe this is possible with "full Scala". For example, look at Tony M. and (in addition) scalaz. How many engineers could pick up on code engineered in that universe? Very, very few.
I know from experience that given enough people, developers who are completely incompetent in a technology will nonetheless use the extreme abilities of that technology. For example, I once spent several nasty months fixing the "code" of a "developer" who knew virtually nothing about C++, but who nonetheless decided she could overload operators at her discretion. Do you have any idea how difficult it is to understand code where "+" means "accumulate into a queue"?
Given, than , I refer back to the (implicit question) of my original post: does it make sense to allow the enterprise to control the degree of complexity of Scala that is used? (I think it does)
I understand that such a topic may be part of the scala language discussions. If so, apologies, I have not seen it.
Ken
However, as I've thought more about it, I've realized that what enterprises need is _controllable_ complexity. If you have 2,000 software engineers using Scala (which is what we all hope for, right?), then some of them _will_ use the most complex features because they think it's cool, regardless of corporate guidelines.
Next, to put this into my own personal perspective: Until recently, I dismissed arguments that Scala was too complex on the grounds that "you can always use as simply a more productive Java".
More recently, I have seen arguments that convinced me that a significant problems with Scala was that _the possibility of using it an an unduly complex manner restricted its corporate applicability_. I see this as a highly valid point. (Large) corporations need to expect that engineers are, to some degree, interchangeable. I think even Google must have that approach these days. It should be expected that an engineer can be dumped into another engineers' job and be able, int not too long, to pick up on what is going on.
I don't believe this is possible with "full Scala". For example, look at Tony M. and (in addition) scalaz. How many engineers could pick up on code engineered in that universe? Very, very few.
I know from experience that given enough people, developers who are completely incompetent in a technology will nonetheless use the extreme abilities of that technology. For example, I once spent several nasty months fixing the "code" of a "developer" who knew virtually nothing about C++, but who nonetheless decided she could overload operators at her discretion. Do you have any idea how difficult it is to understand code where "+" means "accumulate into a queue"?
Given, than , I refer back to the (implicit question) of my original post: does it make sense to allow the enterprise to control the degree of complexity of Scala that is used? (I think it does)
I understand that such a topic may be part of the scala language discussions. If so, apologies, I have not seen it.
Ken
Re: problems of scala complexity
On Fri, Nov 18, 2011 at 21:56, Kenneth McDonald wrote:
> I know from experience that given enough people, developers who are
> completely incompetent in a technology will nonetheless use the extreme
> abilities of that technology. For example, I once spent several nasty months
> fixing the "code" of a "developer" who knew virtually nothing about C++, but
> who nonetheless decided she could overload operators at her discretion. Do
> you have any idea how difficult it is to understand code where "+" means
> "accumulate into a queue"?
I assume the code was something like: "q + e". Do you think it would
be easier to understand the code if it was "q.plus(e)" or "q.add(e)"?
Re: problems of scala complexity
On Fri, Nov 18, 2011 at 6:56 PM, Kenneth McDonald <ykkenmcd [at] gmail [dot] com> wrote:
This mindset has led to the greatest atrocities in engineering management. No two engineers are the same. We all have different capabilities, and the features we choose to work on in a code base reflects that. Making sure engineers can maintain code is *different* than treating them like replacable cogs.
I completely disagree with the assumption that FP style code can't be engineered by many. Scalaz does itself a disservice in two ways:
(1) Non-existent documentation. This is *just as bad* as that old legacy codebase we're all forced to maintain with 1k's of lines of non-documented junk code. (2) Re-inventing itself every revision. Scalaz is tough because it changes so frequently. Luckily 6.x series has been maintained for a while and stable-ish.
Assuming the Scalaz represents functional programming in scala on the whole is bogus.
Treating engineers as replacable cogs is downright insulting.
Let's look at the real problem: You had an engineer who was committing poor code into the codebase with no supervision. I think we've all run into this problem *regardless of the language*. I ran into this problem more often on Java teams and less often with Scala teams. Poor process and hiring does not necessitate a problem of language, but of the company doing the hiring, or its development process. Good engineers will be able to hire other good engineers. There's a lot of smart people out there, that if recongized by a company can ensure a good, smooth development process. I don't think I've ever walked into a shop without at least one guy who knew what's up. Some may not have enough of the boss's ear to turn around the process, but this is a *human* problem.
Right, if your company requires this to ensure correct code, perhaps a plugin. However, given what I've seen in Java, I think some folks can destroy a codebase even without implicits. The fundamental problem still remains: Someone is given no oversight and commit rights to code.
This is a great place to re-hash discussions, especially if you feel the environment has changed, or you have no content to add. Don't hesitate to bring things up. Discussions like these are very healthy for the community, assuming it doesn't devolve.
Re: problems of scala complexity
On Nov 20, 2011 4:44 AM, "Josh Suereth" <joshua [dot] suereth [at] gmail [dot] com> wrote:
>
>
>
> On Fri, Nov 18, 2011 at 6:56 PM, Kenneth McDonald <ykkenmcd [at] gmail [dot] com> wrote:
>>
>> I am moving this from scala-user to scala-debate upon request. First, a bit of context (copied):
>>
>> However, as I've thought more about it, I've realized that what enterprises need is _controllable_ complexity. If you have 2,000 software engineers using Scala (which is what we all hope for, right?), then some of them _will_ use the most complex features because they think it's cool, regardless of corporate guidelines.
>>
>> Next, to put this into my own personal perspective: Until recently, I dismissed arguments that Scala was too complex on the grounds that "you can always use as simply a more productive Java".
>>
>> More recently, I have seen arguments that convinced me that a significant problems with Scala was that _the possibility of using it an an unduly complex manner restricted its corporate applicability_. I see this as a highly valid point. (Large) corporations need to expect that engineers are, to some degree, interchangeable. I think even Google must have that approach these days. It should be expected that an engineer can be dumped into another engineers' job and be able, int not too long, to pick up on what is going on.
>>
> This mindset has led to the greatest atrocities in engineering management. No two engineers are the same. We all have different capabilities, and the features we choose to work on in a code base reflects that. Making sure engineers can maintain code is *different* than treating them like replacable cogs.
>
>
>>
>> I don't believe this is possible with "full Scala". For example, look at Tony M. and (in addition) scalaz. How many engineers could pick up on code engineered in that universe? Very, very few.
>
>
> I completely disagree with the assumption that FP style code can't be engineered by many. Scalaz does itself a disservice in two ways:
>
> (1) Non-existent documentation. This is *just as bad* as that old legacy codebase we're all forced to maintain with 1k's of lines of non-documented junk code
I truly find it amazing that this is repeated so readily given the mountains of documentation out there. It is as if we could choke the world with even more documentation, just so we could increase the occurrence of repetition of this statement.
Aside from the fact that scalaz makes much more rigorous *machine checked* type system guarantees about the behaviour of functions than a typical scala program/library/user does, and aside from the mountain of papers out there documenting exactly the concepts implemented, and aside from the runnable usage examples, and aside from the articles written by people who switched off their blinkers, learned the material and wrote about it, and aside from machine run specifications, then what documentation are you alluding to?
There doesn't exist a scala library out there with more than 10% of the documentation of scalaz. This fact coupled with the ferver with which the contrary is so readily protested fascinates me endlessly.
It's simply cognitive blindness; I can think of no better hypothesis matching observations in reality. Where do you think the many people who overcome this illusory hurdle got their learning material from? That question is not rhetorical; think about the answer carefully.
> (2) Re-inventing itself every revision. Scalaz is tough because it changes so frequently. Luckily 6.x series has been maintained for a while and stable-ish.
It didn't reinvent itself. It is quite stable. You might be referring to the need to adjust to changes and contentions in scala itself. Scalaz aims to be *useful* so if scala decides to change its type inference slightly, or perhaps order of initialisation, scalaz must adjust accordingly in order to maintain the thesis. This is aside from all the contentions that arise from inherent properties of scala. Many have tried and fizzled attempting to resolve all of these contentions. Nobody likes shaving yaks, but it must be done as a consequence sometimes, *including the occasions when others choose to deny the existence of these consequences*.
>
> Assuming the Scalaz represents functional programming in scala on the whole is bogus.
It doesn't at all, but if you are functional programming, you're going to reinvent a good deal of scalaz. Denialists have very few tactics around this fact, which is itself refreshing.
>
> Treating engineers as replacable cogs is downright insulting.
>>
>>
>> I know from experience that given enough people, developers who are completely incompetent in a technology will nonetheless use the extreme abilities of that technology. For example, I once spent several nasty months fixing the "code" of a "developer" who knew virtually nothing about C++, but who nonetheless decided she could overload operators at her discretion. Do you have any idea how difficult it is to understand code where "+" means "accumulate into a queue"?
>>
>
> Let's look at the real problem: You had an engineer who was committing poor code into the codebase with no supervision. I think we've all run into this problem *regardless of the language*. I ran into this problem more often on Java teams and less often with Scala teams. Poor process and hiring does not necessitate a problem of language, but of the company doing the hiring, or its development process. Good engineers will be able to hire other good engineers. There's a lot of smart people out there, that if recongized by a company can ensure a good, smooth development process. I don't think I've ever walked into a shop without at least one guy who knew what's up. Some may not have enough of the boss's ear to turn around the process, but this is a *human* problem.
>
>>
>> Given, than , I refer back to the (implicit question) of my original post: does it make sense to allow the enterprise to control the degree of complexity of Scala that is used? (I think it does)
>
>
> Right, if your company requires this to ensure correct code, perhaps a plugin. However, given what I've seen in Java, I think some folks can destroy a codebase even without implicits. The fundamental problem still remains: Someone is given no oversight and commit rights to code.
>
>>
>>
>> I understand that such a topic may be part of the scala language discussions. If so, apologies, I have not seen it.
>>
>> Ken
>
>
> This is a great place to re-hash discussions, especially if you feel the environment has changed, or you have no content to add. Don't hesitate to bring things up. Discussions like these are very healthy for the community, assuming it doesn't devolve.
>
>
>
Re: problems of scala complexity
On Sat, Nov 19, 2011 at 18:42, Tony Morris wrote:
>
>> (1) Non-existent documentation. This is *just as bad* as that old
>> legacy codebase we're all forced to maintain with 1k's of lines of
>> non-documented junk code
>
> I truly find it amazing that this is repeated so readily given the mountains
> of documentation out there. It is as if we could choke the world with even
> more documentation, just so we could increase the occurrence of repetition
> of this statement.
Or it could be that the documentation is not in the correct place.
> Aside from the fact that scalaz makes much more rigorous *machine checked*
> type system guarantees about the behaviour of functions than a typical scala
> program/library/user does, and aside from the mountain of papers out there
Yeah, well, most people expect documentation to be in English, not in
type declarations. In fact, most people consider type declarations to
be "source code", and define "documentation" to be "not source code".
Disagree all you want with this perspective, but don't come
complaining when people act like people do.
> documenting exactly the concepts implemented, and aside from the runnable
> usage examples, and aside from the articles written by people who switched
Runnable examples? Mmmmm... that sounds suspiciously like "source
code". See above.
> off their blinkers, learned the material and wrote about it, and aside from
Blogs exist where documentation fails.
> machine run specifications, then what documentation are you alluding to?
Well, open http://scalaz.github.com/scalaz/scalaz-2.9.1-6.0.2/doc/index.html#scalaz....
See those 16 out of 69 methods that have some text in English below
it? Well, that's one thing people call "documentation", and I'll let
the numbers speak for themselves.
Another example of things people call "documentation" is this:
http://download.oracle.com/javase/6/docs/technotes/guides/collections/ov....
You might think this is bad for some reason, for all I know. If that's
the case, then just substitute "something bad" for "documentation"
whenever people talk about it. But _don't_ substitute "source code"
for "documentation".
Re: problems of scala complexity
On Nov 20, 2011 7:49 AM, "Daniel Sobral" <dcsobral [at] gmail [dot] com> wrote:
>
> On Sat, Nov 19, 2011 at 18:42, Tony Morris <tmorris [at] tmorris [dot] net> wrote:
> >
> >> (1) Non-existent documentation. This is *just as bad* as that old
> >> legacy codebase we're all forced to maintain with 1k's of lines of
> >> non-documented junk code
> >
> > I truly find it amazing that this is repeated so readily given the mountains
> > of documentation out there. It is as if we could choke the world with even
> > more documentation, just so we could increase the occurrence of repetition
> > of this statement.
>
> Or it could be that the documentation is not in the correct place.
It could be; but I think there are more likely explanations.
>
> > Aside from the fact that scalaz makes much more rigorous *machine checked*
> > type system guarantees about the behaviour of functions than a typical scala
> > program/library/user does, and aside from the mountain of papers out there
>
> Yeah, well, most people expect documentation to be in English, not in
> type declarations. In fact, most people consider type declarations to
> be "source code", and define "documentation" to be "not source code".
No they don't. You have a selection bias. So do I.
Watch this. Most people are absolutely adamant that your statement is wrong. That's because it is. I was able to make this claim by restricting myself to people who understand the subject matter at hand. Of course they see your statement is wrong. To most people, it is bleedingly obvious!
More seriously, I agree there are a lot of people who believe your statement. They are wrong. What now? Are you offering to join the agenda of educating those who seek it out? If not what? There are many people who have overcome this very exact hurdle; it's tough work though.
> Disagree all you want with this perspective, but don't come
> complaining when people act like people do.
I'm well aware of the popular views. There is an enormous amount of documentation in English out there; to be, clear, a FUCKING OVERWHELMING amount that I doubt I will get through in my life time.
>
> > documenting exactly the concepts implemented, and aside from the runnable
> > usage examples, and aside from the articles written by people who switched
>
> Runnable examples? Mmmmm... that sounds suspiciously like "source
> code". See above.
Yes, runnable examples are extremely useful by virtue of their being machine checked.
>
> > off their blinkers, learned the material and wrote about it, and aside from
>
> Blogs exist where documentation fails.
Bollocks. Blogs exist because people learn and want to share it with others who do more than pay lip service to the goal of learning.
>
> > machine run specifications, then what documentation are you alluding to?
>
> Well, open http://scalaz.github.com/scalaz/scalaz-2.9.1-6.0.2/doc/index.html#scalaz.Identity.
> See those 16 out of 69 methods that have some text in English below
> it? Well, that's one thing people call "documentation", and I'll let
> the numbers speak for themselves.
What if many people are wrong? It's not as if there aren't countless examples of many people sharing the same mistake, surviving only by patting each other on the back, confirming the perpetuity of the mistake.
>
> Another example of things people call "documentation" is this:
> http://download.oracle.com/javase/6/docs/technotes/guides/collections/overview.html.
> You might think this is bad for some reason, for all I know. If that's
> the case, then just substitute "something bad" for "documentation"
> whenever people talk about it. But _don't_ substitute "source code"
> for "documentation".
>
Its not bad; its just clumsy and error-prone, and subsequently inefficient. There are far better ways of achieving comprehension of a subject using oh I don't know, maybe COMMUNICATION SKILLS or something. Oh wait I forgot; I don't have those; I was born, with this understanding and everyone else is excluded from sharing it. Cheap stab sorry, but a bit of introspection is desperately in order around here.
> --
> Daniel C. Sobral
>
> I travel to the future all the time.
Re: problems of scala complexity
Hi Tony,
Recently while trying to answer this: https://twitter.com/#!/dibblego/status/130796120107532289
I looked at the code for MA(http://scalaz.github.com/scalaz/
scalaz-2.9.1-6.0.2/doc.sxr/scalaz/MA.scala.html)
I found that there is foldMap and foldMapDefault. The signature
difference between the two is the use of Foldable versus Traverse.
I've indirectly used Foldable and Traverse before, though I'm no
expert, I get a sense of what they can do.
Some testing showed that they yield the same result (on this
particular instance).
scala> val list = List(101,102, 103)
list: List[Int] = List(101, 102, 103)
scala> list.foldMap(i => (i, 1))
res1: (Int, Int) = (306,3)
scala> list.foldMapDefault(i => (i, 1))
res2: (Int, Int) = (306,3)
Why use one versus the other one? The signature doesn't tell me that
when it works on either case. It looks like most scala collections can
be lifted/converted into Traverse and Foldable. I then took a detour
and searched for foldMap and foldMapDefault and haskell. But that
didn't answer my question.
So I had to dig into what the code does, but for foldMapDefault it
wasn't obvious because I don't know what Const does (an enigmatic:
"Used to generate Phantom Applicative Functors and categories from a
Monoidal type A". Ok in restrospect it sorts of make sense but it's
hard, because Const is not informative as a name). Eventually I got a
sense of Const from http://scalaz.github.com/scalaz/scalaz-2.9.1-6.0.2/doc.sxr/scalaz/exampl...
But I feel that a bit of documentation like "use foldMap when abc over
foldMapDefault" or vice versa would be helpful and save time. Also
known issues like this could be documented as well:
scala> (1 to 10000).toList.foldMap(i => (i, 1))
res4: (Int, Int) = (50005000,10000)
scala> (1 to 10000).toList.foldMapDefault(i => (i, 1))
[stack overflows]
Overall, learning to use scalaz is challenging yet rewarding. But I
think that lack of documentation makes it harder than necessary.
--Jean-Laurent
On Nov 19, 6:02 pm, Tony Morris wrote:
> On Nov 20, 2011 7:49 AM, "Daniel Sobral" wrote:
>
[snip]
> > Yeah, well, most people expect documentation to be in English, not in
> > type declarations. In fact, most people consider type declarations to
> > be "source code", and define "documentation" to be "not source code".
>
> No they don't. You have a selection bias. So do I.
>
> Watch this. Most people are absolutely adamant that your statement is
> wrong. That's because it is. I was able to make this claim by restricting
> myself to people who understand the subject matter at hand. Of course they
> see your statement is wrong. To most people, it is bleedingly obvious!
>
> More seriously, I agree there are a lot of people who believe your
> statement. They are wrong. What now? Are you offering to join the agenda of
> educating those who seek it out? If not what? There are many people who
> have overcome this very exact hurdle; it's tough work though.
[snip]
Re: problems of scala complexity
On Sat, Nov 19, 2011 at 7:44 PM, Josh Suereth <joshua [dot] suereth [at] gmail [dot] com> wrote:
I agree with the gist of everything you have written here, Josh, but I assume you meant "or if you have _new_ content to add".
Cheers
-- Martin
Re: problems of scala complexity
Weaker programmers will always find a way to hang themselves.
Nerfing Scala is not going to fix that.
Better to nerf the weak programmers than the language.
Re: problems of scala complexity
believe language design should be collaborative effort of linguists
and computer scientists.... because in the end it depends who will be
the end users of the language and for what purposes it will be
used ...a language to build parsers and/or compilers or for machine
consumption or to write domains specific logic that humans
(programmers/business analyst) can understand ...of course with Scala
you can do both ...however in my opinion, as Russ gave an example, it
is better that you wrap Scala's symbolic soup with more
understandable and "meaningful yet succinct" DSL, if you will. And I
don't think that Scala community should think of arguments against
complexity as negative ...neither should Scala community look down
upon "average Java" developers ....but understand that nothing that we
(humans) create is perfect ...the same people who now say how terrible
Java is, happily worked with Java, built systems and made their
living. I can safely say, after few years there will be cries against
Scala also ...and there will be interfaces/wrappers bulit around Scala
that would make it appeal to wider user base of "Average Java
Developers" ...and perhaps another language with all the power of
Scala, yet without "perceived" complexity (by some) of Scala.
cheers
Arif
On Nov 20, 12:04 am, Adam Jorgensen
wrote:
> Ugh, all this debate about complexity.
>
> Weaker programmers will always find a way to hang themselves.
>
> Nerfing Scala is not going to fix that.
>
> Better to nerf the weak programmers than the language.
Re: problems of scala complexity
I don't think we should be having a 'debate about complexity' - rather
we should be having a debate about 'what to do about complexity' (even
if it is only perceived as complex by some people).
...and (some) strong programmers will always find a way to give other
programmers more rope and cause to hang themselves.
We don't need to 'nerf' Scala (or anyone), we just need to give people
less experienced with Scala more tools to better deal with the Scala we
have now.
Cheers, Eric
On 2011-11-19 9:04 PM, Adam Jorgensen wrote:
> Ugh, all this debate about complexity.
>
> Weaker programmers will always find a way to hang themselves.
>
> Nerfing Scala is not going to fix that.
>
> Better to nerf the weak programmers than the language.
Re: problems of scala complexity
On Fri, Nov 18, 2011 at 6:56 PM, Kenneth McDonald <ykkenmcd [at] gmail [dot] com> wrote:
But this is hardly limited to Scala. You can use Java in ways that can be much more difficult to understand than most Scala code. Also, complex is not the right word -- watch Rich Hickey's presentation on complexity, it is awesome.
Using the visitor pattern (/double dispatch) in Java may confuse an engineer who has never seen the concept. However, the visitor pattern can actually _simplify_ a lot of Java code, where you need to reuse a data structure in many ways.
Now, you want to see complex? Look at most large Java frameworks used in large enterprises. There is so much magic going on that trying to debug a simple problem can lead to days lost as you try to untwine all the various pieces and figure out how they interact. Some of these things have these incredibly complicated testing frameworks to support them, as everything is so intertwined that even testing a small unit of code is a problem unto itself.
I first started using Scala in one project I had previously written in Java. Using concepts like implicits, traits, "enrich" my library pattern (;)), etc. I was able to greatly simplify my library. That is, these concepts let me both simplify my code and also produce a library that was simpler.
Our entire modern society is built on specialization. You want to be able to replace somebody if they're hit by a bus, but software developers do not come in a one-size-fits-all model. Now, clearly your problem has a lot of caveats. Regardless of the organization, you wouldn't expect a newly hired tester to be put in a senior developers position and just figure things out himself. You replace competent developers with competent developers. Can a competent developer learn Scala? Of course. This has already been proven, as shown by Scala's popularity.
I'm a firm believer that people will raise to your expectations of them. If you make it clear that you expect them to be replaceable cogs that should write software that can be maintained by the cheapest developer they can find, since it is expected you'll either be fired, outsourced or quit in a short amount of time, then, yeah, you'll end up with crap. Crap people and crap code. Perhaps the real solution is that enterprises just need to raise their expectations?
How many engineers could pick up/maintain code in most operating system or even the collection libraries in Java, C#, C++, Scala, etc. Does this stop people from using them effectively? No. Scalaz is a tool for software developers. You do not need to be able to create a tool from scratch to use it effectively.
Code reviews were created to solve precisely this problem. Nothing about Scala interferes with this. However, these rules can also "backfire" when you don't trust your developers. One friend told me a story where his code was rejected because it used a linked list, rather than an ArrayList. Of course, he was using a linked list for the O(1) insert/delete while traversing the list. The standards were so strict/arbitrarily applied that they'd rather have an O(n^2) algorithm that used ArrayList than a O(n) one that used a linked list.
No, I do not think it does. It makes sense to let enterprises do what they always did; code reviews. Code reviews are great because they can be tailored for each dept. and individual. If a group needs to write performant code or an easy-to-use, not-so-easy-to-implement library, they shouldn't be gimped because someone higher up is worried about whether he'll be able to ship the dept off to some 3rd world country.
Code reviews ensure new developers aren't making sill mistakes (like overloading + in an unexpected way), while also letting senior developers create complex, but necessary, software that is still understandable by other senior developers, even if not by a newbie.
Re: problems of scala complexity
[1] https://github.com/functionaljava/functionaljava/blob/master/core/src/main/java/fj/data/Iteratee.java
[2] https://github.com/scalaz/scalaz/blob/master/core/src/main/scala/scalaz/Iteratee.scala
On 19 November 2011 00:56, Kenneth McDonald <ykkenmcd [at] gmail [dot] com> wrote:
Re: problems of scala complexity
First, the thread and many like it seems too negatve to me. Scala is used in a large and growing number of enterprises, some of them with more than 100 devs working with it. These people just get on with the job (and love it for the most part); they don't find Scala's ``complexity'' too daunting.
Second, I believe we have come to realize that complexity and power are pretty much aligned for Scala.
Take away the complexity and you take away the power. Here I mean power in the sense: "Number of concepts that you can describe precisely using your language". I want to get away from the common argument that all Turing complete languages are equivalent in power anyway.
Third, if you look at other languages, there is always a set of "best practices" developed that come with it and that are used as guidelines. Certainly Java has a rich culture of best practices. And that cultures changes over time, as developments in Java EE show.
Since Scala's enterprise use is still young, best practices have still a some way to develop and stabilize.
This is your chance! If you put out some clean libraries with easy-to use APIs you have a great chance to influence the future best practices for Scala, as long as they are still in flux. I wish we'd concentrate more on these aspects than wringing our hands.
Can best practices be influenced by tools, such as style checkers? Maybe. It's certainly something we want to pursue. But it would be silly to delegate responsibility for forming best practices to these tools only.
[Cross-posted to scala-user, because it relates to quite a few threads there as well]
-- Martin
Re: problems of scala complexity
First, the thread and many like it seems too negatve to me. Scala is used in a large and growing number of enterprises, some of them with more than 100 devs working with it. These people just get on with the job (and love it for the most part); they don't find Scala's ``complexity'' too daunting.
Second, I believe we have come to realize that complexity and power are pretty much aligned for Scala.
Take away the complexity and you take away the power. Here I mean power in the sense: "Number of concepts that you can describe precisely using your language". I want to get away from the common argument that all Turing complete languages are equivalent in power anyway.
Third, if you look at other languages, there is always a set of "best practices" developed that come with it and that are used as guidelines. Certainly Java has a rich culture of best practices. And that cultures changes over time, as developments in Java EE show.
Since Scala's enterprise use is still young, best practices have still a some way to develop and stabilize.
This is your chance! If you put out some clean libraries with easy-to use APIs you have a great chance to influence the future best practices for Scala, as long as they are still in flux. I wish we'd concentrate more on these aspects than wringing our hands.
Can best practices be influenced by tools, such as style checkers? Maybe. It's certainly something we want to pursue. But it would be silly to delegate responsibility for forming best practices to these tools only.
[Cross-posted to scala-user, because it relates to quite a few threads there as well]
-- Martin
Re: problems of scala complexity
In Soviet Russia:
The population - Factories build Ladas of questionable quality. Food
production facilities - concoct meat paste, sold anywhere else as dog
food. Infrastructure is crumbling. Afghanistan is a disaster.
The Nomenclatura - We are matching the United States military
capacity, especially in the field of high technology. Any who
criticise us are traitors to the people's revolution. We are the land
of plenty. There will be a socialist victory in the Peoples Republic
of Afghanistan.
In Scala:
Users - Too much complexity. Too many versions. Too many funky bugs.
Why do we have to learn yet another build tool? When will releases
stabilise? Java runtime is backward compatible to 1996 - Scala, 6
months at best. Too academic.
The Elite - Fools, you are stupid, code in Visual Basic. The product
is primed for commercial success. All the contractors love Scala (they
know everything). All the graduates love Scala (they are the future).
Everyone at EPFL loves it (they are really smart and untainted by the
drudgery of manual labour and deadlines).
Re: Re: problems of scala complexity
:-)
On 2011-11-21 2:04 AM, bryan hunt wrote:
> In Soviet Russia:
>
> The population - Factories build Ladas of questionable quality. Food
> production facilities - concoct meat paste, sold anywhere else as dog
> food. Infrastructure is crumbling. Afghanistan is a disaster.
>
> The Nomenclatura - We are matching the United States military
> capacity, especially in the field of high technology. Any who
> criticise us are traitors to the people's revolution. We are the land
> of plenty. There will be a socialist victory in the Peoples Republic
> of Afghanistan.
>
> In Scala:
>
> Users - Too much complexity. Too many versions. Too many funky bugs.
> Why do we have to learn yet another build tool? When will releases
> stabilise? Java runtime is backward compatible to 1996 - Scala, 6
> months at best. Too academic.
>
> The Elite - Fools, you are stupid, code in Visual Basic. The product
> is primed for commercial success. All the contractors love Scala (they
> know everything). All the graduates love Scala (they are the future).
> Everyone at EPFL loves it (they are really smart and untainted by the
> drudgery of manual labour and deadlines).
Re: Re: problems of scala complexity
On 2011, Nov 21, at 11:04 AM, bryan hunt wrote:
> Users - Too much complexity. Too many versions. Too many funky bugs.
> Why do we have to learn yet another build tool? When will releases
> stabilise? Java runtime is backward compatible to 1996 - Scala, 6
> months at best. Too academic.
As a user of scala, I have to say that I mostly do not agree with this. I am not a contractor, a student, or an EPFL employee. I have however used Java the last ten years or so. As a manager of an "enterprise" (whatever that means) scala team, I can tell you that it is fully possible to build a working team. You _do_ need team members who are not afraid to learn to pull it off.
I am perfectly willing to accept that not everyone is able to pull off this feat.
(And yes, I am unhappy about the versioning shenanigans of the past. I am told that is in the past, and further improvements and a hot beverage will mollify me)
Geir
Re: Re: problems of scala complexity
On Mon, 21 Nov 2011 11:23:55 +0100, Geir Hedemark wrote:
> On 2011, Nov 21, at 11:04 AM, bryan hunt wrote:
> > Users - Too much complexity. Too many versions. Too many funky bugs.
> > Why do we have to learn yet another build tool? When will releases
> > stabilise? Java runtime is backward compatible to 1996 - Scala, 6
> > months at best. Too academic.
>
> As a user of scala, I have to say that I mostly do not agree with this. I am not a contractor, a student, or an EPFL employee. I have however used Java the last ten years or so. As a manager of an "enterprise" (whatever that means) scala team, I can tell you that it is fully possible to build a working team. You _do_ need team members who are not afraid to learn to pull it off.
>
> I am perfectly willing to accept that not everyone is able to pull off this feat.
>
> (And yes, I am unhappy about the versioning shenanigans of the past. I am told that is in the past, and further improvements and a hot beverage will mollify me)
>
> Geir
I built a cometd, multi-user, agent based, real-time, crisis management
system using 6000 Scala LOC.
It probably would have eaten up about 20,000 Java LOC.
It would have been half as difficult - had the runtime been stable, Java
interop worked as advertised, and wierd gotchas not bit me in the ass
half way through the development cycle.
Bryan Hunt
Re: Re: problems of scala complexity
On 2011, Nov 21, at 11:48 AM, Bryan Hunt wrote:
> It would have been half as difficult - had the runtime been stable, Java
> interop worked as advertised, and wierd gotchas not bit me in the ass
> half way through the development cycle.
I remember similar gripes with java 1.1. Not to mention Ant.
Geir
Re: Re: problems of scala complexity
cheers
Arif
On Mon, Nov 21, 2011 at 6:06 AM, Geir Hedemark <geir [dot] hedemark [at] gmail [dot] com> wrote:
Re: Re: problems of scala complexity
Guys, scala-debate, please, NOT scala-user.
Re: problems of scala complexity
On Saturday, November 19, 2011 8:31:19 AM UTC-6, martin odersky wrote: I had started my threads after seeing several comments from enterprise origins concerned about the complexity of Scala, and almost simultaneously encountering some of these problems myself. But this is all anecdotal, admittedly. Is there any way of polling enterprise sites to find out what their concerns _really_ are? Turing equivalency was always a Straw Man. However, some of the power of Scala does end requiring some quite sophisticated mechanics--for example, "CanBuildFrom" and implicit params. "implicit" isn't difficult to understand, but its implications are, and it doesn't take much of a need to go beyond basic use of collections to run into those implications. Because Java is _so much_ less expressive than Scala, its best practices will have been much easier to formulate, I believe. I will echo a commenter I read ahead to--APIs are the smallest and easiest part. I believe my "rex" api (https://github.com/KenMcDonald/rex) for dealing with regular expressions in a higher-level manner is pretty clean--but that was easy because I really didn't exercise Scala features. The real challenge lies in such things as: 1) How to make complex type sigs readable. (I say make type params readable words, Tony M. says opposite. Obviously I'm right :-), but seriously, what is "best practice" here?) 2) When/how to use implicit params? (I don't even have a clue) 3) When to use highly abstract functionals, vs. more verbose but more concrete "lower-level" calls.
I could probably think of several more _major_ issues. The points are, these don't even exist in the Java universe, and they are not easy to answer. Developing "best practices" for Scala is going to be _much_ harder than for Java.
Cheers,Ken
Re: problems of scala complexity
On Saturday, November 19, 2011 8:31:19 AM UTC-6, martin odersky wrote: I had started my threads after seeing several comments from enterprise origins concerned about the complexity of Scala, and almost simultaneously encountering some of these problems myself. But this is all anecdotal, admittedly. Is there any way of polling enterprise sites to find out what their concerns _really_ are? Turing equivalency was always a Straw Man. However, some of the power of Scala does end requiring some quite sophisticated mechanics--for example, "CanBuildFrom" and implicit params. "implicit" isn't difficult to understand, but its implications are, and it doesn't take much of a need to go beyond basic use of collections to run into those implications. Because Java is _so much_ less expressive than Scala, its best practices will have been much easier to formulate, I believe. I will echo a commenter I read ahead to--APIs are the smallest and easiest part. I believe my "rex" api (https://github.com/KenMcDonald/rex) for dealing with regular expressions in a higher-level manner is pretty clean--but that was easy because I really didn't exercise Scala features. The real challenge lies in such things as: 1) How to make complex type sigs readable. (I say make type params readable words, Tony M. says opposite. Obviously I'm right :-), but seriously, what is "best practice" here?) 2) When/how to use implicit params? (I don't even have a clue) 3) When to use highly abstract functionals, vs. more verbose but more concrete "lower-level" calls.
I could probably think of several more _major_ issues. The points are, these don't even exist in the Java universe, and they are not easy to answer. Developing "best practices" for Scala is going to be _much_ harder than for Java.
Cheers,Ken
Re: problems of scala complexity
I've been using Scala for about 2 years (10 years of Java before that), and i've decided to post here just because of how absolutely precise (in my opinion) Martin's points are.- if you just do your job with Scala instead of looking for reasons why it is complex or something like that, it just starts to work in your hands and you like it more and more- the 'complexity' feeling is mostly the feeling of freedom from (or absence of) 'best practices'- the only thing we need to bust the 'complexity' myth is good libraries/frameworks
Re: problems of scala complexity
I've been using Scala for about 2 years (10 years of Java before that), and i've decided to post here just because of how absolutely precise (in my opinion) Martin's points are.- if you just do your job with Scala instead of looking for reasons why it is complex or something like that, it just starts to work in your hands and you like it more and more- the 'complexity' feeling is mostly the feeling of freedom from (or absence of) 'best practices'- the only thing we need to bust the 'complexity' myth is good libraries/frameworks
Re: Re: problems of scala complexity
Thanks,
-Vlad
On Sun, Nov 20, 2011 at 5:06 AM, Oleg Galako <ojowoo [at] gmail [dot] com> wrote:
Re: Re: problems of scala complexity
- Akka has culled its symbols down to essentially just ! and ?- scalaz is near the end of a rewrite that will make it far easier to use non-symbolic equivalents for many concepts
Specs2 uses more symbols that specs1, but they're tightly contained and well documented, so this isn't a great concern. I'm not so convinced about the mixed identifiers (e.g. be_==), but it does offer alternative notations and I'm generally comfortable allowing slightly different rules to apply in a test framework.
Lift seems to be using ever more symbols and mixed identifiers (e.g. #> and unsafe_!), but the range of web frameworks now available for Scala means that developers have alternatives to chose from if they disagree with Lift's decisions.
Databinder Dispatch is one of the few libraries remaining that is still symbol-heavy, not moving towards a design with at least the *option* of fewer symbols, and doesn't really have a viable alternative (for example, nothing else I know of can offer OAuth support out of the box). The saving grace of dispatch is that it uses a convention which is internally very consistent. Usage of Dispatch is also likely to be constrained to just a small corner of any codebase.
On 20 November 2011 13:06, Oleg Galako <ojowoo [at] gmail [dot] com> wrote:
Re: Re: problems of scala complexity
- Akka has culled its symbols down to essentially just ! and ?- scalaz is near the end of a rewrite that will make it far easier to use non-symbolic equivalents for many concepts
Specs2 uses more symbols that specs1, but they're tightly contained and well documented, so this isn't a great concern. I'm not so convinced about the mixed identifiers (e.g. be_==), but it does offer alternative notations and I'm generally comfortable allowing slightly different rules to apply in a test framework.
Lift seems to be using ever more symbols and mixed identifiers (e.g. #> and unsafe_!), but the range of web frameworks now available for Scala means that developers have alternatives to chose from if they disagree with Lift's decisions.
Databinder Dispatch is one of the few libraries remaining that is still symbol-heavy, not moving towards a design with at least the *option* of fewer symbols, and doesn't really have a viable alternative (for example, nothing else I know of can offer OAuth support out of the box). The saving grace of dispatch is that it uses a convention which is internally very consistent. Usage of Dispatch is also likely to be constrained to just a small corner of any codebase.
On 20 November 2011 13:06, Oleg Galako <ojowoo [at] gmail [dot] com> wrote:
Re: problems of scala complexity
Hi Kevin,
I just noticed this thread.
On Nov 20, 5:55 am, Kevin Wright wrote:
> Oh absolutely! It also doesn't help that many of our current "high
> profile" libraries are particularly heavy on symbolic method names. Things
> are improving in at least two notable libraries though:
>
> - Akka has culled its symbols down to essentially just ! and ?
> - scalaz is near the end of a rewrite that will make it far easier to use
> non-symbolic equivalents for many concepts
>
Hey, you forgot ScalaTest, which I would point out was designed with
the philosophy of: *avoid any operators that users are not already
experts in.* So ScalaTest originally just had ===, which means
equality, < for less than, > for greater than, <= for less than or
equals, and >= for greater than or equals. At some point I did add one
non-obvious operator, the - in FreeSpec, because after much searching
for a way to avoid it, it was the best solution. In 2.1, which is
coming next year, I'll be adding a few more operators. One is +/- as
an alias for plusOrMinus. This one has been requested by multiple
users over time, and I've done several informal surveys asking people
if they can guess its meaning, and people always have. So I decided to
add that one. I'm also adding &&, ||, and ! for boolean-like
expressions with the triple equals operator, and they will mean
exactly what they look like they mean. Lots of new features are
coming, but no other operators than these few that pass the "obvious
test."
I have been bummed that so many non-obvious operators have been
showing up in Scala APIs. The trouble is that although obvious
operators makes code easier to read (a + b is nicer than a add b), non-
obvious operators makes code harder for casual users to understand,
because they have to look up what all the meanings are. And a lot of
users are casual, in that they don't use your library often enough to
remember the symbols for whatever reason. And to Java programmers, at
least, it makes Scala code look cryptic.
Moreover I would claim that 99% of the time, if you work at it, you
can find a better alternative to a non-obvious operator. Compare Specs
tables to ScalaTest tables. Specs uses lots of operators:
http://etorreborre.github.com/specs2/guide/org.specs2.guide.Matchers.htm...
ScalaTest doesn't use any:
http://www.scalatest.org/user_guide/table_driven_property_checks#testing...
You just don't need non-obvious operators most of the time. There's a
better way if you look for it.
The feeling I get sometimes from all this is that we're a bunch of
really smart, passionate Scala experts designing libraries for each
other, rather than for the mainstream. Because we can certainly learn
and understand each others cryptic operators, and maybe even enjoy the
conciseness of the expressions we can make out of them. But the
mainstream developer is not that interested. It's not to say they
aren't smart. They may be super-smart, but just passionate about
designing airplane wings or getting their startup off the ground or
something else rather than becoming experts in our libraries. They're
just not that into us.
Bill
> Specs2 uses more symbols that specs1, but they're tightly contained and
> well documented, so this isn't a great concern. I'm not so convinced about
> the mixed identifiers (e.g. be_==), but it does offer alternative notations
> and I'm generally comfortable allowing slightly different rules to apply in
> a test framework.
>
> Lift seems to be using ever more symbols and mixed identifiers (e.g. #> and
> unsafe_!), but the range of web frameworks now available for Scala means
> that developers have alternatives to chose from if they disagree with
> Lift's decisions.
>
> Databinder Dispatch is one of the few libraries remaining that is still
> symbol-heavy, not moving towards a design with at least the *option* of
> fewer symbols, and doesn't really have a viable alternative (for example,
> nothing else I know of can offer OAuth support out of the box). The saving
> grace of dispatch is that it uses a convention which is internally very
> consistent. Usage of Dispatch is also likely to be constrained to just a
> small corner of any codebase.
>
> On 20 November 2011 13:06, Oleg Galako wrote:
>
>
>
>
>
>
>
> > I've been using Scala for about 2 years (10 years of Java before that),
> > and i've decided to post here just because of how absolutely precise (in my
> > opinion) Martin's points are.
> > - if you just do your job with Scala instead of looking for reasons why it
> > is complex or something like that, it just starts to work in your hands and
> > you like it more and more
> > - the 'complexity' feeling is mostly the feeling of freedom from (or
> > absence of) 'best practices'
> > - the only thing we need to bust the 'complexity' myth is good
> > libraries/frameworks
Re: problems of scala complexity
On 2011-11-20 7:58 PM, Bill Venners wrote:
> Hi Kevin,
>
> I just noticed this thread.
>
> On Nov 20, 5:55 am, Kevin Wright wrote:
>> Oh absolutely! It also doesn't help that many of our current "high
>> profile" libraries are particularly heavy on symbolic method names. Things
>> are improving in at least two notable libraries though:
>>
>> - Akka has culled its symbols down to essentially just ! and ?
>> - scalaz is near the end of a rewrite that will make it far easier to use
>> non-symbolic equivalents for many concepts
>>
> Hey, you forgot ScalaTest, which I would point out was designed with
> the philosophy of: *avoid any operators that users are not already
> experts in.*
Good philosophy.
> So ScalaTest originally just had ===, which means
> equality,< for less than,> for greater than,<= for less than or
> equals, and>= for greater than or equals. At some point I did add one
> non-obvious operator, the - in FreeSpec, because after much searching
> for a way to avoid it, it was the best solution. In 2.1, which is
> coming next year, I'll be adding a few more operators. One is +/- as
> an alias for plusOrMinus. This one has been requested by multiple
> users over time, and I've done several informal surveys asking people
> if they can guess its meaning, and people always have. So I decided to
> add that one. I'm also adding&&, ||, and ! for boolean-like
> expressions with the triple equals operator, and they will mean
> exactly what they look like they mean. Lots of new features are
> coming, but no other operators than these few that pass the "obvious
> test."
>
> I have been bummed that so many non-obvious operators have been
> showing up in Scala APIs. The trouble is that although obvious
> operators makes code easier to read (a + b is nicer than a add b), non-
> obvious operators makes code harder for casual users to understand,
> because they have to look up what all the meanings are. And a lot of
> users are casual, in that they don't use your library often enough to
> remember the symbols for whatever reason. And to Java programmers, at
> least, it makes Scala code look cryptic.
>
> Moreover I would claim that 99% of the time, if you work at it, you
> can find a better alternative to a non-obvious operator. Compare Specs
> tables to ScalaTest tables. Specs uses lots of operators:
>
> http://etorreborre.github.com/specs2/guide/org.specs2.guide.Matchers.htm...
>
> ScalaTest doesn't use any:
>
> http://www.scalatest.org/user_guide/table_driven_property_checks#testing...
>
> You just don't need non-obvious operators most of the time. There's a
> better way if you look for it.
>
> The feeling I get sometimes from all this is that we're a bunch of
> really smart, passionate Scala experts designing libraries for each
> other, rather than for the mainstream. Because we can certainly learn
> and understand each others cryptic operators, and maybe even enjoy the
> conciseness of the expressions we can make out of them. But the
> mainstream developer is not that interested. It's not to say they
> aren't smart. They may be super-smart, but just passionate about
> designing airplane wings or getting their startup off the ground or
> something else rather than becoming experts in our libraries. They're
> just not that into us.
Well said.
I would add, that for any software developer use whatever style you like
just so long as you know who your audience is. If you are part of a
small clique of experts in something, go with whatever works for you.
However, if you want to foster broad appeal of your APIs or libraries,
favor clarity and familiarity over concise expressiveness.
I would not even be opposed to going beyond the ASCII set of characters
and using some math or Greek symbols where they are exceptionally clear
- such as ≠ ≤ ≥ but there needs to be an easy/obvious way for people
without unicode or UTF-8 support to interoperate, and I am not sure we
are really there yet.
Cheers, Eric
>
> Bill
>
>> Specs2 uses more symbols that specs1, but they're tightly contained and
>> well documented, so this isn't a great concern. I'm not so convinced about
>> the mixed identifiers (e.g. be_==), but it does offer alternative notations
>> and I'm generally comfortable allowing slightly different rules to apply in
>> a test framework.
>>
>> Lift seems to be using ever more symbols and mixed identifiers (e.g. #> and
>> unsafe_!), but the range of web frameworks now available for Scala means
>> that developers have alternatives to chose from if they disagree with
>> Lift's decisions.
>>
>> Databinder Dispatch is one of the few libraries remaining that is still
>> symbol-heavy, not moving towards a design with at least the *option* of
>> fewer symbols, and doesn't really have a viable alternative (for example,
>> nothing else I know of can offer OAuth support out of the box). The saving
>> grace of dispatch is that it uses a convention which is internally very
>> consistent. Usage of Dispatch is also likely to be constrained to just a
>> small corner of any codebase.
>>
>> On 20 November 2011 13:06, Oleg Galako wrote:
>>
>>
>>
>>
>>
>>
>>
>>> I've been using Scala for about 2 years (10 years of Java before that),
>>> and i've decided to post here just because of how absolutely precise (in my
>>> opinion) Martin's points are.
>>> - if you just do your job with Scala instead of looking for reasons why it
>>> is complex or something like that, it just starts to work in your hands and
>>> you like it more and more
>>> - the 'complexity' feeling is mostly the feeling of freedom from (or
>>> absence of) 'best practices'
>>> - the only thing we need to bust the 'complexity' myth is good
>>> libraries/frameworks
Re: problems of scala complexity
On 21 November 2011 03:58, Bill Venners <bill [at] artima [dot] com> wrote:
Re: problems of scala complexity
Usage of non-obvious symbols and operators is one of the most significant factors leading to all this perception of complexity. Perhaps the single most important aspect holding back Scala adoption in my mind. Cryptic operators may save key strokes. The price paid in terms of readability is extremely high and indefensible. I sincerely hope more people adopt your philosophy. Regards,Dinesh
On Sun, Nov 20, 2011 at 10:58 PM, Bill Venners <bill [at] artima [dot] com> wrote:
Re: problems of scala complexity
Any symbol's utility is entirely derived from from its cognitive associativity. If you see a symbol and relate it to a useful concept it is a useful symbol to you. + is useful because almost everyone learns addition at an early age, and the association is obvious. Reusing it for string concatenation complicates matters though as this is not a commutative operation. Note that overloading is the problem here and not the use of a non-ascii symbol.
If I use a different symbol, say "zusatz", have I improved anything? Clearly I have not, unless I am German. For all non-German speakers this has no associativity and is less useful from a universal understandability point of view.
However, there are many things in computing and math that are not so universally understood. Some things that are very specific to certain domains, or even sub-domains. Is it necessary that symbols within these domains adopt a strict latin alpha-numeric symbolic structure?
The answer of course is, it depends. Some things have clear and well understood alphanumeric names. Sometimes they do not at all, and make more sense as "operators". Consider:
(3.some |@| 4.some |@| 5.some) { _ + _ + _ }
what would you call |@| ? it combines arguments into an ApplicativeBuilder, so maybe applicativeBuild?
(3.some applicativeBuild 4.some applicativeBuild 5.some) { _ + _ + _ }
This is clearly longer, and I'd argue less readable (some will say less correct as well).
Now, the first time a non-haskell background Scala newbie comes across this they are going to be confused. They are going to wonder what the hell is this |@| thing and they'll spend some time researching it. Later on though they might wonder how they are going to do combine three Option[A]s and an (A, A, A) => B to get an Option[B] and remember that surprisingly useful |@| thing.
The problem isn't the terms themselves – the problem is whether or not they encode something that will be useful later on. Are the abstractions useful – if they are then they will get used and the names will get association due to reuse.
I think Perl has a lot to answer for here though. Everyone automatically says "Perl has symbols, Perl was bad, confusing, unmaintainable – Ergo: Symbols are bad".
I am not saying I agree with indiscriminate usage of symbols, only that usage of symbols is not in itself a bad thing, certainly not "The price paid in terms of readability is extremely high and indefensible.".
On 21 November 2011 15:49, Dinesh V <viswad [at] gmail [dot] com> wrote:
Re: problems of scala complexity
Is it possible to build tools into the IDE to help the reader understand what they are seeing?
I mainly use Eclipse and I really like how when I hover the mouse over various parts of the code I get javadoc tool-tips. One thing I really like about these tool-tips is you can click to keep them open, re-size them, click links and go back and forth. The main thing is by integrating the javadoc with the tool-tips it can really lessen the WTF factor considerably.
Can something similar be done in Scala IDEs? Maybe even going beyond the scaladoc to some mechanism the helps people unfamiliar with the domain, or particular library/API style to better fathom what things mean?
Cheers, Eric
On 2011-11-20 11:55 PM, Jed Wesley-Smith wrote:
Re: problems of scala complexity
On Mon, Nov 21, 2011 at 8:42 AM, Eric Kolotyluk <eric [dot] kolotyluk [at] gmail [dot] com> wrote:
Note that Scala doesn't have operator overloading, the problem we're discussing is that methods can be named using arbitrary UTF characters.
IDE's already support this, it's called Javadoc :-)
But the problem with symbols is much deeper than this. For example, suppose you come across |@| in Scalaz code and you have never studied Haskell, much less applicatives. Your first reflex will be to google it. Searching for |@| alone yields no result (unsurprisingly) but "scalaz |@|" shows more promise. However, the first two links point to Scalaz' main project page, and none of them even contain "|@|" anywhere.
This result should be obvious if you know how Google search works (it probably simply discarded |@| form the query) but it's one of the main problems with symbols: they're hard to look up.
Google code search is of no help either, by the way.
You're pretty much dead in the water at this point.
So your only hope is to dive into the Scalaz code. Finding the definition of |@| is pretty easy, but take a look at the source... Yup, not a single comment, and a strong warning at the top telling you not to use it directly.
To make things worse, "applicative builders" don't exist in Haskell, so even if you decided to bite the bullet and educate yourself in Haskell (never a bad idea, but not everybody has the time), walking the ladder from functors to applicatives and finally to monads will be a long and arduous climb.
tl;dr: FP concepts can already be daunting in and of themselves, don't make things worse by using symbols in your source.
-- Cédric
Re: problems of scala complexity
This is not an argument against well-chosen symbolic methods (i.e. operators), but it is an argument in favor of fanatic efforts at documentation given that some of the usual approaches do not work. Cryptic anything is bad, not just symbolic method names. (Thus, poorly-chosen method names are bad, whether it is << instead of print, or dividedBy instead of /.)
|@| is a bit of a questionable case. In general, I think
a :: b :: Nil
x #:: y #:: Stream.Empty
is less clear than
List(a,b)
Stream(a,b)
and thus
a |@| b
would be less ideal than one of
ApF(a,b)
(a,b) toApF
in most cases.
On the other hand, mimicing BNF with ~ is so incredibly helpful that it basically makes the difference between me using parser combinators and not. If one is copying some nontrivial structure from a document that uses the circled star notation, one could be equally desiring of |@|.
Any time one starts writing
a.plus(b.plus(c),b.times(d)) // a + b + c + b*d
it is a clear indication that symbolic method names are needed.
--Rex
Re: problems of scala complexity
2011/11/21 Rex Kerr <ichoran [at] gmail [dot] com>
I agree. If you have /. as the def, it isn't google-visible. However, if its scaladoc includes text like 'divide' and 'divided by', there's a fighting chance people will be able to find it. Symbols can be far clearer than names in some situations, but without scaladoc they can't be googled for, and without a federated, type-aware scala search engine, in practice google is what people use to find out about things they aren't intimately knowledgeable about..
Matthew
--
Dr Matthew PocockIntegrative Bioinformatics Group, School of Computing Science, Newcastle Universitymailto: turingatemyhamster [at] gmail [dot] com gchat: turingatemyhamster [at] gmail [dot] commsn: matthew_pocock [at] yahoo [dot] co [dot] uk irc.freenode.net: drdozerskype: matthew.pococktel: (0191) 2566550mob: +447535664143
Re: problems of scala complexity
On Nov 22, 2011 3:54 AM, "Rex Kerr" <ichoran [at] gmail [dot] com> wrote:
>
> 2011/11/21 Cédric Beust ♔ <cedric [at] beust [dot] com>
>>
>>
>> But the problem with symbols is much deeper than this. For example, suppose you come across |@| in Scalaz code and you have never studied Haskell, much less applicatives. Your first reflex will be to google it. Searching for |@| alone yields no result (unsurprisingly) but "scalaz |@|" shows more promise. However, the first two links point to Scalaz' main project page, and none of them even contain "|@|" anywhere.
>
>
> This is not an argument against well-chosen symbolic methods (i.e. operators), but it is an argument in favor of fanatic efforts at documentation given that some of the usual approaches do not work. Cryptic anything is bad, not just symbolic method names. (Thus, poorly-chosen method names are bad, whether it is << instead of print, or dividedBy instead of /.)
>
> |@| is a bit of a questionable case.
It also has nothing to do with haskell. It doesn't appear in haskell anywhere and exists solely for purposes that are specific to scala.
That's twice now that someone has decided to appeal to haskell unwarranted on the nature of |@|, which is an obvious indication that there has been no effort whatsoever to gather any understanding of the subject matter. In my mind that is an immediate forfeiture and it is no wonder this discussion goes in circles.
Re: problems of scala complexity
On Mon, Nov 21, 2011 at 12:06 PM, Tony Morris <tmorris [at] tmorris [dot] net> wrote:I think we're running in circles mostly because you don't seem to even bother reading the messages you reply to. Here is what I wrote, again:
-- Cédric
Re: problems of scala complexity
Oh look it's mister honest.
On Nov 22, 2011 7:02 AM, "Cédric Beust ♔" <cedric [at] beust [dot] com> wrote:Re: problems of scala complexity
Actually, it ends up being a choice between:
(a |@| b)(_ + _)
ApF[({type l[a]=State[Int, a]})#l](a, b)(_ + _)
type IntState[a]=State[Int, A] ApF[IntState](a, b)(_ + _)
As I said earlier, I'm pursuing alternatives, but there are some limitations in scalac that even Miles hasn't been able to dream up a way around. We're pretty close, though, and some small changes we need in scalac are on the horizon.
-jason
Re: problems of scala complexity
Heiko
--
Heiko Seeberger
Twitter: hseeberger
Blog: heikoseeberger.name
Company: Typesafe - Enterprise-Grade Scala from the Experts
Author of Durchstarten mit Scala, a German tutorial-style Scala book
On Nov 21, 2011, at 7:08 PM, Jason Zaugg wrote:
RE: problems of scala complexity
It matters, it matters: for instance (a tuple) doesn’t compile… J J
From: scala-debate [at] googlegroups [dot] com [mailto:scala-debate [at] googlegroups [dot] com] On Behalf Of Heiko Seeberger
Sent: November-21-11 2:00 PM
To: scala-debate [at] googlegroups [dot] com
Subject: Re: [scala-debate] problems of scala complexity
By the way: I don't believe the "Java Joe is too stupid, all (s)he knows is how to use Google" story. Therefore it does not matter whether it is |@| or , (a tuple).
Heiko
--
Heiko Seeberger
Twitter: hseeberger
Blog: heikoseeberger.name
Company: Typesafe - Enterprise-Grade Scala from the Experts
Author of Durchstarten mit Scala, a German tutorial-style Scala book
On Nov 21, 2011, at 7:08 PM, Jason Zaugg wrote:
2011/11/21 Rex Kerr <ichoran [at] gmail [dot] com>
and thus
a |@| b
would be less ideal than one of
ApF(a,b)
(a,b) toApF
in most cases.
Actually, it ends up being a choice between:
(a |@| b)(_ + _)
ApF[({type l[a]=State[Int, a]})#l](a, b)(_ + _)
type IntState[a]=State[Int, A]
ApF[IntState](a, b)(_ + _)
As I said earlier, I'm pursuing alternatives, but there are some limitations in scalac that even Miles hasn't been able to dream up a way around. We're pretty close, though, and some small changes we need in scalac are on the horizon.
-jason
Re: problems of scala complexity
> Note that Scala doesn't have operator overloading, the problem we're
> discussing is that methods can be named using arbitrary UTF characters.
I'm sorry, but how is this a problem? Even remotely? Symbols are used
everywhere, in all kinds of fields. Where does it say that symbols
cannot be used in a coherent manner that will aid the description of
logic or a solution?
This is purely a familiarity problem.
> IDE's already support this, it's called Javadoc :-)
> But the problem with symbols is much deeper than this. For example, suppose
> you come across |@| in Scalaz code and you have never studied Haskell, much
> less applicatives. Your first reflex will be to google it. Searching for |@|
> alone yields no result (unsurprisingly) but "scalaz |@|" shows more promise.
> However, the first two links point to Scalaz' main project page, and none of
> them even contain "|@|" anywhere.
>
> This result should be obvious if you know how Google search works (it
> probably simply discarded |@| form the query) but it's one of the main
> problems with symbols: they're hard to look up.
> Google code search is of no help either, by the way.
> You're pretty much dead in the water at this point.
> So your only hope is to dive into the Scalaz code. Finding the definition of
> |@| is pretty easy, but take a look at the source... Yup, not a single
> comment, and a strong warning at the top telling you not to use it directly.
> To make things worse, "applicative builders" don't exist in Haskell, so even
> if you decided to bite the bullet and educate yourself in Haskell (never a
> bad idea, but not everybody has the time), walking the ladder from functors
> to applicatives and finally to monads will be a long and arduous climb.
It might be a climb, but the end results speak for themselves.
Additionally, there is a "culture" with programming these days that
the IDE seems to do more than the developer. This is truly concerning.
An IDE is nothing more than a tool (as is everything else) you need to
use it correctly. you can be just as productive with Scaladoc and a
text editor. Perhaps the issues are with how things are defined,
rather with how people expect that new and important concepts that
need to be understood require a little effort?
Re: problems of scala complexity
On 2011-11-21 9:23 AM, Gary Pampara wrote:
>> Note that Scala doesn't have operator overloading, the problem we're
>> discussing is that methods can be named using arbitrary UTF characters.
The confusion over operator overloading was my fault as I was comparing
my experiences with operator overloading in C++ (not being able to
recognize what was going on) against my experiences with Scala where
there can be many new operators (that can also make it hard to recognize
what is going on).
> I'm sorry, but how is this a problem? Even remotely? Symbols are used
> everywhere, in all kinds of fields. Where does it say that symbols
> cannot be used in a coherent manner that will aid the description of
> logic or a solution?
>
> This is purely a familiarity problem.
>> IDE's already support this, it's called Javadoc :-)
>> But the problem with symbols is much deeper than this. For example, suppose
>> you come across |@| in Scalaz code and you have never studied Haskell, much
>> less applicatives. Your first reflex will be to google it. Searching for |@|
>> alone yields no result (unsurprisingly) but "scalaz |@|" shows more promise.
>> However, the first two links point to Scalaz' main project page, and none of
>> them even contain "|@|" anywhere.
>>
>> This result should be obvious if you know how Google search works (it
>> probably simply discarded |@| form the query) but it's one of the main
>> problems with symbols: they're hard to look up.
>> Google code search is of no help either, by the way.
>> You're pretty much dead in the water at this point.
>> So your only hope is to dive into the Scalaz code. Finding the definition of
>> |@| is pretty easy, but take a look at the source... Yup, not a single
>> comment, and a strong warning at the top telling you not to use it directly.
>> To make things worse, "applicative builders" don't exist in Haskell, so even
>> if you decided to bite the bullet and educate yourself in Haskell (never a
>> bad idea, but not everybody has the time), walking the ladder from functors
>> to applicatives and finally to monads will be a long and arduous climb.
> It might be a climb, but the end results speak for themselves.
> Additionally, there is a "culture" with programming these days that
> the IDE seems to do more than the developer. This is truly concerning.
I would disagree and say that the IDE should do more than the developer,
and that they still don't do enough. The more the IDE does for me, the
more I can focus my thoughts and intent of solving the problem at had,
the less I can worry about the language, libraries/APSs, etc.
>
> An IDE is nothing more than a tool (as is everything else) you need to
> use it correctly. you can be just as productive with Scaladoc and a
> text editor. Perhaps the issues are with how things are defined,
> rather with how people expect that new and important concepts that
> need to be understood require a little effort?
Again I disagree. To me using Scaladoc and a text editor is similar to
using a horse and buggy, where as using and IDE is like using a car
which can get me where I need to be faster.
Lately I have been writing some Scala code in Eclipse, and I am
constantly reminded that the Scala editor is like a Model T Ford
compared to the SUV that the Java editor is. From my perspective, the
Java editor in Eclipse significantly reduces the complexity of Java for
me - the complexity of the process of writing code.
Cheers, Eric
Re: problems of scala complexity
Sorry, I should have said "I often ran into similar problems with C++ because of operator overloading." The problem of know knowing what I was seeing.
I meant something beyond javadoc.
Re: problems of scala complexity
Hi Jed,
On Nov 20, 11:55 pm, Jed Wesley-Smith
wrote:
> This is such an overstated straw-man argument.
>
> Any symbol's utility is entirely derived from from its cognitive
> associativity. If you see a symbol and relate it to a useful concept it is
> a useful symbol to you. + is useful because almost everyone learns addition
> at an early age, and the association is obvious. Reusing it for string
> concatenation complicates matters though as this is not a commutative
> operation. Note that overloading is the problem here and not the use of a
> non-ascii symbol.
>
> If I use a different symbol, say "zusatz", have I improved anything?
> Clearly I have not, unless I am German. For all non-German speakers this
> has no associativity and is less useful from a universal understandability
> point of view.
>
> However, there are many things in computing and math that are not so
> universally understood. Some things that are very specific to certain
> domains, or even sub-domains. Is it necessary that symbols within these
> domains adopt a strict latin alpha-numeric symbolic structure?
>
> The answer of course is, it depends. Some things have clear and well
> understood alphanumeric names. Sometimes they do not at all, and make more
> sense as "operators". Consider:
>
> (3.some |@| 4.some |@| 5.some) { _ + _ + _ }
>
> what would you call |@| ? it combines arguments into an ApplicativeBuilder,
> so maybe applicativeBuild?
>
> (3.some applicativeBuild 4.some applicativeBuild 5.some) { _ + _ + _ }
>
> This is clearly longer, and I'd argue less readable (some will say less
> correct as well).
>
I'd agree applicativeBuild is less readable than the |@| form once you
learn the meaning, and Haskell programmers probably already recognize
it. But to non-Haskell programmers it will look cryptic until they
learn it (if they learn it). But I'd point out the difference between |
@| and the operators in specs or sbt is that the latter operators are
just made up, invented out of thin air. The operators in specs have no
meaning in the testing domain, just as the operators in sbt have no
meaning in the build domain. To learn those operators, you have to
become an expert in that library and that knowledge does not benefit
you outside the library. To learn |@| by contrast, you need to become
an expert in something larger than scalaz, and you gain more than just
learning how to use scalaz. That's the difference.
Another example is domain-specific libraries for scientists. Say I'm
asked to write a library for a bunch of bio-informaticians who are
working in a certain area. Those folks may have some symbols they use
to communicate precisely. All are familiar with them. They use them in
the papers they write for each other, etc. I think it is likely that
using similar symbols in the API you make for them could make the code
easier for everyone to write and read. Sure, when a new programmer
gets hired who hasn't worked in bio-informatics before, that person
will have some symbols to learn, which at first will look cryptic to
them. But in learning about those, the new programmer isn't just
learning about the API, they are learning about the corner of bio-
informatics they are now working in. They are learning about the
domain, and that's going to help them be productive in that
environment.
The key is, do those symbols already have meaning in some domain and
are your users (or at least most of them) already experts in those
symbols. If not, then try and find a different way to say it that
doesn't use operators. For example, in specs you compose matchers with
^^, which is a made up operator. In ScalaTest you say "compose", which
is longer to type. But that's not actually a ScalaTest method.
ScalaTest matchers are Function1s, and so that's the compose method on
Function1. So if you don't know about compose, you'll need learn that,
but in the process you'll learn about function composition in Scala
(and in functional programming) in general, whereas with ^^ you just
learn something that has meaning only in specs.
Bill
> Now, the first time a non-haskell background Scala newbie comes across this
> they are going to be confused. They are going to wonder what the hell is
> this |@| thing and they'll spend some time researching it. Later on though
> they might wonder how they are going to do combine three Option[A]s and an
> (A, A, A) => B to get an Option[B] and remember that surprisingly useful
> |@| thing.
>
> The problem isn't the terms themselves – the problem is whether or not they
> encode something that will be useful later on. Are the abstractions useful
> – if they are then they will get used and the names will get association
> due to reuse.
>
> I think Perl has a lot to answer for here though. Everyone automatically
> says "Perl has symbols, Perl was bad, confusing, unmaintainable – Ergo:
> Symbols are bad".
>
> I am not saying I agree with indiscriminate usage of symbols, only that
> usage of symbols is not in itself a bad thing, certainly not "The price
> paid in terms of readability is extremely high and indefensible.".
>
> On 21 November 2011 15:49, Dinesh V wrote:
>
>
>
>
>
>
>
>
>
> > Usage of non-obvious symbols and operators is one of the most significant
> > factors leading to all this perception of complexity. Perhaps the single
> > most important aspect holding back Scala adoption in my mind.
>
> > Cryptic operators may save key strokes. The price paid in terms of
> > readability is extremely high and indefensible.
Re: problems of scala complexity
I just want to point out that "^^" in specs2 was not completely taken out of thin air. I actually took the inspiration from the Parsers combinators library, because this is how you transform a Parser[T] into a Parser[U]. The analogy here is similar, where you transform a Matcher[T] into a Matcher[U]. So if you learn the specs2 way, you will also get the idea for Parsers. "compose" would not fit here because specs matchers are not Function1s.
About DataTables, I do agree that they're heavy on syntax but they fit my purpose which is to get a visually tabular notation for my examples. Which drives me to say that I wrote specs/specs2 first and foremost to learn Scala, and to give myself a library which I enjoy using. As it so happens, other people enjoy using it too, and I'm very dedicated to supporting them with whatever silly bug / sick feature I impose upon them. But then I think we should accept that one size does not fit all. There is room for different tastes in the ecosystem and we should just be grateful that Scala, as a language, gives us several ways of expressing ourselves. I love specs/specs2 because that's how I like it and I also love ScalaTest because that gives other users the possibility to use something different without having to push me in a direction where I might not want to go. I also love it because you gave me ideas for specs2 and because you're pushing hard to make sure that ScalaTest is not breaking client code and this is a lesson for every library maintainer.
It would be a sad world if we only had ScalaTest and not specs2, Scalatra and not Unfiltered, and so on just because we have to please the majority.
Eric.
Re: problems of scala complexity
Hi Eric,
On Nov 21, 2:36 pm, etorreborre wrote:
> Hi Bill,
>
> I just want to point out that "^^" in specs2 was not completely taken out
> of thin air. I actually took the inspiration from the Parsers combinators
> library,
> because this is how you transform a Parser[T] into a Parser[U]. The analogy
> here is similar, where you transform a Matcher[T] into a Matcher[U]. So if
> you learn the specs2 way, you will also get the idea for Parsers. "compose"
> would not fit here because specs matchers are not Function1s.
>
That's interesting. I hadn't made that connection with the parser
combinators "eyebrows" operator, but I see it now that you mention
it.
> About DataTables, I do agree that they're heavy on syntax but they fit *my*purpose which is to get a visually tabular notation for my examples. Which
> drives me to say that I wrote specs/specs2 first and foremost to learn
> Scala, and to give myself a library which I enjoy using. As it so happens,
> other people enjoy using it too, and I'm very dedicated to supporting them
> with whatever silly bug / sick feature I impose upon them. But then I think
> we should accept that one size does not fit all. There is room for
> different tastes in the ecosystem and we should just be grateful that
> Scala, as a language, gives us several ways of expressing ourselves. I love
> specs/specs2 because that's how I like it and I also love ScalaTest because
> that gives other users the possibility to use something different without
> having to push me in a direction where I might not want to go. I also love
> it because you gave me ideas for specs2 and because you're pushing hard to
> make sure that ScalaTest is not breaking client code and this is a lesson
> for every library maintainer.
>
> It would be a sad world if we only had ScalaTest and not specs2, Scalatra
> and not Unfiltered, and so on just because we have to please the majority.
>
I wholeheartedly agree. Sorry, I'm not trying to say we should all do
things the same way, just that a Scala API designer can, if they try,
usually find a way to avoid a made-up, non-obvious operator. I
highlighted specs because I also wrote a test framework and could
point out specific examples of alternatives to using non-obvious
operators. If I'd written a build system I might have picked on sbt. I
think those <+= etc symbols in sbt will make it harder for casual
users to figure out sbt build files, and I expect a great many users
of build tools are casual, i.e., non-expert-in-the-build-tool ones.
But it is hard for me to say for sure what might be an alternative way
to avoid those operators given I haven't ever tried to create a build
tool. I haven't walked in those shoes. Maybe there aren't any
satisfying alternatives to those operators. I myself did put one non-
obvious operator in ScalaTest, the dash in FreeSpec:
http://www.scalatest.org/scaladoc/1.6.1/#org.scalatest.FreeSpec
I thought long and hard to find a way around that, but there just
wasn't one that was as satisfying, so in it went even though it was
non-obvious.
Anyway, the I think the Scala community has definitely benefited from
having multiple test frameworks, web frameworks, etc., with different
design goals and designer tastes and visions to chose from. We inspire
each other and serve different niches. But still I think the community
of Scala library designers in general has been way too loose on the
operator front lately, and I think that lessens the approachability of
Scala to the masses to some extent. They aren't the end of the world,
but I think Scala would be more approachable if we used fewer
operators.
Bill
Re: Re: problems of scala complexity
Am Dienstag, 22. November 2011, 02:04:42 schrieb Bill Venners:
> Hi Eric,
...
> Anyway, the I think the Scala community has definitely benefited from
> having multiple test frameworks, web frameworks, etc., with different
> design goals and designer tastes and visions to chose from. We inspire
> each other and serve different niches. But still I think the community
> of Scala library designers in general has been way too loose on the
> operator front lately, and I think that lessens the approachability of
> Scala to the masses to some extent. They aren't the end of the world,
> but I think Scala would be more approachable if we used fewer
> operators.
>
> Bill
As Eric said - one size doesn't fit all. I suppose heavy operator use will be
here to stay. Scala gives this option - people will use it.
What I can imagine to ease this situation for those who prefer descriptive
names (the "casual user"): if we had a possibility to add a descriptive alias
(e.g. as annotation) which could be used by IDEs to support hoover help for
those operators. This could help even in the absence of documentation.
Maybe it would be enough just to add the "long name" as the first word in the
docs to enable a quick "ah yes - that was it".
At least everybody delivering public libraries should be aware that operators
while intended to speed up or tidy up source code can achieve the opposite.
Just my 5cents
Greetings
Bernd