This page is no longer maintained — Please continue to the home page at www.scala-lang.org

Classloader for macros

9 replies
xeno.by
Joined: 2011-07-18,
User offline. Last seen 42 years 45 weeks ago.

A few days ago we've got the first end-to-end prototype of macros
working: https://github.com/scalamacros/kepler/commits/topic/macros
(if you wish to take a look, go to sandbox in the root of the clone
and run the test script), and this brought an important question.

Here's a simplified version of the test script mentioned above:
> scalac -Xexperimental Printf.scala -d ./macros
> scalac -toolcp ./macros:$toolcp -Xexperimental Test.scala
> scala Test
hello world

Our implementation of macros involves two-step compilation scheme.
During the first step, you compile macros themselves (i.e. macros get
transformed from macro defs into tree transformers and then get
compiled into bytecodes). During the second step, you compile your
application as usual. Then you run the application in a normal fashion
(which is expected, since macros only affect compilation).

To glue the first and the second compilation runs, we need to somehow
tell the compiler where to look for the bytecodes of macro bodies.
This raises the question: how do we do that?

Basically, there are three alternatives:
1) Use normal classpath: scalac -cp ...
2) Use tool classpath: scalac -toolcp ...
3) Use separate classpath: scalac -macrocp ...

Today we've discussed this point with Martin and came to the following
conclusions.

The first option seems to be the most natural. If we want macros to
feel totally like regular methods (and we do our best to achieve
that), then it's the obvious choice.

The second option seems dangerous. Why would we allow anyone to mess
with compiler's internal classpath? Our current implementation does
exactly that just because it's simple to implement, but this will be
changed once the decision is finalized.

The third option seems awkward. Despite of, say, Nemerle going this
route, we wouldn't like the users jump through hoops to use macros.
That would negate our efforts towards seamless integration of macros
into the language.

However, classpath matters have proven to be very subtle, so I'd like
to ask for your feedback. Do you also prefer option #1? What would you
like to add to the discussion?

Ismael Juma 2
Joined: 2011-01-22,
User offline. Last seen 42 years 45 weeks ago.
Re: Classloader for macros
On Wed, Dec 21, 2011 at 4:07 PM, Eugene Burmako <xeno [dot] by [at] gmail [dot] com> wrote:
However, classpath matters have proven to be very subtle, so I'd like
to ask for your feedback. Do you also prefer option #1?

Yes, please. It's great that Scala is just a normal library when it comes to deployment and we should do our best to keep it that way, in my opinion.
Best,Ismael
extempore
Joined: 2008-12-17,
User offline. Last seen 35 weeks 3 days ago.
Re: Classloader for macros


On Wed, Dec 21, 2011 at 8:07 AM, Eugene Burmako <xeno [dot] by [at] gmail [dot] com> wrote:
However, classpath matters have proven to be very subtle, so I'd like
to ask for your feedback. Do you also prefer option #1? What would you
like to add to the discussion?

You can do #1, and if sufficient need is demonstrated for a separate classpath, #3 can be made available without interfering with #1.  If a -macrocp was given, put it in front of the regular classpath when compiling macros, otherwise compile as usual.
odersky
Joined: 2008-07-29,
User offline. Last seen 45 weeks 6 days ago.
Re: Classloader for macros

On Wed, Dec 21, 2011 at 5:35 PM, Paul Phillips wrote:
>
>
> On Wed, Dec 21, 2011 at 8:07 AM, Eugene Burmako wrote:
>>
>> However, classpath matters have proven to be very subtle, so I'd like
>> to ask for your feedback. Do you also prefer option #1? What would you
>> like to add to the discussion?
>
>
> You can do #1, and if sufficient need is demonstrated for a separate
> classpath, #3 can be made available without interfering with #1.  If a
> -macrocp was given, put it in front of the regular classpath when compiling
> macros, otherwise compile as usual.
>
I also think we should do #1. This brings up the question, what's the
best way to do this? Is there an easy way to obtain a Java ClassLoader
from a Scala ClassPath?

Cheers

extempore
Joined: 2008-12-17,
User offline. Last seen 35 weeks 3 days ago.
Re: Classloader for macros


On Wed, Dec 21, 2011 at 9:54 AM, martin odersky <martin [dot] odersky [at] epfl [dot] ch> wrote:
I also think we should do #1. This brings up the question, what's the
best way to do this? Is there an easy way to obtain a Java ClassLoader
from a Scala ClassPath?

Well, I'm glad you asked.
So what I'd like to do, and all I really need is a little rubber stamping, is re-model everything which presently operates on classpaths to operate on a different abstraction, something with a new name to evade conceptual conflation, like "ClassfileProvider." It'd at its most basic look something like
  trait ClassfileProvider {    def classBytes(className: String): Array[Byte]  }
This would be augmented by more capable variations such as:
  trait ClassfileLoader extends ClassfileProvider {    def classLoader: ScalaClassLoader    def classFile(className: String): Class[_]  }
  trait EnumerableClassfileProvider extends ClassfileProvider {     def allClassNames(): Iterable[String]    def allClassFiles(): Iterable[Class[_]]  }
etc.  And then standard implementations like
  class JarClassfileLoader(jar: File) extends EnumerableClassfileProvider { ... }   class MavenClassfileLoader(group: String, id: String, version: String) extends EnumerableClassfileProvider { ... }  // ... URL, Directory, Virtual (like the repl), Hierarchical, ...
And mechanisms for composition etc.  I want to be able to invoke scala like this (although it might have to be something other than '-cp')

  scala -cp asm%asm%3.2:org.scalaz%scalaz-core%7.0 my.fancy.Program
...and have the standard way of translating "classprovider expressions" into "classproviders" resolve those through the ivy cache.  The existing classpath is just one specific case of that (where all "expressions" are filesystem paths, and the provider is always a "filesystem provider".)
Lots of good stuff emerges from this approach.  I can provide a stub implementation in fairly short order, and I also have a large classpath rewrite which is I think an undeniable improvement but which I've had lying around for months because it mucks with the interface and I assume that will break the IDE.  Let me know.
odersky
Joined: 2008-07-29,
User offline. Last seen 45 weeks 6 days ago.
Re: Classloader for macros

On Wed, Dec 21, 2011 at 7:23 PM, Paul Phillips wrote:
>
>
> On Wed, Dec 21, 2011 at 9:54 AM, martin odersky
> wrote:
>>
>> I also think we should do #1. This brings up the question, what's the
>> best way to do this? Is there an easy way to obtain a Java ClassLoader
>> from a Scala ClassPath?
>
>
> Well, I'm glad you asked.
>
> So what I'd like to do, and all I really need is a little rubber stamping,
> is re-model everything which presently operates on classpaths to operate on
> a different abstraction, something with a new name to evade conceptual
> conflation, like "ClassfileProvider." It'd at its most basic look something
> like
>
>   trait ClassfileProvider {
>     def classBytes(className: String): Array[Byte]
>   }
>
> This would be augmented by more capable variations such as:
>
>   trait ClassfileLoader extends ClassfileProvider {
>     def classLoader: ScalaClassLoader
>     def classFile(className: String): Class[_]
>   }
>
>   trait EnumerableClassfileProvider extends ClassfileProvider {
>     def allClassNames(): Iterable[String]
>     def allClassFiles(): Iterable[Class[_]]
>   }
>
> etc.  And then standard implementations like
>
>   class JarClassfileLoader(jar: File) extends EnumerableClassfileProvider {
> ... }
>   class MavenClassfileLoader(group: String, id: String, version:
> String) extends EnumerableClassfileProvider { ... }
>   // ... URL, Directory, Virtual (like the repl), Hierarchical, ...
>
Looks like an excellent abstraction to me. I think now that the IDE is
out is the best moment to change over. Ccing the IDE team to see if
they have issues.

Eugene: In light of this I think we can defer the classloader issue
for the moment (i.e. simply include output directory in tools
classpath, so that the normal classloader works). It's a hack, but it
will go away once ClassFileProviders are in.

Cheers

Viktor Klang
Joined: 2008-12-17,
User offline. Last seen 1 year 27 weeks ago.
Re: Classloader for macros
ClassProviderFactorySingletonMetaBuilder

On Wed, Dec 21, 2011 at 7:23 PM, Paul Phillips <paulp [at] improving [dot] org> wrote:


On Wed, Dec 21, 2011 at 9:54 AM, martin odersky <martin [dot] odersky [at] epfl [dot] ch> wrote:
I also think we should do #1. This brings up the question, what's the
best way to do this? Is there an easy way to obtain a Java ClassLoader
from a Scala ClassPath?

Well, I'm glad you asked.
So what I'd like to do, and all I really need is a little rubber stamping, is re-model everything which presently operates on classpaths to operate on a different abstraction, something with a new name to evade conceptual conflation, like "ClassfileProvider." It'd at its most basic look something like
  trait ClassfileProvider {    def classBytes(className: String): Array[Byte]  }
This would be augmented by more capable variations such as:
  trait ClassfileLoader extends ClassfileProvider {    def classLoader: ScalaClassLoader    def classFile(className: String): Class[_]  }
  trait EnumerableClassfileProvider extends ClassfileProvider {     def allClassNames(): Iterable[String]    def allClassFiles(): Iterable[Class[_]]  }
etc.  And then standard implementations like
  class JarClassfileLoader(jar: File) extends EnumerableClassfileProvider { ... }   class MavenClassfileLoader(group: String, id: String, version: String) extends EnumerableClassfileProvider { ... }  // ... URL, Directory, Virtual (like the repl), Hierarchical, ...
And mechanisms for composition etc.  I want to be able to invoke scala like this (although it might have to be something other than '-cp')

  scala -cp asm%asm%3.2:org.scalaz%scalaz-core%7.0 my.fancy.Program
...and have the standard way of translating "classprovider expressions" into "classproviders" resolve those through the ivy cache.  The existing classpath is just one specific case of that (where all "expressions" are filesystem paths, and the provider is always a "filesystem provider".)
Lots of good stuff emerges from this approach.  I can provide a stub implementation in fairly short order, and I also have a large classpath rewrite which is I think an undeniable improvement but which I've had lying around for months because it mucks with the interface and I assume that will break the IDE.  Let me know.



--
Viktor Klang

Akka Tech LeadTypesafe - Enterprise-Grade Scala from the Experts

Twitter: @viktorklang
xeno.by
Joined: 2011-07-18,
User offline. Last seen 42 years 45 weeks ago.
Re: Classloader for macros

Well, I played with this a bit, and there is an easy way to obtain a
classloader that corresponds to -cp. You just use
ScalaClassLoader.fromURLs(global.classPath.asURLs). So I've been able
to experiment with the idea of a special classloader for macros.

It was relatively easy to instantiate a mirror with a custom
classloader, but then I stumbled upon an "object is not an instance of
a class" problem. Possibly, I have overlooked something, so this is
not a blocker, but it raises a question. Is it okay that macro
expansion is executed in the context of a classloader that is
different from compiler's one?

On Dec 21, 7:40 pm, martin odersky wrote:
> On Wed, Dec 21, 2011 at 7:23 PM, Paul Phillips wrote:
>
> > On Wed, Dec 21, 2011 at 9:54 AM, martin odersky
> > wrote:
>
> >> I also think we should do #1. This brings up the question, what's the
> >> best way to do this? Is there an easy way to obtain a Java ClassLoader
> >> from a Scala ClassPath?
>
> > Well, I'm glad you asked.
>
> > So what I'd like to do, and all I really need is a little rubber stamping,
> > is re-model everything which presently operates on classpaths to operate on
> > a different abstraction, something with a new name to evade conceptual
> > conflation, like "ClassfileProvider." It'd at its most basic look something
> > like
>
> >   trait ClassfileProvider {
> >     def classBytes(className: String): Array[Byte]
> >   }
>
> > This would be augmented by more capable variations such as:
>
> >   trait ClassfileLoader extends ClassfileProvider {
> >     def classLoader: ScalaClassLoader
> >     def classFile(className: String): Class[_]
> >   }
>
> >   trait EnumerableClassfileProvider extends ClassfileProvider {
> >     def allClassNames(): Iterable[String]
> >     def allClassFiles(): Iterable[Class[_]]
> >   }
>
> > etc.  And then standard implementations like
>
> >   class JarClassfileLoader(jar: File) extends EnumerableClassfileProvider {
> > ... }
> >   class MavenClassfileLoader(group: String, id: String, version:
> > String) extends EnumerableClassfileProvider { ... }
> >   // ... URL, Directory, Virtual (like the repl), Hierarchical, ...
>
> Looks like an excellent abstraction to me. I think now that the IDE is
> out is the best moment to change over. Ccing the IDE team to see if
> they have issues.
>
> Eugene: In light of this I think we can defer the classloader issue
> for the moment (i.e. simply include output directory in tools
> classpath, so that the normal classloader works). It's a hack, but it
> will go away once ClassFileProviders are in.
>
> Cheers
>
>  -- Martin

loverdos
Joined: 2008-11-18,
User offline. Last seen 2 years 27 weeks ago.
Re: Classloader for macros

Hi Paul,

On Dec 21, 2011, at 8:23 PM, Paul Phillips wrote:

>
>
> On Wed, Dec 21, 2011 at 9:54 AM, martin odersky wrote:
> I also think we should do #1. This brings up the question, what's the
> best way to do this? Is there an easy way to obtain a Java ClassLoader
> from a Scala ClassPath?
>
> Well, I'm glad you asked.
>
> So what I'd like to do, and all I really need is a little rubber stamping, is re-model everything which presently operates on classpaths to operate on a different abstraction, something with a new name to evade conceptual conflation, like "ClassfileProvider." It'd at its most basic look something like
>
> trait ClassfileProvider {
> def classBytes(className: String): Array[Byte]
> }
>
> This would be augmented by more capable variations such as:
>
> trait ClassfileLoader extends ClassfileProvider {
> def classLoader: ScalaClassLoader
> def classFile(className: String): Class[_]
> }
>
> trait EnumerableClassfileProvider extends ClassfileProvider {
> def allClassNames(): Iterable[String]
> def allClassFiles(): Iterable[Class[_]]
> }
>
> etc. And then standard implementations like
>
> class JarClassfileLoader(jar: File) extends EnumerableClassfileProvider { ... }
> class MavenClassfileLoader(group: String, id: String, version: String) extends EnumerableClassfileProvider { ... }
> // ... URL, Directory, Virtual (like the repl), Hierarchical, ...
>
> And mechanisms for composition etc. I want to be able to invoke scala like this (although it might have to be something other than '-cp')
>
> scala -cp asm%asm%3.2:org.scalaz%scalaz-core%7.0 my.fancy.Program
>
> ...and have the standard way of translating "classprovider expressions" into "classproviders" resolve those through the ivy cache. The existing classpath is just one specific case of that (where all "expressions" are filesystem paths, and the provider is always a "filesystem provider".)
>
> Lots of good stuff emerges from this approach. I can provide a stub implementation in fairly short order, and I also have a large classpath rewrite which is I think an undeniable improvement but which I've had lying around for months because it mucks with the interface and I assume that will break the IDE. Let me know.

Let's keep in mind, from the operations perspective, that an application deployed to a server will most certainly not be given the flexibility to download needed jars but they will have to be at some local CLASSPATH (normally near the application).

That said, I, too, like the idea of the providers stuff. Maybe you would like to have a look at my context-based resource provisioning library with similar usage semantics:

https://github.com/loverdos/streamresource

It basically generalizes the getResource() pattern and delegates resource resolving/provisioning to resource contexts (= the resolvers/providers) that can be composed in hierarchies.

https://github.com/loverdos/streamresource/blob/master/src/main/scala/co...
https://github.com/loverdos/streamresource/blob/master/src/main/scala/co...
https://github.com/loverdos/streamresource/blob/master/src/main/scala/co...

The last one, CompositeStreamResourceContext, is the entity that can give a prioritization to a collection of resource contexts (which one is asked first to provide a resource and so on). My typical use-case is: where do I find an application's properties file? So, I have a default classpath-based context that resolves from the classloader and then I override this with a filesystem-based context. So, if the application is started with a manually provided properties file in the filesystem, then that one is chosen, otherwise the properties from the classpath is chosen.

Best,
Christos

extempore
Joined: 2008-12-17,
User offline. Last seen 35 weeks 3 days ago.
Re: Classloader for macros

On Thu, Dec 22, 2011 at 4:20 AM, Christos KK Loverdos
wrote:
> Let's keep in mind, from the operations perspective, that an application deployed to a server will most certainly not be given the flexibility to download needed jars but they will have to be at some local CLASSPATH (normally near the application).

That some environments face constraints which others do not is pretty
much the defining principle of the work. I'm not doing anything nutty
like creating new network dependencies. You'll still be able to use
classpaths exactly as you do now.

> https://github.com/loverdos/streamresource

Thanks, I will take a look.

Copyright © 2012 École Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland