information about compiler internals

Hello NG,

currently I am trying to become acquainted with the scala compiler and
it's plugin system.

I need this information as I want to develop a language extension which
- introduce some new keywords and extend the syntax
- integrate some semantic checks
- integrate a transformation for some existing/new AST-Nodes

So I am searching for information of how to
- adapt the parsing
- create new AST nodes
- traverses the AST in order to do semantic checks
- transforms the AST

If I understand right the two latter points are done by using a
Sub-PluginCompinont and a Traverser respectively mix in the trait
Transform.
But I have found no information about how to adapt the syntax or how
to adapt the parsing so that new AST nodes are generated.

The only documents/examples about the compiler and its plugins , I
have found, are
http://lampsvn.epfl.ch/svn-repos/scala/scala/branches/devel-base-2.8.0/S...
and the example
http://www.scala-lang.org/node/140
http://lampsvn.epfl.ch/trac/scala/browser/scala/trunk/docs/examples/plug...

Is there any more documentation/information or can anybody give me
more detailed information about the compiler (plugins)?

Thanks in advance.

Cheers,
Thomas

Re: information about compiler internals

Added by Thomas Pawlitzki on 2009-01-15, 11:27

Hi again,

> Is there any more documentation/information or can anybody give me
> more detailed information about the compiler (plugins)?

Some information about the compiler I found in
http://infoscience.epfl.ch/record/64423/files/ScalableComponent.pdf

Cheers,
Thomas

Re: information about compiler internals

Added by Antonio Cunei on 2009-01-15, 15:07

Thomas,

There is little documentation available in general on the compiler, but you
may find the following useful: these are two recordings of code
walk-through sessions that Martin did some time back.

http://www.scala-lang.org/sites/default/files/martin_ordersky_scala_inte...
http://www.scala-lang.org/sites/default/files/martin_ordersky_scala_inte...

The video quality is not excellent, but the information should be of use.
Hope this helps!
Toni

Thomas Pawlitzki wrote:
> Hi again,
>
>> Is there any more documentation/information or can anybody give me
>> more detailed information about the compiler (plugins)?
>
> Some information about the compiler I found in
> http://infoscience.epfl.ch/record/64423/files/ScalableComponent.pdf
>
> Cheers,
> Thomas
>

Re: information about compiler internals

Added by Thomas Pawlitzki on 2009-01-16, 08:07

Hi Toni,

> There is little documentation available in general on the compiler, but you
> may find the following useful: these are two recordings of code walk-through
> sessions that Martin did some time back.
>
> http://www.scala-lang.org/sites/default/files/martin_ordersky_scala_inte...
> http://www.scala-lang.org/sites/default/files/martin_ordersky_scala_inte...

Thank you for this links.
I will have a look at them.

Cheers,
Thomas

Re: information about compiler internals

Added by daniel.kroeni on 2009-01-16, 08:17
Hi Thomas,
I found another link: http://lamp.epfl.ch/~emir/bqbase/2005/06/02/nscTutorial.html

Cheers, Daniel

On Fri, Jan 16, 2009 at 8:04 AM, Thomas Pawlitzki <thomas [dot] pawlitzki [at] gmail [dot] com> wrote:
Hi Toni,

> There is little documentation available in general on the compiler, but you
> may find the following useful: these are two recordings of code walk-through
> sessions that Martin did some time back.
>
> http://www.scala-lang.org/sites/default/files/martin_ordersky_scala_internals_2008-10-29.avi
> http://www.scala-lang.org/sites/default/files/martin_ordersky_scala_internals_2008-11-05.avi

Thank you for this links.
I will have a look at them.

Cheers,
Thomas

Re: information about compiler internals

Added by Thomas Pawlitzki on 2009-01-22, 11:47

Hello Daniel,
hello all,

> I found another link:
> http://lamp.epfl.ch/~emir/bqbase/2005/06/02/nscTutorial.html

Thanks for this link.
This text contains really useful information.

I think that after reading the texts and looking at the source code I
have a dim idea of the compiler internals.

But I still miss a possibility to adapt/extend the syntax. I found the
location where the parsing is done:
compiler/scala/tools/nsc/ast/parser/Parsers
So I still don't know what to do if I want to add (for instance) a new
TopStatSeq.
I think I have to
- register a new keyword
- register a new token to the new keyword (ScannerConfiguration)
- define a new match in the Parser.topStatSeq() method to create a new AST-Node
- do my own checks and transforms with my new AST-Node

But how can I do this in a compiler plugin without hacking the
compiler sources? I think here is my gap in my moderate scala
knowledge.
I read about implicit declarations [1] but I guess this that this
would not solve my problem as with implicit I can not override
existing methods, right?

Perhaps there is someone here who can give me some concrete
information about my problem.

Thanks in advance.

Cheers,
Thomas

[1] http://www.scala-lang.org/node/114

Re: information about compiler internals

Added by washburn on 2009-01-22, 11:57
Thomas Pawlitzki wrote:
a4df10ce0901220243k5b2cbe68u84f2c5c06fc2564e [at] mail [dot] gmail [dot] com" type="cite"> 
But I still miss a possibility to adapt/extend the syntax. I found the
location where the parsing is done:
compiler/scala/tools/nsc/ast/parser/Parsers
So I still don't know what to do if I want to add (for instance) a new
TopStatSeq.
I think I have to
 - register a new keyword
 - register a new token to the new keyword (ScannerConfiguration)
 - define a new match in the Parser.topStatSeq() method to create a new AST-Node
 - do my own checks and transforms with my new AST-Node

But how can I do this in a compiler plugin without hacking the
compiler sources?
It is not presently possible to write a compiler plugin that will extend the syntax of the language.   It may be possible in the future to replace the parser phase, but for performance reasons I doubt the default parser will ever provide hooks for the kind of extension you want.  I recall someone writing a plugin that sort of works around this using XML for their syntax, but I cannot remember enough about it to find it in the mailing list archives.

-- 
[ Geoff Washburn 
| geoffrey [dot] washburn [at] epfl [dot] ch 
| http://lamp.epfl.ch/~washburn/ ]

Re: information about compiler internals

Added by Jorge Ortiz on 2009-01-22, 12:57
I believe this is the thread you're talking about:

http://www.nabble.com/-scala--Extending-Scala-to15856708.html#a15856708

--j

On Thu, Jan 22, 2009 at 2:53 AM, Geoffrey Alan Washburn <geoffrey [dot] washburn [at] epfl [dot] ch> wrote:
Thomas Pawlitzki wrote:
But I still miss a possibility to adapt/extend the syntax. I found the
location where the parsing is done:
compiler/scala/tools/nsc/ast/parser/Parsers
So I still don't know what to do if I want to add (for instance) a new
TopStatSeq.
I think I have to
 - register a new keyword
 - register a new token to the new keyword (ScannerConfiguration)
 - define a new match in the Parser.topStatSeq() method to create a new AST-Node
 - do my own checks and transforms with my new AST-Node

But how can I do this in a compiler plugin without hacking the
compiler sources?
It is not presently possible to write a compiler plugin that will extend the syntax of the language.   It may be possible in the future to replace the parser phase, but for performance reasons I doubt the default parser will ever provide hooks for the kind of extension you want.  I recall someone writing a plugin that sort of works around this using XML for their syntax, but I cannot remember enough about it to find it in the mailing list archives.

-- 
[ Geoff Washburn 
| geoffrey [dot] washburn [at] epfl [dot] ch 
| http://lamp.epfl.ch/~washburn/ ]

Copyright © 2013 École Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland