I need to start writing the second part of the book soon. This one is supposed to take a DSL implementation through all the interesting stages that I would like to cover. However, I am not sure yet what the subject of the DSL will be.
I need something that has enough scope to last for about a hundred pages, complex enough to expose usual problem when writing DSL and not tied to a specific domain so strongly that it would be hard to outsiders to grasp.

I am also interested in knowing what kinds of patterns and problems you would like me to cover.

My current line of thinking is to build several DSL around the backend of an online store. That domain should be fairly familiar to all, and it is rich enough to offer a lot of things to discuss. It is also a good place to discuss several types of DSL.

I was thinking of the following DSL:

• Message routing / dispatch DSL
• Order processing DSL
• Then we will extend that to be a generic rule engine DSL
• Perhaps creating a testing DSL for those DSL or for the domain itself

The main problem that I have now is that building those DSL is very simple, I don't think that it would be enough to cover everything the we need to build DSL. In that light, I am looking for either more ideas or challenges on the use of the above mentioned DSL.

This is just a quick note for public review, you are probably aware that I am doing deployments by doing a "svn up && build". I am now thinking about how we can apply the same idea to deploying DSL. This ensures, at the very least, that our DSL are under source control. But that has led me to another thought, if we are enforcing SCM for the DSL, why not enforce unit testing as well?

Part of the loading process of a DSL can be loading the DSL and its unit tests, executing the unit tests and only accepting them if they all passed. Failure in the unit tests or lack of unit tests would cause the DSL load process to fail.

Thoughts?

Now, before Greg hurls a modopt on me, I want to be clear that this isn't the same thing that Spec# is doing. But it is a very cool way to specify constraints that must always be valid when a method exists.

Here is the code:

[AttributeUsage(AttributeTargets.Class)]
class EnsureAttribute(AbstractAstAttribute):
	
	expr as Expression
	
	def constructor(expr as Expression):
		self.expr = expr
		
	def Apply(target as Node):
		type as ClassDefinition = target
		for member in type.Members:
			method = member as Method
			continue if method is null
			block = method.Body
			method.Body = [|
				block:
					try:
						$block
					ensure:
						assert $expr
			|].Block

And the usage:

[ensure(name is not null)]
class Customer:
	name as string
		
	def constructor(name as string):
		self.name = name
	
	def SetName(newName as string):
		name = newName

Now, any attempt to set the name to null will cause an assertion exception. This technique is quite powerful, and very easy to use. A few years ago I wrote a design by contract implementation for boo that was far more ambitious (handling inheritance, etc). I remember it being much more complicated, and while things like quasi quotation do make it easier, it is not that big a change.

I think that mostly it is the way I write code now, striving to simplicity is something that I am trying to apply recently, and I think it works.

A meta-method is a shortcut into the compiler; it is a method that accepts AST nodes[1] and returns an AST node.

Let us implement this very simple scenario, the assert statement. Now, because Boo already has that, we will use “verify” as the method name. Here is the full method implementation:

[Meta]
static def verify(expr as Expression):
	return [|
		unless $expr:
			raise $(expr.ToCodeString())
	|]

We are using quasi quotation to save us typing. This is a static method decorated with the [Meta] attribute, and accepting an AST expression. This is all you need in order to create a meta-method. When you have a meta-method, you can call it, like this:

verify 1 == 2

Now the interesting tidbit happens. When the compiler sees a call to a meta-method, it doesn’t emit the code to call this method at runtime. Instead, during compilation, the meta-method is executed. We pass it the AST of the arguments of the method code (including anonymous blocks), and then we replace this method call with the result of calling the meta-method.

It is important that you’ll understand that after compilation, where in the code we had this:

verify 1 == 2

The actual compiled bits will have this:

unless 1 == 2:
	raise “1 == 2”

Please go over it again, to make sure that you understand how it works. It is similar to text substitution macros in C and C++, but this is actual code that is running during compilation that gets to output any code that it wants back into the compilation process, not mere text preprocessing. In addition to that, we are dealing directly with the compiler’s AST, not just copying lines of text.

This seems to be something that a lot of people have a hard time grasping. The compiler will ask you, at compilation time, what kind of transformation you want to do on the code. It will then take the result of the transformation (the method return value) and put it where the method call used to be.

The Boo code above can also be translated to the following C#, which is a bit more explicit about what is going on:

[Meta]
public static UnlessStatement verify(Expression expr)
{
	UnlessStatement unlessS = new UnlessStatement();
	unless.Condition = Expression.Lift(expr);
	RaiseStatement raise = new RaiseStatement();
	raise.Exception = Expression.Lift(expr.ToCodeString());
	unless.Statements.Add(raise);
	return unless;
}

Both have the same exact semantics.

We have actually used meta-methods before, when we implemented the “when” keyword for the scheduling DSL. Meta-methods are used in DSL quite often. They are usually the first step that we need to take into the compiler when we run into the limits of what the compiler gives us out of the box.

[1] An ast node is generic term to all the types that compose the abstract syntax tree of the language.

Then it must be an IQuackFu.

IQuackFu is Boo’s answer to the Method Missing / Message Not Understood from dynamic languages. Since Boo is a statically typed language[1], and since method missing is such a nice concept to have, we use this special interface to introduce this capability.

You are probably confused, because I didn’t even explain what method missing is. Let us go back and look at an example, shall we? We want to look at the following xml:

<People> 	<Person> 		<FirstName>John</FirstName> 	</Person> 	<Person> 		<FirstName>Jane</FirstName> 	</Person> </People> 

Now we want to display the first names in the xml. We can do it using XPath, but the amount of code required makes this awkward. We can also generate some sort of strongly typed wrapper around it, assuming that we have a schema for this, we can use a tool to generate the schema, if we don’t have it already…

Doesn’t it look like a lot of work? We can also do this:

doc = XmlObject(xmlDocument.DocumentElement)
for person as XmlObject in doc.Person:
	print person.FirstName

But we are using a generic object here, how can this work? This works because we intercept the calls to the object and decide how to answer them at runtime. This is the meaning of the term “method missing”. We “catch” the method missing and decide to do something smart about it (like returning the data from the xml document).

At least, this is how it works in dynamic languages. For a statically typed language, the situation is a bit different; all method calls must be known at compile time. That is why Boo introduced the idea of IQuackFu. Let us check the implementation of XmlObject first, and then we will discuss how it works:

class XmlObject(IQuackFu):
	_element as XmlElement 

	def constructor(element as XmlElement):
		_element = element 

	def QuackInvoke(name as string, args as (object)) as object:
		pass # ignored 

	def QuackSet(name as string, parameters as (object), value) as object:
		pass # ignored 

	def QuackGet(name as string, parameters as (object)) as object:
		elements = _element.SelectNodes(name)
		if elements is not null:
			return XmlObject(elements[0]) if elements.Count == 1
			return XmlObject(e) for e as XmlElement in elements 

	override def ToString():
		return _element.InnerText

We didn’t implement the QuackInvoke and QuackSet, because they are not relevant to the example at hand, I think that QuackGet will make the point. Now, just to complete the picture, we will write the first code sample, the use of XmlObject, as the compiler will output it.

doc = XmlObject(xmlDocument)
for person as XmlObject in doc.QuackGet(“Person”):
	print person.QuackGet(“FirstName”)

The way it works, when the compiler finds that it can’t resolve a method (or a property) in the usual way, it then check if the type implements the IQuackFu interface. If it does implement IQuackFu, it translates the method call into the equivalent method call.

The example of the Xml Object is a really tiny one of the possibilities. Convention based methods are an interesting idea[2] that is widely used in Ruby. Here is an example that should be immediately familiar to anyone who dabbled in Rails’ ActiveRecord:

user as User = Users.FindByNameAndPassword(“foo”, “bar”)

Which will be translated by the compiler to:

user as User = Users.QuackInvoke(“FindByNameAndPassword”, “foo”, “bar”)

The Users’ QuackInvoke method will parse the “method name” and issue a query by name and password.

You can do some very interesting things with IQuackFu...

[1] Well, it is statically typed unless you explicitly tell the compiler that you want late bound semantics. Aside from working against IDispatch COM interfaces, I have rarely found that ability useful. One case I did find it useful, however, was when I wanted to introduce Context Parameters, which we will discuss in a few pages.

[2] For the adventurous sorts, you can also do something called Lazy Methods, in which you generate a method if and only if it is being called. This is an interesting exercise in extending the compiler, but for all intents and purposes, IQuackFu answers this need very well.

This quote has me floored:

Well, I'm an asp.net developer, not really a web developer. It is like web development with training wheels, only the training wheels are really heavy, uneven, and make riding the bike harder

Just a nod toward the people that cling to static typing with both hands, their teeth and the tail:

RouteTable.Routes.Add(new Route
{
 Url = “admin/[controller]/[action]“,
 Defaults = new
 {
  Controller = “Admin“,
  Acton = “Index”
 },
 Validation = new
 {
  Conrtoller = “Admin|Users|Categories”
 },
 RouteHandler = typeof(MvcRouteHandler)
});

Now, instead of abusing the language to get this, can we get first class support for this things?

Rob Conery asks how the MS MVC platform should handle XSS attacks. In general, I feel that frameworks should do their best to ensure that to be secure by default. This means that I feel that by default, you should encode everything that comes from the user to the app. People seems to think that encoding inbound data will litter your DB with encoded text that isn’t searchable and consumable by other applications.

That may be the case, but consider, what exactly is getting encoded? Assuming that this is not a field that require rich text editing, what are we likely to have there?

Text, normal text, text that can roundtrip through HTML encoding without modifications.

HTML style text in most of those form fields are actually rare. And if you need to have some form of control over it, you can always handle the decoding yourself. Safe by default is a good approach. In fact, I have a project that uses just this approach, and it is working wonderfully well.

Another approach for that would be to make outputting HTML encoded strings very easy. In fact, it should be so easy that it would be the default approach for output strings.

Here, the <%= %> syntax fails. It translate directly to Response.Write(), which means that you have to take an extra step to get secured output. I would suggest changing, for MS MVC, the output of <%= %> to output HTML encoded strings, and provide a secondary way to output raw text to the user.

In MonoRail, Damien Guard has been responsible for pushing us in this direction. He had pointed out several places where MonoRail was not secure by default. As a direct result of Damien's suggestions, Brail has gotten the !{post.Author} syntax, which does HTML encoding. This is now considered the best practice for output data, as well as my own default approach.

Due to backward comparability reasons, I kept the following syntax valid: ${post.Author}, mainly because it is useful for doing things like output HTML directly, such as in: ${Form.HiddenField("user.id")}. For the same reason, we cannot automatically encode everything by default, which is controversial, but very useful.

Regardless, having a very easy way ( !{post.Author} ) to do things in a secure fashion is a plus. I would strongly suggest that the MS MVC team would do the same. Not a "best practice", not "suggested usage", simply force it by default (and allow easy way out when needed).

I have been getting a lot of questions about how I make the diagrams for the blog.

I got some very strange suggestions from people, from having a full blown art department dedicated to producing those to drawing the diagrams on physical paper and then taking pictures of that on a wooden table.

The truth is far more boring, I am afraid.

I generally use the following tools to produce the diagrams:

• Google Image Search
• Power Point
• MS Paint
• Visual Studio

I use visual studio's class diagram designer to create the class diagrams, usually from empty projects, not the real ones. Then I copy the image to power point, where I do some mixing & matching.

Power Point is really nice in this regard, because it offers a rich set of effects that even a graphical dummy like me can use effectively.

I use Google Image Search to find relevant images, and then drag them into Power Point as well, making the same style of effects there. Then, if needed, I move it to MS Paint for final processing, and from there, to the blog.

This approach works well for someone that needs three tries to draw a straight line using a ruler, it is also something that doesn't take very long, which is also important.

So, this issue cause some head scratching today. We are using WIndsor's Automatic Transaction Management with NHibernate's flush-on-commit option, so if a transaction doesn't commit, nothing is written to the database.

Anyway, this is a story about refactoring, and what it showed us. We performed the following refactoring:

Some things that is important to understand, the LoginController is decorated with [Transactional], and there is a [Transaction] attribute on CreateUserLoggedInAuditRecord.

When it was on the controller, it just worked. When we moved it to its own class, it didn't work. To be rather more exact, it worked, it just never committed the transaction. That was weird. After some head scratching I found out that I forgot to put [Transactional] on the UsageRegistrationImpl. With a small smile of geeky  triumph, I run the code again. It didn't save.

That was really worrying, and I had no idea what was going on. Since this is rarely popular, I repeatedly run the code, hoping that something would turn up and that no one would pull the old quote about insanity.

After a few repetitions, I suddenly saw the light.

It had to do where I placed the pointcut. A pointcut, in AOP terms, is where the AOP can interfere with the running code. Let us take a look at how it worked when we used the LoginController directly. Because we (well, the transaction facility) asked the container to create an interceptor for it, we got the following classes at runtime:

The login controller is the original class, the login controller proxy was generated at runtime, and any invocation of any of its methods would fire the transaction interceptor, so it would get a chance to create/rollback/commit a transaction if needed. Since those methods are virtual, this means that even if I am calling methods on the same class, they will be intercepted correctly.

Now, when I moved to the interface + implementing class, we have a different behavior. Now, we use the interface pointcuts in order to inject behavior, it looks like this:

Windsor will create a proxy interface implementation that would call the AOP interceptors and will forward to the UsageRegistrationImpl.

The problem was with the RegisterUserLoggedIn method. It was similar to this:

public virtual void RegisterUserLoggedIn(string username)
{
	// do other things
	CreateUserLoggedInAuditRecord(username);
}

[Transaction]
public virtual void CreateUserLoggedInAuditRecord(string username)
{
	//do database stuff
}

Given the story so far, you can obviously see the problem. When we call the CreateUserLoggedInAuditRecord() method, we call it from the UsageRegistrationImpl class, so we never pass through any of the pointcuts.

When we used the method from the controller directly, we made a virtual method call, which was intercepted, but since in this case, we were using the interface as our pointcut, this simply by passed the whole thing.

That was an interesting lesson, and one that I'll need to remember for the future.