Let me start by stating the scenario. hibernatingrhinos.com is down. The server is compromised, dead, whatever.

I don't know what actually happened to is, but it is no longer bootable.

I don't have complete backups. That is totally and completely my fault. I am an idiot. That is not the point of this post. I should have had backups, and ensured that they work correctly, no arguments here.

What I want to talk about now is the support experience that I got from VPSLand.

We can start with this bad news:

Just to give you some background, I had two "chats" with the "support". In the first it became clear that the "complimentary weekly backups" that they offer are a myth and doesn't really exists. They can't create a new VPS instance and hook the current virtual disk to the new instance, so I can get the data off of the old machine.

The only solution that they offer was to rewipe the entire machine. Losing all the data in it.

Here is my second chat transcript.

VPSLAND.com Sales: Oren, I already told you that there is no back up available for your VPS and it's not possible for us to take back up for your VPS
Oren Eini: okay, then don't
Oren Eini: I need the server itself back up
VPSLAND.com Sales: We want this in written
VPSLAND.com Sales: Please update your ticket again
Oren Eini: Can I get the actual image?
Oren Eini: The VM file?
VPSLAND.com Sales: No
Oren Eini: why not?
VPSLAND.com Sales: Don't you understand, I already told you that your VPS is not up and it's not possible to provide you any data
VPSLAND.com Sales: Any type of data
Oren Eini: I want the VM FILE
Oren Eini: Not any data from it
Oren Eini: The ACTAUL FILE
Oren Eini: The VM is sitting on a file.
Oren Eini: If I have that, _I_ can get the data
Oren Eini: I don't want anything from the machine.
Oren Eini: I want the file that _is_ the machine
VPSLAND.com Sales: Can I have the path for VM ?
VPSLAND.com Sales: VM FILE *
Oren Eini: I don't know. It is on your system
VPSLAND.com Sales: Well Oren, we are not able to mount your VPS on main node which is the reason. We are not able to take back up of your VPS. So Its not possible to provide you VM FILES
Oren Eini: I want the file that you are mounting
Oren Eini: Not the VM itself
VPSLAND.com Sales: Have read my last message carefully
VPSLAND.com Sales: ?
Oren Eini: Yes.
Oren Eini: You are trying to mount the VPS
Oren Eini: You are failing.
Oren Eini: I want the VPS file
Oren Eini: The thing that you are trying to mount
VPSLAND.com Sales: Yes, The VM files are also mounted under root directory
VPSLAND.com Sales: Which is not mounting
VPSLAND.com Sales: So its not possible for us to provide you any VM FILES
Oren Eini: Sorry?
Oren Eini: what root directory are you talking about?
VPSLAND.com Sales: Is there anything else I can help you with ?
Oren Eini: I want the file that you are trying to mount.
VPSLAND.com Sales: You do not understand that, right ?
Oren Eini: The .vhd file
VPSLAND.com Sales: So just listen to me, we are not able to provide you any type of VM FILES
VPSLAND.com Sales: Would you like to reinstall your VPS or NOT ?
Oren Eini: Let me go back to the beginning
Oren Eini: There is the VPS, right?
VPSLAND.com Sales: Ae are not able to provide you any type of VM FILES
VPSLAND.com Sales: Ae = We*
Oren Eini: Please bear with me.
Oren Eini: There is the VPS
Oren Eini: You are trying to mount that on the VM Server
Oren Eini: The VPS is a file
VPSLAND.com Sales: Read above chat once again
Oren Eini: I want that file
Oren Eini: It is not a file inside the VPS.
Oren Eini: It is the file that _is_ the VPS.
Oren Eini: It is the vhd file that you are trying to mount
VPSLAND.com Sales: I understand that, and I apologize for the inconvenience. The fact of the matter is that you are talking to the Sales department right now and unfortunately we don't have any control over such technical issues.
Oren Eini: Okay, than can I get a SUPPORT person to talk to me?
Oren Eini: I am talking to support.vpsland.com
Oren Eini: Not sales.vpsland.com
Oren Eini: I need a TECHNICAL person to solve this issue
VPSLAND.com Sales: They do not have Live Chat support
Oren Eini: Do they have a phone?
VPSLAND.com Sales: Contact them through the email to support@vpsland.com
Oren Eini: Can they answer my emails in a reasonable time frame?
VPSLAND.com Sales: We do not have phone support
Oren Eini: 12 hours between emails is not acceptable
VPSLAND.com Sales: Update to your ticket ULX-445304
Oren Eini: Again, 12 hours between responses are NOT helpful
Oren Eini: If there is support stuff, I would like to communicate with them.
Oren Eini: ASAP
VPSLAND.com Sales: No, they are in the Support Department
Oren Eini: That is lovely.
Oren Eini: Please contact them and tell them that I would like to get a tech people respond to this ticket in a reasonable time frame
VPSLAND.com Sales: Sure, I will update them for you
Oren Eini: Thank you, bye

As of now, for a support issue that is marked as critical, I still don't have any response from their tech support. This is about 30 minutes after the chat, by the way.

Now, this isn't an issue of violating SLA, I don't have one with them for support, I find this deeply disturbing in term of customer service.

I won't be doing business with them again.

Previously on the Multi Tenancy series:

• Multi Tenancy - Definition and scenario.
• Reviewing Litware HR - P&P Multi Tenancy Sample.
• Multi Tenancy - The Physical Data Model
• Multi Tenancy - Extensible Data Model
• Multi Tenancy - Extensible Behaviors
• Multi Tenancy - Development Structure
• Multi Tenancy - Where do you put variability?
• Multi Tenancy - Scriptability and DSL
• Multi Tenancy - multi tenant apps and frameworks
• Multi Tenancy - Keep It Simple, REALLY Simple
• Multi Tenancy - User Interface

Yes, I said that the UI post was the last one before I got to the actual implementation details, but I am having a Skype chat now about the subject, and it is important.

The approach that I have outlined so far is focused on building a system that supports variability at all levels, from the database to the UI, through the entities, services and external integration points. The sweet spot for that is if you have a multi tenant application where a tenant want to have high degree of control on the way the application works.

This high degree of control often means that the tenant desires to change specific parts of the applications in ways that make sense to this tenant, but may not make sense to others. A good example would be payroll system. Here, each tenant has their own processes for how to handle this, and they tend to maintain a high degree of control over that.

A good example from the other side would be something like Subtext. Here, we have a multi tenant application that is essentially the same for all the tenants. The behavior of the system is essentially the same for everyone.

That doesn't means that if you need variability in the application, you should immediately jump to the approach that I outline here. If you have a limited number of variability points, and limited number of variability options (that is, you have X amount of strategies that you can configure for each tenant), you might want to consider that approach.

I would probably avoid that anyway, but it is something to take into account. Some people would feel that a few configuration options makes their life easier than a composite, contextual, application.

My rule of the thumb is that if you have both data extensibility and process variability you need to use the composite approach that I have spoken off. Even if you don't have data extensibility, but "merely" lot of variability in the process across the tenants that cannot be standardize, you will want to take the composite approach.

Previously on the Multi Tenancy series:

• Multi Tenancy - Definition and scenario.
• Reviewing Litware HR - P&P Multi Tenancy Sample.
• Multi Tenancy - The Physical Data Model
• Multi Tenancy - Extensible Data Model
• Multi Tenancy - Extensible Behaviors
• Multi Tenancy - Development Structure
• Multi Tenancy - Where do you put variability?
• Multi Tenancy - Scriptability and DSL
• Multi Tenancy - multi tenant apps and frameworks
• Multi Tenancy - Keep It Simple, REALLY Simple

Note, this is nearly the end as far as the theory and the constraints of Multi Tenancy applications goes, the next posts in this series are high level design, trying to touch all the points that were brought up so far, and then a big surprise...

So far, we have spoken only on the backend stuff, we haven't touched on how we are going to deal with the UI at all. We need to think about what kind of UI changes we need to handle.

Skinning / Templating

This include everything that is not actually functional to the application. That is, the customer wants their own logo instead of the default one, they may want different colors, etc. This is what I call templating or skinning.

Semi localization

I don't have a good name for this, but broadly, this include things like "our date format is", "our time format is", "we always enter FirstName, LastName", etc. This is mainly formatting issues, most of them are locale oriented, but there are a lot of similar issues that aren't handled by the locale.

As a simple example, in Israel, it is very rare for someone to have a middle name, so an application build to display the user's name as: {FirstName} {MiddleInitial}. {LastName} would look ridiculous ("Ayende . Rahien").

Different Presentation

One client may want something to be a report, another may want a tree, and yet another as a graph (please, don't ask). Very often, the issue of field ordering comes into play. For that matter, the customer may have no interest in some fields, and want to hide them. (see: middle name in Israel example).

Handling additional responsibilities

This goes beyond just showing / hiding fields, here we need to handle the extra fields that the tenant has defined. This can be interestingly complex, especially when you have thinks like related fields. This is interesting because we want to handle this without having to modify the core page.

Crazy Tenant Syndrome

I am not sure how to call this one, but this is usually the case when you have a big client that wants to have things their ways. In the old system, all text boxes were UPPER CAPS, so they have to be in the new system as well. They only sell to France and Germany, so they must never see the full countries list.

I am sure that you know the type, if not the actual situation.

Now, how do we handle all of this?

Again, divide and conquer is the only feasible strategy. We split the application into separate pieces, and deal with of them individually. The templating issue is fairly obvious, we will maintain a css / layout for each tenant. By the same token we can handle each of the other issues by putting them in their individual boxes, which we can then shuffle at will.

And yes, we will touch on the next post in this series.

I look at a bit of code that dealt with traversing expression an expression tree, using recursion, of course. The edge condition immediate popped to mind was unbounded expression. I decided to see if I can kill the compiler using this. Why? Because.

The first thing to do is to find out how deep a stack we usually need. I wrote this simple test:

class Program
{
	static void Main(string[] args)
	{
		Recursive(1);
	}

	static void Recursive(int i)
	{
		Console.WriteLine(i);
		Recursive(i+1);
	}
}

The last result was: 79994

Obviously this change based on how much stack space each function takes, but it is a good number to go with. I started with this code:

class Program
{
	static void Main(string[] args)
	{
		using(var fw = File.CreateText("text.txt"))
		{
			for (int i = 0; i < 80000; i++)
			{
				fw.Write(" a > "+i +" && ");
				if(i%10==0)
					fw.WriteLine();
			}
		}
	}
}

I then took the file ( slightly over 1 MB in size) and pasted the content to Visual Studio.

That was a mistake:

Okay, I can deal with this, let us try a different approach:

class Program
{
	static void Main(string[] args)
	{
		using(var fw = File.CreateText("text.cs"))
		{
			fw.WriteLine("public class Program {");
			fw.WriteLine("	static void Main(string[] args) {");
			fw.WriteLine("		var a = -1;");
			fw.Write    ("		var test = ");
			for (int i = 0; i < 80000; i++)
			{
				fw.Write(" a > "+i +" && ");
				if(i%10==0)
					fw.WriteLine();
			}
			fw.WriteLine(" a < 0;");
			fw.WriteLine("System.Console.WriteLine(test);");
			fw.WriteLine("	}");
			fw.WriteLine("}");

		}
	}
}

Trying to compile that produces:

fatal error CS1647: An expression is too long or complex to compile near ''

The help for CS1647 is:

There was a stack overflow in the compiler processing your code. To resolve this error, simplify your code. If your code is valid, contact Product Support.

The is valid, I guess, just not really reasonable. What is scary is that this is something that was added for 2.0, so at the 1.0 days, someone actually run into this issue.

Some experimentation showed that the C# compiler can handle expressions composed of 23,553 nodes. Now it is the time to get to the next stage, now the code is this:

class Program
{
	static void Main(string[] args)
	{
		using(var fw = File.CreateText("text.cs"))
		{
			fw.WriteLine("using System;");
			fw.WriteLine("using System.Linq.Expressions;");
			fw.WriteLine("public class Program {");
			fw.WriteLine("	static void Main(string[] args) {");
			fw.Write    ("		Expression<Predicate<int>> test = (a) => ");
			for (int i = 0; i < 11700; i++)
			{
				fw.Write(" a > "+i +" && ");
				if(i%10==0)
					fw.WriteLine();
			}
			fw.WriteLine(" a < 0;");
			fw.WriteLine("System.Console.WriteLine(test);");
			fw.WriteLine("	}");
			fw.WriteLine("}");

		}
	}
}

Note that I had to dramatically simplify the expression. Before it handled 23 thousands and change, but now it chokes on merely 12 thousands.

What is really surprising is that after compiling the code, it is running and seems to do the expected thing. Amazing.

Anyway, here is a completely useless post, but now I know that the C# compiler has well defined behavior for stack overflows. :-)

Previously on the Multi Tenancy series:

• Multi Tenancy - Definition and scenario.
• Reviewing Litware HR - P&P Multi Tenancy Sample.
• Multi Tenancy - The Physical Data Model
• Multi Tenancy - Extensible Data Model
• Multi Tenancy - Extensible Behaviors
• Multi Tenancy - Development Structure
• Multi Tenancy - Where do you put variability?
• Multi Tenancy - Scriptability and DSL
• Multi Tenancy - multi tenant apps and frameworks

One of the most common approaches for handling multi tenancy is composed of the following core ideas: Dynamic and Configurable.

You build the application to be dynamic, so you can change the entities layout, the UI look and feel, modify business logic using configuration, etc.

This is wrong!

This is wrong mainly because it is so bloody complex as an approach. Let us take the example of the UI, dynamic UI is fairly simple to start with. However, when you need to start considering things like validation, error handling, different field types, and so on, you get to realize that this is not really as simple as it seems.

Don't get me started on the 3 tables DB (tblEntities, tblFields, tblFieldValues) that get you so much dynamic it is dynamite.

The main issue is that trying to build something using this approach is just bloody hard. You have to keep the dynamicism of the application at mind at all point. Most of the time, you wouldn't even notice that your choice prevent you from applying the dynamicism in the app, but you are still paying the price for using this.

In many ways, it is similar to having VB.Net Option Implicit (If I got it right, the one where it doesn't give you a compilation error if you call non existing method, only at runtime). In fact, this is a great corollary. The VB.Net "dynamic" stuff is so limited that it is useless in most scenarios. Compare what you can do with a dynamic language like Python or Ruby vs. what you can do with VB.Net. There is very little value of having this option in VB.Net. It is usually a mine, not a feature. (Yeah, okay, it is really good feature if you are working with IDispatch, I know. Work with me here).

By the same token adding dynamic features to your application, runtime configurable and so on, will tend to be a mine more often than help.

My approach is quite different. Instead of trying to add configurability to the application, or adding dynamic behaviors, we can choose a static approach, one that allows use to make the best use of the tools and approaches that we have.

How do we get the variability that we need?

Oh, that is easy. Seams.

Each individual piece has no concept of dynamic / configuration / fubar. It is self contained and cohesive unit that can be combined with other pieces.

If I want to modify the behavior of the system, I can just plug in a new piece, and the entire machine keep on chugging along, without really noticing that the new Widgetometor is bright orange and flashing in Morse code.

Using this approach, we don't have to pay the price for using dynamic, configurable to the Nth degree. We can write the application as we would always will, and when variability calls...

Just plug the new stuff in.

Keep is simple, and keep it simple throughout!

Here are a few interesting things that I found about .Net 3.5 Service Pack 1:

1. Serialization hangs or throws an OutOfMemoryException with static delegate and ISerializable on 3.5 SP1
2. ExecutionEngineException with ParameterInfo.GetOptionalCustomModifiers and GetRequiredCustomModifiers on 3.5 SP1
3. .NET 3.5 SP1 breaks use of WPF under IIS
4. .NET 3.5 SP1 seems to break .NET 2.0 applications with assembly loading error.
5. .NET Framework 3.5 SP1 breaks type verification

I just googled for "3.5 SP1" on connect.microsoft.com, and took only the verified (by Microsoft) items for the SP1 RTM. There are more, but I didn't feel like spending a lot of time digging there.

I don't remember previous service packs having regression bugs. Certainly not regression bugs as big as these. Number 2 is the one responsible for breaking Rhino Mocks, by the way.

Those are all regressions. That is, it used to work in previous version, now it doesn't.

Not happy at all.

Here is a challenge to Microsoft, Fix This. And fix this in a way that ensure that everyone get the fix. If this means SP1 Refresh, great. But fix this in a way that means that I don't have to answer "you need to call PSS and get KB32423 before you can run" for the next two years.

And fix this from Connect, if you ever want Connect to be useful for something.

I am not really sure what to think about this book. On the one hand, it is entertaining. On the other hand, it seems to be filled with... propaganda?

I am not sure that this is an accurate statement, but I am on 2.5 hours, and so far it has been interesting. The description of the handling of a crisis and analysis of how it could be prevented is fascinating in itself. But the last twenty minutes or so seems to be focused really heavily on showing off the US healthcare system and interesting political views.

The start of the book is wonderful, as I already mentioned, and I am going to hear it all, because it is still entertaining, but I really hope it would get to some more interesting bits soon.

Two interesting observations, the story begins with a really interesting part, which establish the credentials of the protagonist and a certain style. In particular, Do. The. Math. is an expression that is often used to compare farming techniques (hand labor vs. mechanical). I have just realized that it is used in a more controversial form. It uses the trust established for this term earlier to support this statement.

I am not sure that I agree or disagree about what he is saying. I don't have all the facts, but again, it is interesting, although discomforting. I want to listen to a plot, not a lecture.

A few days ago I posted about Ugly Linq. Ever since then, I kept thinking about how ugly it is to handle this by hand. Suddenly, it hit me that I don't have to do it that way.

Boo already has the facilities to take a compiler AST and translate that into the code that would recreate this AST. In particular, this makes the code we previously had to write to this:

public class ConditionMacro : AbstractAstMacro
{
	public override Statement Expand(MacroStatement macro)
	{
		Expression serialize = new CodeSerializer().Serialize(macro.Arguments[0]);
		var body = new Block();
		body.Statements.Add(new ReturnStatement(macro.Arguments[0]));
		return new ExpressionStatement(
			new MethodInvocationExpression(
					AstUtil.CreateReferenceExpression(typeof(Condition).FullName),
					new BlockExpression(body),
					serialize
				)

			);
	}
}

And what that means is that given this code:

condition a > 10

We can get this result:

And that is it.

You get both the actual compiled expression and the AST that describes this. This is critically important because you can now take this piece of AST and do transformations / views on it.

And that is important if you want to have reliable graphical representation on top of a textual DSL, which is what my chapter 10 is going to cover.

Damn, this is simple! Thanks Rodrigo!

I just started to listen to the Last Centurion. I am under the first minute, and I like it already.  It is apparently in blog style, but I don't know, since I am listening to it. I like it because the very first minute the hero starts talking about nitpickers :-)

One of the common misconceptions about a DSL is that it to think about each DSL script independently. As a simple example, let us go back to the routing DSL, we may have a rule like this:

priority 10
when msg is NewOrder and msg.Amount > 10000:
    dispatch_to "orders@strategic" 

And our focus is mainly on the actual language and syntax that we want to get. This is a mistake, because we aren't considering the environment in which the DSL lives. In the same sense that we rarely consider a code file independently, we should not consider each script independently.

With the routing DSL (and yes, I am stretching the example), we may need to perform additional actions, rather than just dispatch the message. For example, we may want to log all strategic messages. As you can see, we can easily add this to the DSL:

priority 10
when msg is NewOrder and msg.Amount > 10000:
     log "strategic messages" , msg
     dispatch_to "orders@strategic"

However, this is a violation of SoC, and we care about such things with a DSL just as much as we care about them with code. So we can do it like this, leave the original snippet and add another, like this:

when destination.Contains("strategic"):
	log "strategic messages", msg

Now the behavior of the system is split across several files, and it is the responsibility of the DSL engine to deal with this appropriately. One way to arrange this would be this folder structure:

• /routing_scripts
• /routes
• /orders
• dispatch_big_new_orders_to_strategic_customers_handling
• dispatch_standard_new_orders_to_normal_customers_handling
• /behaviors
• /after
• /log_strategic_messages
• /dispatch_to_error_queue_if_not_dispatched

The DSL engine can tell (from the message) that it needs to execute only the routing rules in /routes/orders, and it can execute the before and after actions without getting the routing scripts tied to different concerns.

If you want to be a stickler, we are actually dealing with two dialects that are bound to the same DSL engine. In this case, they are very similar to one another, but that doesn't have to be the case.

Multi file DSL doesn't have to be just about combining different DSLs together, they are also about binding different scripts. Let us look at another possible folder structure:

/routing_scripts

• /routes
• /orders
• dispatch_big_new_orders_to_strategic_customers_handling
• convention_based_dispatching
• dispatch_to_error_if_not_dispatched

In this case, the DSL is based around the idea of executing things in reverse depth order. That is, when a message arrives, we try to match it to the deepest scope possible (in this case, handling strategic customers), and we go up until we reach the root.

This is still, however, just another way of bringing different scripts together. Albeit in a fairly interesting ways.

Let us consider a completely different example, order management. We might have something like this:

// this is part of the rules for processing an order
when order.Total > 1000:
	large_order

// this is the large_order script
order.TaxRate = 0.10
if user.Location in (complex_taxation.Locations):
	complex_taxation.Handle(order)

// this is the complex_taxation/mars script
if amount % 13 == 0:
	return 0.0
return amount * 0.04

Admittedly not the best example as far as business logic goes, but you can see the interaction of three different scripts and how they rely on one another to split the complexity of the system up. It also means that changing the taxation rules on mars is not going to touch how we are going to handle an order, which is good (SoC again).

In this case, I don't think that there is any doubt that those scripts are truly part of a multi file DSL.

Again, when designing and building a DSL, thinking about just a single file is the easiest way to go, but you ought to consider the entire environment when you make such decisions. A language that does not support SoC is bound to get extremely brittle very fast.