I am currently writing the chapter about creating professional DSL. One of the subjects that I am dealing with is creating graphical representations of a textual DSL.

Here is the DSL script in question:

specification @vacations:
	requires @scheduling_work
	requires @external_connections
	
specification @salary:
	users_per_machine 150
	
specification @taxes:
	users_per_machine 50

specification @pension:
	same_machine_as @health_insurance

I am not sure what kind of UI representation this should have.

For that matter, let us take the example that I commonly use, of a rule engine for an ordering system:

when order.total > 1000 and order.payment_method = 'CreditCard':
	apply_discout_of 5.precent

How would you visualize that?

As you know, I had a server crash, that took down the server, and result in a very trying times trying to contact support. VPSLand support insisted repeatedly that all the data was lost and that there isn't anything to be done about it.

Somehow, however, a technical person got involved. The first time I heard about that was when I got an alert for a full disk in the hosted server, which made me hopeful. A few hours later it was up again. I hurriedly logged in and backed up everything that I cared about to both S3 and locally. It went down a few more times, but at this point, I am past caring.

The server will move to a different host as soon as I have the time for that. That old adage about the customer being right? It might not always be right, but I am not going to do business with a place that isn't going to do the most they can do to help me.

I intend to arrive to New York today, and stay there until Sep 5th or so.

As usual, if anyone wants to meet for drinks, talks and bits, I am game.

Rob Conery has another MVC Storefront post, this time focusing on using Windows Workflow.

Those are my random impressions:

• You probably do want to test your work flow. In the same way you want to have an integration test for the system.
• The sequence work flow seems to be a very heavy weight approach to just orchestrating actions in the application.
• I wonder what the perf implications of creating a workflow here would be. My gut feeling is that this is not good, but I don't really have data for that.
• There is probably an issue here with the WF being run in async, I am not sure where it is getting its threads, but if it is from the thread pool, then it is consuming request handling threads, which can kill a site.

As an aside, here is the checkout workflow:

And here is how I would write this:

ValidateOrder()
AuthorizePayment()
order.Status = OrderStatus.Verified
SaveOrder()

Much easier, I think :-) And even more flexible.

Let us take a look at this test. When I saw it, the code under test called to me: "Help me, I being intimately harassed here!"

The problem is that this tests knows all sorts of unrelated things about the code under test. It knows that it should redirect, it knows where it should redirect, etc.

What is the relation between kicking off the checkout process and redirecting? Orthogonal concerns should be in different tests. I may want to send the users to Promotions, instead of Receipt, in the future. Should this test break?

So let us try this one:

We removed the obvious cases of too-intimate tests. But we still have very bad behavior in the test. Specifically, there is a big issue with how we are asserting that the pipeline was kicked off.

On the surface, it may appear that we are doing state based testing, but what we are actually testing is the behavior of the checkout process itself. This is not something that we want to test in this test. In this test, we just want to verify that the checkout process is started.

This test will fail when we change the checkout process, even though the behavior that we intended to test remained the same.

Let us take a look at this test. When I saw it, the test cried out in pain to me: "Help me, I am so overworked."

This test is doing far too much. I drew the battle line on the test, just to give you a clear indication on what is going on.

What we are asserting are things that have nothing to do with what the test is supposed to test.This is also the classic "A test should have a single assertion" example, I think.

The test pass by side effect. It will actually be a problem down the line.

A few hours ago I responded to a mail message that had:

The scenic route is also the poor-man's route...
Pushing a Typemock purchase at the current state of things won't bear any fruit.

I have run into similar sentiments with regards to ReSharper, dotTrace, RedGate's SQL Compare, etc.

I am not sure how to respond to such a statement, because it the underlying assumptions are all wrong. Let us do the math.

TypeMock (enterprise) costs 500$. That is the most expensive version they have (although if you ask nicely, I am pretty sure they will agree to raise the price). Resharper is 350$ for the VB & C# edition, and just 250$ for the C# edition. dotTrace is 500$, and SQL Compare is 400$.

So you need a 500$ tool, or 2000$ set of tools, or something of this order.

What is the ROI of that?

I hit oDesk in order to get some estimates about outsourced costs. It seems that at the high end, in oDesk, you can get a C# dev for around 60$ an hour, but the common rate is something around 20$ an hour.

None of those tools is really mandatory. With a bit of work, you can get things working without them. But that bit of work is expensive.

Taking SQL Compare as an example, it costs 400$. Without it, you are forced to spent time maintaining alter scripts, find out that Joe forgot to checkin the latest script so it is now breaking in QA, etc. Using the common outsource rate as a metric, SQL Compare would cover itself if it would save 2.5 days. From experience with projects that did manual schema management, it takes a lot more than that over the life time of a project.

R# saves a lot of time. It saves it in tiny increments, so it is sometimes not noticeable, but it does. Using the outsource rate again, it would pay for itself by saving a day and a half. I write software 3 to 5 times faster when I have R# than without it. It would be a steal for triple the price (but don't tell JetBrains that).

Trying to put a legacy system under test without TypeMock is possible, but you are going to spend a whole lot more than the 3 days it takes to pay it back. And this math is stupid! I am taking the low end of the pool. If we will take the 60$ an hour, which is more reasonable it takes just a day to see the ROI of any of those tools.

Tools matter, a lot. And we need to consider the moral effects as well. Another big productivity enhancer.

Now, tools aren't silver bullets. You should evaluate a tool to see its worth, absolutely, and there are some tools that may be valuable in one scenario and worthless in the other. But if you believe that a tool will save you time and effort, just do the math. It is usually worth it.

So, given that this math is trivial, why do so many organizations choose the stupid approach? The answer is quite simple, budget.

A department has a budget of 100$. But usually it is not a budget of 100$ outright. It tends to be things like, 85$ allocated to X, 10$ allocated to Y, etc. Salaries are not included in most budgets. Headcount does, but the salary is not something that is usually within a particular department. So given that a salary is fixed and usually outside the direct control of the department, the worth of a time for employees tends to be dismissed. That is already budgeted for, after al, and outside my control. Buying a tool, however, comes out of my budget, which is a problem.

This is the story of trying to achieve local optimum by harming the organization as a whole in all its glory.

Jacob have an interesting perspective on testability:

The thing is that using Typemock means that you can unit test literally any public method of any public class, regardless of any and all internal dependencies that class might have. And you can do so without changing the design and/or architecture of your software at all.

In other words, using Typemock means that everything is testable. Unit testable. Seriously. Everything.
....
Which is why I had such a hard time reading Jeffrey Palermo's latest blog post entitled “Inversion of Control is NOT about testability”. Since I know Jeffrey Palermo is a .Net developer, my initial response is a big fat “Duh. He must be using Typemock.” Sadly, this is not the case.
...

“Testable design” only has value in the things it allows us to do—namely, unit test our classes. If we can unit test our classes as easily no matter what design patterns we choose, then that frees us up to explore other aspects of design choices. It isn't that <le design du jour> ceases to have value, it's just that testability is no longer a factor in evaluating its utility.
...

So, uh, forget all I just said. Spend lots of time making sure your .Net projects are testable. Also: Typemock sucks. Don't bother going there...

There are so many things that I disagree with in this post, I am not sure where to even begin. Before that, I need to point out, as usual, that TypeMock is an awesome tool, and that it can bring a lot of value to the table. It doesn't fit the way that I , personally, works, but that is a personal opinion, and I am just a little bit biased. I just want to make it clear that this post is not an attempt to dismiss TypeMock or its value.

With that out of the way, let us concentrate on the actual content of the post.

First, there is no silver bullet. TypeMock is wonderful, but it is not the answer to every testability prayer (is there such a thing?). If the code is bad, TypeMock can do a lot to help, but it will still be a PITA to test this code.

Second, it looks like Jacob missing the actual message in Palermo's post about "this is not about testability". Good design is one which preserves separation of concerns, maintain the single responsibility principal, amenable to change, etc. This also happen to be a design that is easily testable, but that is beside the point.

Good design is simple one where each piece of the code does one thing, and is not dependent on everything else in the application. To violate that means that you would run into issues the first time you need to make a modification. The reason that we want to avoid dependencies as much as possible is that we have to avoid the cascading change reaction.

I didn't have to look far for an example, two posts prior to this one, Jacob has given an excellent example why a design that is testable with TypeMock (and thus, holy*), run into problems with real world requirements. This is not my scenario, it is his. I can come up with quite a lot of war stories about the problems that code with too much dependencies caused.

Let us take Rhino Mocks as an example (I still think of it as my best work), the project is over 3 years old, started in .Net 1.1 and C# 1.0. It has got through numerous versions and grown through two (and a half :-)) major framework versions. The code is stable and as usable as the the day I started this project. If you dig into Rhino Mocks, you'll find that the code is heavily segregated. That allowed me to add functionality and modify the way things work without difficulty, even when dealing with major changes, like the move to Dynamic Proxy 2 or support the AAA syntax.

That is what you get out of having a good design, software that is maintainable.

* sorry, this is a snipe at Jacob, not TypeMock.

Let me start by stating the scenario. hibernatingrhinos.com is down. The server is compromised, dead, whatever.

I don't know what actually happened to is, but it is no longer bootable.

I don't have complete backups. That is totally and completely my fault. I am an idiot. That is not the point of this post. I should have had backups, and ensured that they work correctly, no arguments here.

What I want to talk about now is the support experience that I got from VPSLand.

We can start with this bad news:

Just to give you some background, I had two "chats" with the "support". In the first it became clear that the "complimentary weekly backups" that they offer are a myth and doesn't really exists. They can't create a new VPS instance and hook the current virtual disk to the new instance, so I can get the data off of the old machine.

The only solution that they offer was to rewipe the entire machine. Losing all the data in it.

Here is my second chat transcript.

VPSLAND.com Sales: Oren, I already told you that there is no back up available for your VPS and it's not possible for us to take back up for your VPS
Oren Eini: okay, then don't
Oren Eini: I need the server itself back up
VPSLAND.com Sales: We want this in written
VPSLAND.com Sales: Please update your ticket again
Oren Eini: Can I get the actual image?
Oren Eini: The VM file?
VPSLAND.com Sales: No
Oren Eini: why not?
VPSLAND.com Sales: Don't you understand, I already told you that your VPS is not up and it's not possible to provide you any data
VPSLAND.com Sales: Any type of data
Oren Eini: I want the VM FILE
Oren Eini: Not any data from it
Oren Eini: The ACTAUL FILE
Oren Eini: The VM is sitting on a file.
Oren Eini: If I have that, _I_ can get the data
Oren Eini: I don't want anything from the machine.
Oren Eini: I want the file that _is_ the machine
VPSLAND.com Sales: Can I have the path for VM ?
VPSLAND.com Sales: VM FILE *
Oren Eini: I don't know. It is on your system
VPSLAND.com Sales: Well Oren, we are not able to mount your VPS on main node which is the reason. We are not able to take back up of your VPS. So Its not possible to provide you VM FILES
Oren Eini: I want the file that you are mounting
Oren Eini: Not the VM itself
VPSLAND.com Sales: Have read my last message carefully
VPSLAND.com Sales: ?
Oren Eini: Yes.
Oren Eini: You are trying to mount the VPS
Oren Eini: You are failing.
Oren Eini: I want the VPS file
Oren Eini: The thing that you are trying to mount
VPSLAND.com Sales: Yes, The VM files are also mounted under root directory
VPSLAND.com Sales: Which is not mounting
VPSLAND.com Sales: So its not possible for us to provide you any VM FILES
Oren Eini: Sorry?
Oren Eini: what root directory are you talking about?
VPSLAND.com Sales: Is there anything else I can help you with ?
Oren Eini: I want the file that you are trying to mount.
VPSLAND.com Sales: You do not understand that, right ?
Oren Eini: The .vhd file
VPSLAND.com Sales: So just listen to me, we are not able to provide you any type of VM FILES
VPSLAND.com Sales: Would you like to reinstall your VPS or NOT ?
Oren Eini: Let me go back to the beginning
Oren Eini: There is the VPS, right?
VPSLAND.com Sales: Ae are not able to provide you any type of VM FILES
VPSLAND.com Sales: Ae = We*
Oren Eini: Please bear with me.
Oren Eini: There is the VPS
Oren Eini: You are trying to mount that on the VM Server
Oren Eini: The VPS is a file
VPSLAND.com Sales: Read above chat once again
Oren Eini: I want that file
Oren Eini: It is not a file inside the VPS.
Oren Eini: It is the file that _is_ the VPS.
Oren Eini: It is the vhd file that you are trying to mount
VPSLAND.com Sales: I understand that, and I apologize for the inconvenience. The fact of the matter is that you are talking to the Sales department right now and unfortunately we don't have any control over such technical issues.
Oren Eini: Okay, than can I get a SUPPORT person to talk to me?
Oren Eini: I am talking to support.vpsland.com
Oren Eini: Not sales.vpsland.com
Oren Eini: I need a TECHNICAL person to solve this issue
VPSLAND.com Sales: They do not have Live Chat support
Oren Eini: Do they have a phone?
VPSLAND.com Sales: Contact them through the email to support@vpsland.com
Oren Eini: Can they answer my emails in a reasonable time frame?
VPSLAND.com Sales: We do not have phone support
Oren Eini: 12 hours between emails is not acceptable
VPSLAND.com Sales: Update to your ticket ULX-445304
Oren Eini: Again, 12 hours between responses are NOT helpful
Oren Eini: If there is support stuff, I would like to communicate with them.
Oren Eini: ASAP
VPSLAND.com Sales: No, they are in the Support Department
Oren Eini: That is lovely.
Oren Eini: Please contact them and tell them that I would like to get a tech people respond to this ticket in a reasonable time frame
VPSLAND.com Sales: Sure, I will update them for you
Oren Eini: Thank you, bye

As of now, for a support issue that is marked as critical, I still don't have any response from their tech support. This is about 30 minutes after the chat, by the way.

Now, this isn't an issue of violating SLA, I don't have one with them for support, I find this deeply disturbing in term of customer service.

I won't be doing business with them again.

Previously on the Multi Tenancy series:

• Multi Tenancy - Definition and scenario.
• Reviewing Litware HR - P&P Multi Tenancy Sample.
• Multi Tenancy - The Physical Data Model
• Multi Tenancy - Extensible Data Model
• Multi Tenancy - Extensible Behaviors
• Multi Tenancy - Development Structure
• Multi Tenancy - Where do you put variability?
• Multi Tenancy - Scriptability and DSL
• Multi Tenancy - multi tenant apps and frameworks
• Multi Tenancy - Keep It Simple, REALLY Simple
• Multi Tenancy - User Interface

Yes, I said that the UI post was the last one before I got to the actual implementation details, but I am having a Skype chat now about the subject, and it is important.

The approach that I have outlined so far is focused on building a system that supports variability at all levels, from the database to the UI, through the entities, services and external integration points. The sweet spot for that is if you have a multi tenant application where a tenant want to have high degree of control on the way the application works.

This high degree of control often means that the tenant desires to change specific parts of the applications in ways that make sense to this tenant, but may not make sense to others. A good example would be payroll system. Here, each tenant has their own processes for how to handle this, and they tend to maintain a high degree of control over that.

A good example from the other side would be something like Subtext. Here, we have a multi tenant application that is essentially the same for all the tenants. The behavior of the system is essentially the same for everyone.

That doesn't means that if you need variability in the application, you should immediately jump to the approach that I outline here. If you have a limited number of variability points, and limited number of variability options (that is, you have X amount of strategies that you can configure for each tenant), you might want to consider that approach.

I would probably avoid that anyway, but it is something to take into account. Some people would feel that a few configuration options makes their life easier than a composite, contextual, application.

My rule of the thumb is that if you have both data extensibility and process variability you need to use the composite approach that I have spoken off. Even if you don't have data extensibility, but "merely" lot of variability in the process across the tenants that cannot be standardize, you will want to take the composite approach.