+972 52-548-6969
3rd party integration assumption - other system was written by a drunken monkey typing with his feet
If you have heard me speak, you are probably are aware that I tend to use this analogy a lot. Any 3rd party system that I have to integrate with was written by a drunken monkey typing with his feet.
So far, I am sad to say, that assumption has been quite accurate over a large range of projects.
You are probably already familiar with concepts of System Boundary and Anti Corruption Layer and how to apply them in order to keep the crazy monkey shtick away from your system, so I am not going to talk about those.
What I do want to talk about now is something slightly different, it is not related to the actual 3rd party system itself, it is related to its management.
One of the more annoying things about 3rd party stuff is that it is usually broken in… interesting ways. So you really want to be able to test that thing during the integration phase, so you would know what to expect. That seems like a very simple concept, right? All you have to do is to hit the QA env. and have a ball. That bring us back to the system management, and the really big screw ups that are happening there.
Before we continue, I want to state that usually, when I am integrating with a 3rd party, I usually do so out of some business reason, so it tend to be the case that I want to give them money, customers, pay per operation, or something else that is to the benefit of that 3rd party! Making integration with your software hard has a direct affect on the number of people who are trying to give you money!
That is why I am so surprised by the amount of trouble that you have to go through in some organizations (I would say, the majority of the organization). Here is a partial list of things that I run into recently:
- Having no QA env. – We were told to basically just work off of the spec and push it to production. You can imagine how big a success that was.
- Having a QA env. that was a different version than the one in production, don’t tell the people that are integrating with you anything about that.
- A variation of the above, have QA env. that is significantly different than production.
- Having a QA env. that has real world effects. For example, if I am testing my bulk mail integration, I do not expect all those emails to be actually sent! Or, in another memorable case, if I am integrating with a merchant provider and testing authorization, I do not want to see those in my real credit report!
- Having a QA env. that requires frequent human intervention. For example:
- In order to validate that your integration has been successful, you have to call someone and wait until they verify that yes, the appropriate values are there in the appropriate system. Each and every time.
- The integration is a one way operation (imagine something like CreateUser, which would fail if it already exists), and you cannot use any dummy values (imagine that you need to pass a valid credit card to that function), you have to have real ones. So every time you test the integration, you have to call someone and have them reset that information.
- Having a QA env. that is down for two weeks just as we were suppose to test the integration.
That is why I am saying that the system management is such a crucial thing. And why I am so surprised and disappointed to see so many organizations get it wrong in ways that are so not funny.
If you are building a system that people will integrate with, consider, as soon as possible, the implications of not having a good testing environment for your system. As a matter of fact, I suggest building QA hooks from day one, so you can pass a flag to the system that would tell it “this is only for tests”, which would mean that any external action on the system would be omitted, but all the logic and behavior are retained.
Comments
I gave up on clients having any sense long ago and just let them pay extra $$$ to have me hang out and do all the stupid sh*t you just listed.
For some bizzare reason they think it's cheaper in the long run. It takes a while to get used to, and some consultants never do, but when you're bored just login to your bank and stare at your account balance growing.
Clients never learn. Just keep churning out a great product and let them do what they're best at - giving you $$$.
@Mr_Simple: Assuming we have a client willing to spend all this time and money. The majority of my clients like so much to make decisions, but somehow always finds it natural to never take responsibility for said decisions. It's funny how we, the vendors, are always the one who should know.
Ayende, I know you're aware of this, but for your readers, in case they aren't: If a 3rd party component has side-effects that you know about, that's easy to handle - you wrap the component in your own interface and remove the direct dependency. Then, for the QA env., you just use an implementation that doesn't have the side-effects - e.g. if the program had to email a bunch of people, you have an email service that just logs it to a shared-drive or whatever you're allowed to, instead of actually emailing all of your clients. This is easy to switch out using a plug-in architecture, or an IOC container, depending on your situation.
Conversely, if a 3rd party component has side-effects that you do NOT know about, then I would think you'd have some kind of way of getting money back from them for negligence. I don't know though - just a thought.
Kyle,
Sorry that I wasn't clear. I am not talking about integrating a component into my system. I am talking about integrating a separate _system_.
Something like payment authorization, creating customers in a separate system, etc.
Ah, my mistake. Still, the technique described is useful of course. :) Integrating WITH a separate system is much harder to control that kind of thing, yeah.
@Thomas Eyde
Authority & Responsibility are twins that cannot be separated. Corporate Survival 101
Give the client all the options but let them make the decision. Lawyering 101
Above all - give great service and take the $$$ to the bank!
I can't agree more. I used to work regularily with one of the "big three" credit bureaus and we got bit by the "QA Environment is different than their Production Environment" pretty badly one time.
Also, it is amazing how many services don't have QA environments at all, or they do, but they share the same database as their production system which cause no end of troubles.
QA is ALWAYS different from production!!!!!
This is not to say that QA shoudn't be there. I usually work for big companies and these have a QA environment. Indeed usually the first one to mess up the QA are the dev people.
To tell you the through the biggest time consuming task is to get these companies to decide on what they want (Organization). In other words, to get them to conclude that the best solution is actually the first solution presented.
I was "brought" up learning that UI layer changes far more often then the middleware, and te middlerware changes far more often then the core system etc etc,
The fact is, unless you are producing a product or generic framework, everything changes all the time. So the only solution is to work as a team, learning and teaching each other (from business to dev people). For that matter we need to some tolerance.
Quality at a budget (as of cost sensible action) is an ongoing effort for which there is not silver bullet. No automatic system, no TDD nothing that solves it. You need good people for that, but also a fair amount of tolerance and understading so that the load can be bared by all.
I read the a Udi article where everything was done by the book, a system that took to years of development etc etc. They had the budget to do everything by the book still it only survived in Production for a few hours as far as he described.
Was it coded by monkeys? No, but I'm sure other (not you Ayende) would be saying just the same about their effort.
Blaming QA is just too easy.
For me the only solution is to ship the stuff in feature by feature rather then trying to solve all the problems in one batch. Ship little, ship fast, ship often. I mean, put stuff in production every fourtnight rather then every three months or more. That is the only way to assure quality at a lower cost.
That is my opinion.
Nuno
PS: I'm not a QA person by the way. Just a developer.
Nuno,
You missed the point of this post by about a mile.
QA in this case meant the QA env. of 3rd party integration, not QA for my project.
I think I did, but I don't see the point of such complains if we don't take the broader prespective. I agree with you that no QA env is a no can do. I also agree that in big companies the management of the QA env is not given to the right people.
Probably the problem is in the methodology aligned to the environment. So we need to be creative. If QA env lack of management becomes a bottleneck then it becomes a PM problem. But we need to be able to communicate why. If they say no, then we need to change methodology.
That is what I was stating.
Nuno
PS: Third party in big companies is just around the corner. So communication, comunication, comunication .....
Comment preview