Databases

SQL CE Transaction Handling

Ayende Rahien — Mon, 31 Mar 2008 16:50:13 GMT

Update: Yes, I am crazy! Turn out that I forgot to do "command.Transaction = tx;" and then I went and read some outdated documentation, and got the completely wrong picture, yuck! I still think that requiring "command.Transaction = tx;" is bad API design and error prone (duh!).

Someone please tell me that I am not crazy. The output out this program is:

Wrote item
Wrote item
Wrote item
3
Wrote item
Wrote item
4
Wrote item
Wrote item
5
Wrote item
Wrote item
6
Wrote item

This is wrong on so many levels...

public class Program
{
	const string connectionString = "Data Source=test.dsf";

	public static void Main(string[] args)
	{
		File.Delete("test.dsf");
		var engine = new SqlCeEngine(connectionString);
		engine.CreateDatabase();

		using (var connection = new SqlCeConnection(connectionString))
		{
			connection.Open();
			SqlCeCommand command = connection.CreateCommand();
			command.CommandText = @"CREATE TABLE Test(Id INT IDENTITY PRIMARY KEY, Name NVARCHAR(25) NOT NULL)";
			command.ExecuteNonQuery();
		}

		ThreadPool.QueueUserWorkItem(ReadFromDb);

		using (var connection = new SqlCeConnection(connectionString))
		{
			connection.Open();
			using(IDbTransaction tx = connection.BeginTransaction(IsolationLevel.Serializable))
			{
				while(true)
				{
					using (SqlCeCommand command = connection.CreateCommand())
					{
						command.CommandText = @"INSERT INTO Test(Name) VALUES('A');";
						command.ExecuteNonQuery();
					}
					Console.WriteLine("Wrote item");
					Thread.Sleep(500);
				}
			}
		}
	}

	private static void ReadFromDb(object state)
	{
		Thread.Sleep(1000);
		using (var connection = new SqlCeConnection(connectionString))
		{
			connection.Open();
			using (IDbTransaction tx = connection.BeginTransaction(IsolationLevel.Serializable))
			{
				while (true)
				{
					using (SqlCeCommand command = connection.CreateCommand())
					{
						command.CommandText = @"SELECT COUNT(*) FROM Test;";
						Console.WriteLine(command.ExecuteScalar()); 
					}
					Console.WriteLine("Wrote item");
					Thread.Sleep(500);
				}
			}
		}
	}
}

SQLite vs. SQL CE

Ayende Rahien — Tue, 22 Jan 2008 02:18:06 GMT

Those two seems to be the most common embedded databases in the .NET world. This is important to me, since I want to do testing against embedded database.

SQL CE can be used with SQL Management Studio, which is nice, but it has three major issues for me so far:

It doesn't support memory only operations. SQLite does, and it means a difference of 12 seconds vs. 40 seconds in running ~100 tests that hit the DB. This is important, especially kicking up everything seems to take about 10 seconds anyway (using Test Driven.Net)
It doesn't support paging (WTF!)
It doesn't support comparing to a sub query, so this is not legal:
select * from account where 1 = (select allow from permissions)

Right now I am experimenting with just how much I can twist around to get everything to break.

I am on the third NH bug this week, and counting :-)

Serialazble isolation level on rows that does not exists

Ayende Rahien — Fri, 18 Jan 2008 01:38:55 GMT

Recently I was asked how to solve this problem. An external service makes a call to the application, to create / update an entity. This call can arrive in one of few endpoints. The catch is that sometimes the external service send the call to create a new entity to all the end points at the same time. This obviously caused issue when trying to insert the same row twice.

I suggested using serializable isolation level to handle this scenario, but I wasn't sure what kind of guarantees is makes for rows that do not exists. I decided to write this simple test case (warning: test code / quick hack, don't write real code like this!).

private static void InsertToDatabase(object state)
{
    int value = 0;

    while (true)
    {
        value += 1;
        try
        {
            using (SqlConnection connection =
                    new SqlConnection("data source=localhost;integrated security=sspi;initial catalog=test"))
            {
                connection.Open();
                using (SqlTransaction transaction =
                        connection.BeginTransaction(IsolationLevel.Serializable))
                using (SqlCommand command = connection.CreateCommand())
                {
                    command.CommandText = "SELECT Id FROM Items WHERE Id = @id";
                    command.Parameters.AddWithValue("id", value);
                    command.Transaction = transaction;
                    if (command.ExecuteScalar() != null)
                        continue;
                    command.CommandText = "INSERT INTO Items (Id) VALUES(@id)";
                    command.ExecuteNonQuery();
                    Console.WriteLine("{1}: Wrote {0}", value, Thread.CurrentThread.ManagedThreadId);

                    transaction.Commit();
                }
            }
        }
        catch (SqlException e)
        {
            if (e.Number == 1205) //transaction deadlock
            {
                Console.WriteLine("{0}: Deadlock recovery", Thread.CurrentThread.ManagedThreadId);
                continue;
            }
            Console.WriteLine(e);
        }
    }
}

As a note, performance for this really sucks if you have contention, and I got a lot of transaction deadlocks when running it with multiply threads.

What I have observed was that it indeed inserted only a single id. All other isolation levels (including snapshot) will produce duplicate rows in this scenario.

Assumption proven, now I only need to find what kind of a lock it takes on a row that doesn't exists.

Another paging approach

Ayende Rahien — Thu, 06 Dec 2007 11:20:25 GMT

I have hard time believing that, but this is something that I learned from how MS CRM works. This is the first time that I ever saw this approach, although it seems so obvious when you think about it.

Updated: Fixed a bug found by Nathan Baulch, thanks!

The idea here is to solve the problem of SQL 2000 not really having a good way to do paging. The basic pattern is this:

select top [N] * from [TableName] 
where [orderCriteria] > @orderCriteriaPreviousMaxValue or 
(
	[Id] > @PreviousID and 
	[orderCriteria] = @orderCriteriaPreviousMaxValue
) 
order by [orderCriteria], [Id]

What do I mean by that?

Let us say that I want to get the first page of customers, I would issue the following query:

select top 25 * from Customers 
order by Customers.Name, Customers.Id

Now, I want to get the next page, I can do this using this approach:

select top 25 * from Customers
where Customers.Name >= @CustomerNamePreviousMaxValue or
(
	Customers.Id > @PreviousId and 
	Customers.Name > @CustomerNamePreviousMaxValue
)
order by Customers.Name, Customers.Id

All we need to remember is the last row's value from the previous query, and pass it to this one. Using this approach we can efficently page through result sets, without having to deal with the limitations of SQL Server limited top capabilities.

The usage of the filtering and the ordering ensure that we will always get the next result back. I find it quite elegant. What is surprising, I don't think that I have ever seen it in any other place.

Perfoming joins without having all the data in memory

Ayende Rahien — Thu, 06 Dec 2007 10:22:41 GMT

Probably the easier way to perform a join is by a nested loop, given dataset A and dataset B, joining all the rows to dataset C is simple:

for row_a in A:
	for row_b in B:
		if condition(row_a, row_b):
			add join(row_a, row_b), C

Supporting left/right/cross joins is simple matter from here, but this has the issue of having to have both datasets in memory. I run into it while processing big files, I don't want to have to hold them in memory, especially if I need several level of joins in order to correctly process them.

I thought about bypassing the entire issue by simply writing the data down to a sqlite DB, and doing the join there. While this is possible, I would rather avoid having to do this.

Any suggestions on where to look to solve this issue?

The CRM Horror

Ayende Rahien — Sun, 07 Oct 2007 17:33:36 GMT

It is rare that I get to the foint where I am just flat out speechless from seeing something. Today I went beyond that, I was flat out speechless and aghast.

The image that you see here is a small part of the FilteredAccount view in Micorosft CRM. Yes, you got that right, a small part.

I got to this from experimenting with the DB model, trying to figure out how things work. I was strongly adviced not to make any use of any sort of view that started with Filtered. "It would make you cry", they said, "don't say we didn't warn you".

I took a peek inside, to see what all the fuss was about. I disagree with Microsoft of quite a few subjects, but they usually manage to hit the target, even if that target is on a completely wrong continent as far as I am concerned.

Some background, before we continue. Filtered Views in MS CRM are used to show the user just the data that they are supposed to see, according to role based permissions. Using the connected usernmae as the way to filter the view.

As a result of that, all the members of the organization has connect access to the CRM DB, and can just open query analyzer and start executing queries against the DB.

An immediate result of that is that you need to start managing security at the DB level, since those views will not be very good if you can just query the underlying tables.

I gues it would not suprise anyone that I have serious issues with this model, but let us put this aside for a moment. I have issues with this model, but they are manageable ones.

Anyway, using the filtered views is the only supported way of handling accessing the database directly, mostly meant for reports. I think that you can guess how efficent it is going to query a filtered view, right?

I counted, the FilteredAccount view has:

5 row level functions
40 outer joins
2 subqueries

At this point, I am considering creating the DETD - Developers for Etichal Treatment of Databases. Surely this is cruel and unusal behavior to put this burden on an unsuspecting database. And I am pretty sure that it is a violation of the Geneva Convention to inflict such a thing on your unsuspecting users.

Left forever wonderring when that query will ever return...

More on Data Layer Componentization

Ayende Rahien — Sat, 06 Oct 2007 23:56:44 GMT

A while ago I commented on Alex's data-layer componentization idea. Alex replied, and Diego as well, but I didn't get around to answering them until now.

To remind you, the basic idea is:

A data-layer component simply allows for information to be shared between different applications.

By supporting re-use data layer componentization allows you to break the duplication habit. You can still have silos, but the natural tendency to duplicate data disappears. Each new silo will instead contain only new types of data and where old types are required, simply pointers, or cross silo foreign keys.

I am going to get to answering their points in a bit, but one thing that I do agree on is the idea of a DataServer, to quote Alex:

I like to think of this as a DataServer as opposed to a Database. The idea being that a DataServer is the place you go to get data in the shape that makes sense for you (i.e. the conceptual model), and is responsible for getting that data from one or more data-sources or databases on your behalf.

Except that I see this in an entirely different light. I don't like the idea of distributing the data over many sources, I would much rather pull all this information to my own DB, and shift the problem from downed application to stale data. That can be a problem, but that depends on the application at hand. See the end of the post about my ideas of this, for now, I want to get to Alex's points.

I mentioned that taking dependencies on several databases cause a server decrease in the reliability of the system, any of them going down will cause the entire system to go down:

Of course systems with multiple dependencies are less robust, but the alternative opens you up to relying on out of date or incorrect data, which hardly seems any better?

It really depends on the application at hand, most business application can work with stale data without much problem, and recovering from that isn't really a problem. But it is far less common to find a scenario where possibly stale answer is better than a downed application.

Another point that I raised was that changing a single application can ripple to all the other applications in the organization.

This is essentially the dual schema problem n’est pas? So you handle it the same way you always handle the dual schema problem, with an intermediary conceptual model. Incidentally there is no reason you have to have only one conceptual model for any given data model either. If one application changes and another doesn't create a different conceptual model.

That is actually the N'th Schema problem. I am nervous about making changes to one application just because another have changed, much less N other applications. Shotgun Surgery in a grand level. Even if we assume that we need to merely update the mapping layer, I have had enough problems with that to cause me to call it simple or painless. And if we are talking about a change that affect the conceptual model as well, this gets even more interesting.

Another concern was about business logic that is required to actual do something with the data. Using this data server approach, you need the business logic to make sense of the data, and if this business logic change in the original application, what happens to the rest of the applications?

I’m not claiming that the ability to share data-layers suddenly negates the need for a business layer! In fact by sharing data-layers it finally becomes possible to share business layers. That is what Base4 tried to do, it was after all a client/server ORM.

I am not seeing how you can share a business layer from multiply application in a single application. You can build a web service that would get the data, already processed, but that is not sharing the business layer from another application, that is accessing that other application for the data. I mentioned security as well, row and column level security that is bounded by business rules, Alex gave much the same answer for this as well, and I have the same answer, I can't see it working after the first change.

Cross databases FK:

Until relational databases matured I am sure the management of standard FK’s was tricky too, we need to recognize that we are the age of distribution, all the data necessary often can’t be in one place, it is an interconnected world and we need to understand this. This is why we need data-servers not just databases.

Nevertheless, this doesn't solve the problem. And trying to do cross DB FKs is a painful performance issue, with no real good answers. That leads well into the problem of performance when you need to pull the data from disparate sources:

Yes doing joins across databases is likely to be slow, but using this performance issue, to say it shouldn't be done is a little like saying downloading a movie from the internet takes a long time, so we should never do it. I don't care if it takes a long time, I want to see the damn movie!

That is fine and good, but saying that you don't care about performance stops when you realize that you have to send the result back to the client in 500ms or less. There were few to none movie downloads in the 90s, because download speed and cost made it impractical in the extreme. Only when it got to the point where you could download a movie fairly easily is started to be really popular.

There are two kinds of performance problems, the small that you don't care about in advance (calling the DB in a loop is a favorite of mine) and the architectural ones that you really want to at least consider ahead of time. Calling external resources is right there on the top of the list as far as I am concerned. If in order to serve a request I need to make 5 requests by design, then I am going to think it really hard before I go on. It is simply not the place I want to start with.

Just because something has issues, doesn’t mean it isn’t useful. I am not claiming that data layer componentization solves every problem; rather that it is a very nice trick to have in the arsenal.

No argument here, I certainly think that Alex has a lot of merit here, I am trying to point out the other side. I don't think that this is something that is quite relevant yet.

Alex then went own to point the flaws in my own preferred approach, bringing all the data into the application's DB using some sort of an ETL process:

For example Ayende's preferred route of using an ETL process to create local copies.

More often than not developers don't actually know that there is another database that has the information they need or simply don't bother with an ETL process anyway.
Often times every database thinks it is the master, and the natural result is no database can be trusted.
Sometimes It is just not practical to duplicate all data, sometimes there is just too much, or sometimes the cost of transfer is too high, or it takes too long to do it, or you can't tolerate dirty data, or you have too many places to copy that data.
In order to avoid (2), unwieldy and error prone ‘human’ processes need to develop, people need to remember to go into 2, 3, 6, 15 (?) systems to keep everything in sync.

To answer the points in order:

If they don't know there is another DB, how would they access the data at the first place. I find it hard to believe that a developer will think "Hm, I guess I need the customer data, let us just invent one"
Defining who is the master of the data is often critical, absolutely. Master-Master merging can be a PITA of a high order.
I can usually copy parts of the data, or do diffs, to decrease the cost of moving the data. The point about stale data is a very good one, I'll answer it in the end of this post.
I don't think so, in order to avoid the master/master problem you need to define who the master is, and how you go about updating that piece of the data.

Diego's post is a more thorough analysis of why taking the ETL approach to the extreme is not a good idea. He focus on the cost of making a change to the system after it was deployed, specifically, what happen when you have N ETL processes that needs to be modified as a result of a schema change?

The cheeky answer to that, have the ETL work of the conceptual schema, at which point you have a single place to go and modify, the ETL processes.

But no, I don't think that I would recommend this approach for all the systems in an organization, assuming N systems that needs each other information, you would have NxN processes, and that is problematic.

So, what is my view of the data server? Well, let us start by removing the data, and then the server.

I have said it before, a application database is an implementation detail, you don't want outsiders to start pawing over your data. The way I consider it, define a set of web services for each application, that allows external applications to use the data in the application. If you needs BA reports, ETL the data to a reporting database, you are likely to want to that anyway.

No, I didn't drink the WS koolaid, and I am not really happy that I am suggesting a WS as a "Solution", but that is just my biases against buzzwards. At its heart, this is the same idea, but with WS instead of data, and with the conceptual schema captured as XML messages.

The major difference is that I have more tools at my disposal to work with WS than with databases. Versioning, for one, is something that isn't trivial by any means, but it is something that WS does much better than databases. Most of the other concerns are also mitigated by this approach. Business level decisions, be it logic or security, for instance, are enforced by the service, which is part of the application, so changing that would immediately affect everything else.

But, didn't I just said that I want to minimize the amount of external dependencies that I have? Doesn't it count as the same thing?

Yes and no. I wouldn't want my application to talk to a dozen services any more than I would want it to talk to a dozen databases. That is where service aggregator come into place. They aggregate related services into a single place. This is the place where I would apply caching, for instance, but not where I would put business logic. This is also a good place to decide what to do about downed service. I can either use a cached value, retry a number of times, try an alternate method, or fail the whole operation.

If this looks like an ESB, I am sorry about it, this is not the intention. I view the service aggregator as a way to aggregate the information from disparate sources into a single place. I can certainly see a way to create some framework for this, and rollout an aggregator per application, to meet that application need. Although I think that several core services or core aggregators should do for most of what the applications need to do.

And yes, that is not my ideal, but I can't really see a way to work with disparate sources without handling this disparity somewhere, so I don't see a way around that.

Schema to wince by...

Ayende Rahien — Tue, 02 Oct 2007 21:12:44 GMT

I was just sent this table schema (minor modifications to protect the responsible party):

CREATE TABLE [dbo].[tblEmployees] 
(
           [iEmployeeSirial] [int] IDENTITY (1, 1) NOT NULL ,
           [vcEmployeeID] [nvarchar] (15) COLLATE Hebrew_CI_AI NULL ,
           [vcEmployeeName] [nvarchar] (50) COLLATE Hebrew_CI_AI NOT NULL ,
           [vcAddress] [nvarchar] (60) COLLATE Hebrew_CI_AI NULL ,
           [vcCity] [nvarchar] (30) COLLATE Hebrew_CI_AI NULL ,
	   // many more fields
)

I am not going to enjoy the integration phase...

Re: Saving to Blob

Ayende Rahien — Sat, 18 Aug 2007 17:11:46 GMT

Mats (NPersist) is talking about Saving to Blob, basically suggesting that you can turn your persistence strategy into Table(Id, Blob):

An alternative would be to serialize the non-identity fields into a blob of some kind and save that to just one column in the database. The database table would then have primary key columns matching the identity fields and then just one more blob column for the serialized non-identity fields of the object.

Since this basically prevent any useful filtering on the data, he suggest that you can extract columns outside the blob, much in the way we currently do with indexes.

As far as I am concerned, this is the worst possible persistence strategy. It means that you have taken some of the great strengths of the database and turned it into a stupid file server, for no real benefit. Mats suggest that this can have positive performance implications, but I doubt that this is the case. Just consider the cost of doing something as simple as providing a sortable, paged, grid using this approach. If the user has chosen to sort on a column that is inside the blob, then you would need to pull all the table in memory, sort it, and then hand the application just the page it is interested in.

It also completely destroy the possibility of working with the database with database oriented tools. I can go into my DBs and look at the tabular data, but 0x20495bacfca12312 doesn't have much meaning to me. ETL tools, reporting, etc are going to become useless. Not to mention that now you are in versioning hell. Serialization is notoriously difficult to handle correct across versions, and to have all the data in the DB in a format that is sensitive to serialization is quite a problem.

Handling associations between objects is another issue, I can use the DB's FK capabilities if I am using the standard mode, but a blob can have no FK enforcement by the DB. In fact, I can't really think of a good reason to do this, and I can think of plenty of reasons not to.

As far as I know, there were "Object <-> Blob Mappers" on the Java side at one time, and they are one of the points that always come up when an argument about OR/M come up, because that is a really bad way to handle this requirement.

Note: There is a really good article on this issue with Java Serialized Objects & Reporting that I have read a few years ago in an Oracle site, but I can't find it.

Re: SSIS - 15 Faults Rebuttal

Ayende Rahien — Fri, 27 Jul 2007 20:13:35 GMT

Seems like this is the new trend right now :-) Phil Brammer has posted a rebuttal to my original post. Here are my answers:

Bad Errors:
You have to understand though, that this isn’t .Net. SSIS has many components/engines at work that obtaining the correct error isn’t always at the heart of the SSIS engine. It could be a database error. It could be an ADO error. Whatever it may be, I agree, some are cryptic, but I’ve generally been able to diagnose my errors. And if there is an error I don’t know about, I contact the community and finally Microsoft through product support.

Error handling is a key part of any platform, this should be a first level concern, that it "isn't always at the heart of the SSIS engine" is a serious issue. Errors happens, and I need to know the entire chain of issues in order to figure how to fix it. To clarify how important that is to me, I am using .Net because it has exceptions, rather than HRESULT.

Random Errors:
The fact these are random should banish this item from your “SSIS 15 Faults” list.

The product sometimes gives an error, for no deterministic reason, and you consider this a non issue?

Keeping track of what it shouldn’t:
Never had this happen. NEVER. NEVER, NEVER, NEVER, NEVER. Did I make my point?

Yeah, your point is "It works on my machine".

Sorry excuse for deployment:
I don’t understand any of the points made here. Deploying to a server has never been a problem. It is in the manual that metadata cannot change between databases/tables. So if you’re moving to an environment that has its metadata different than another, you need to reconcile that first.

I had the package fail with validation errors when moving to production when the DB schema was identical, invalid meta data errors. I don't want to reconcile anything, the schema is identical for the tables that I am working on. And let me point out that "it is in the manual" carry little weight with me, I am disliking the need, not disputing its documentation.

UI Formatting instructions:
Ah, well, what would you say if the SSIS dev team decided to make SSIS packages binary? THEN WHAT? At least you have an XML file that can be parsed.

Uh, you do realize that for all intents and purposes, the file is binary, right? The fact that it happens to be XML based doesn't make it human readable, source control friendly or diffable. All of the above are critical to development.

Bad interoperability:
Is your Oracle example an SSIS problem, or the Oracle driver’s problem?????????

That is probably an SSIS error, but see my previous point about bad errors to know what I am not sure.