Oren Eini

CEO of RavenDB

a NoSQL Open Source Document Database

Get in touch with me:

oren@ravendb.net +972 52-548-6969

Posts: 7,590
|
Comments: 51,218
Privacy Policy · Terms
filter by tags archive
time to read 4 min | 723 words

I am getting some strange perf numbers from MSMQ, and I can’t quite figure out what is going on here.

The scenario is simple, I have a process reading from queue 1 and writing to queue 2. But performance isn’t anywhere near where I think it should be.

In my test scenario, I have queue 1 filled with 10,000 messages, each about 1.5 Kb in size. My test code does a no op move between the queues. Both queues are transactional.

Here is the code:

private static void CopyData()
{
var q1 = new MessageQueue(@".\private$\test_queue1");
var q2 = new MessageQueue(@".\private$\test_queue2");
q2.Purge();
var sp = Stopwatch.StartNew();
while (true)
{
using (var msmqTx = new MessageQueueTransaction())
{
msmqTx.Begin();

Message message;
try
{
message = q1.Receive(TimeSpan.FromMilliseconds(0), msmqTx);
}
catch (MessageQueueException e)
{
Console.WriteLine(e);
break;
}

q2.Send(message, msmqTx);

msmqTx.Commit();
}
}
Console.WriteLine("{0:#,#}", sp.ElapsedMilliseconds);
}

Using this code, it takes 236.8 seconds to move 10,000 messages. If I use System.Transactions, instead of MSMQ’s internal transactions, I get comparable speeds.

Just to give you an idea, this is about 40 messages a second, this number is ridiculously low.

Changing the code so each operation is a separate transaction, like this:

private static void CopyData()
{
var q1 = new MessageQueue(@".\private$\test_queue1");
var q2 = new MessageQueue(@".\private$\test_queue2");
q2.Purge();
var sp = Stopwatch.StartNew();
while (true)
{

Message message;
try
{
message = q1.Receive(TimeSpan.FromMilliseconds(0), MessageQueueTransactionType.Single);
}
catch (MessageQueueException e)
{
Console.WriteLine(e);
break;
}

q2.Send(message, MessageQueueTransactionType.Single);
}
Console.WriteLine("{0:#,#}", sp.ElapsedMilliseconds);
}

Means that it takes 16.3 seconds, or about 600 messages per second, which is far closer to what I would expect.

This is on a quad core machine 8 GB RAM (4 GB free), so I don’t think that it is the machine that is causing the problem. I can see similar results on other machines as well.

Am I missing something? Is there something in my code that is wrong?

time to read 7 min | 1378 words

One of the most annoying bug reports that we got for NH Prof is that it crash with out of memory exception after long use. We did the usual checkups, and the reason for the memory leak was obvious, something kept a lot of objects internal to WPF in memory. In fact, here are the heavy hitters, as extracted from the dump:

Count Size (kliobytes) Type
5,844 3,337 System.Byte[]
69,346 5,474 System.String
49,400 37,198 System.Object[]
1,524,355 47,636 MS.Utility.SingleItemList`1[[System.WeakReference,mscorlib]]
3,047,755 71,432 MS.Internal.Data.ValueChangedEventArgs
1,523,918 71,434 MS.Utility.ThreeItemList`1[[System.WeakReference,mscorlib]]
3,048,292 71,444 MS.Utility.FrugalObjectList`1[[System.WeakReference,mscorlib]]
3,048,292 95,259 System.Windows.WeakEventManager+ListenerList
3,047,755 166,674 MS.Internal.Data.ValueChangedEventManager+ValueChangedRecord
3,056,462 191,029 System.EventHandler
7,644,217 238,882 System.WeakReference

As you can see, this just says that we are doing something that cause WPF to keep a lot of data in memory. In fact, this looks like a classic case of “memory leak” in .NET, where we aren’t releasing references to something. This usually happen with events, and it was the first thing that I checked.

It took a while, but I convinced myself that this wasn’t that. The next step was to try to figure out what is causing this. I’ll skip the sordid tale for now, I’ll queue it up for posting at a later date. What we ended up with is a single line of code that we could prove caused the issue. If we removed it, there was no leak, if it was there, the leak appeared.

That was the point where I threw up my hands and asked Christopher to look at this, I couldn’t think of something bad that we were doing wrong, but Christopher and Rob are the experts in all things WPF.

Christopher managed to reproduce this in an isolated fashion. here is how it goes:

public class TestModel : INotifyPropertyChanged
{
private readonly DispatcherTimer timer;
private int count;

public event PropertyChangedEventHandler PropertyChanged = delegate { };

public TestModel()
{
timer = new DispatcherTimer(DispatcherPriority.Normal)
{
Interval = TimeSpan.FromMilliseconds(50)
};
timer.Tick += Timer_Tick;
timer.Start();
}

public IEnumerable Data
{
get
{
return new[]
{
new {Name = "Ayende"},
};
}
}

private void Timer_Tick(object sender, EventArgs e)
{
if (count++ % 100 == 0)
{
GC.Collect(2, GCCollectionMode.Forced);
Console.WriteLine("{0:#,#}", Process.GetCurrentProcess().WorkingSet64);
}
PropertyChanged(this, new PropertyChangedEventArgs("Data"));
}
}

This is a pretty standard model, showing data that updates frequently. (The short time on the time is to show the problem in a short amount of time.) Note that we are being explicit about forcing a GC release here, to make sure it isn’t just waste memory that haven’t been reclaimed yet.

And the XAML:

<Grid>
<ItemsControl ItemsSource="{Binding Data}">
<ItemsControl.ItemTemplate>
<DataTemplate>
<StackPanel Orientation="Horizontal">
<TextBlock Text="{Binding Name}" />
</StackPanel>
</DataTemplate>
</ItemsControl.ItemTemplate>
</ItemsControl>
</Grid>

Executing this will result in the following memory pattern:

image

Looking at the code, I really don’t see anything that is done wrong there.

I uploaded a sample project that demonstrate the issue here.

Am I going crazy? Am I being stupid? Or is select really broken?

time to read 2 min | 237 words

One of the warning signs for bad multi threaded code is calling code that you don’t own while holding a lock. I just discovered a bug in NH Prof that is related to just that issue.

But I wasn’t really stupid, and the story of this bug is a pretty interesting one.

Let us look at what happened. I have a deadlock in the application, where two threads are waiting for one another. In the UI thread, I have:

image

And on a background thread, I have:

image

This is actually code that happen in two separate classes. And it sounded very strange, until I walked up the stack and found:

image

All nice and tidy, and it looks good. Except that Sessions.Add invoke the SessionsCollectionChanged, which perform a block UI call, while the UI is trying to enter a lock that is currently being held.

Like I said, this is a classic threading issue, but it is pretty hard to see, unless you can actually get the stack trace.

time to read 1 min | 155 words

I don’t have a lot of patience for repeated bugs. I just got a bug report that turn out to be in the result of the same feature as a previous bug I fixed.

I could have fixed the bug. But I didn’t bother. A repeated bug in the same area for the same reason usually indicate a fragile design. In this particular case, the feature wasn’t important, so I just ripped it all out, root & branch.

If it was important, I would have still ripped the whole thing apart, and then I would rebuild it from scratch, using a different design.

Fragile designs are one of the worst enemies that you can have, they will keep dropping things in your lap until you finally fix them once and for all. And I find that usually just starting from scratch on a feature implementation is the best way of doing that.

time to read 1 min | 174 words

I want to point you to this bug:

image

Do you see the Won’t Fix there?

image

Let us take a look at what ExecutionEngineException is all about, okay?

image

Now, the real problem here is that along with this issue there is a failing sample, so it is not a case of not being able to reproduce the error. Granted, the customer has managed to find a workaround for their problem, but the bug in the framework is still there, and was not fixed. I would assume that CLR crashing bugs that “should never occur” would be given some priority, even if the customer managed to find a workaround.

time to read 1 min | 114 words

Here is a crash report that I got.

System.ArgumentException: Destination array was not long enough. Check destIndex and length, and the array's lower bounds.
  at System.Array.Copy(Array sourceArray, Int32 sourceIndex, Array destinationArray, Int32 destinationIndex, Int32 length, Boolean reliable)
  at System.Collections.Generic.List`1.CopyTo(T[] array, Int32 arrayIndex)
  at System.Collections.ObjectModel.Collection`1.CopyTo(T[] array, Int32 index)
  at System.Linq.Buffer`1..ctor(IEnumerable`1 source)
  at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)

Now, it is tempting to blame Microsoft for this, but it is actually my fault.

Care to guess why?

time to read 1 min | 87 words

This is a story from a recent planning meeting that we had. One of the developers was explaining how he is implementing a certain feature, and he went over a potential problem that we would have. That problem is a potential one, because it would only show up if we extend the project in a certain direction, which isn’t currently planned.

My response for that was: “Let the users submit a bug report for that if it happens”.

It is a cruder form of YAGNI.

Git horror

time to read 2 min | 226 words

I am trying to do something that I think is very simple, clone an SVN repository to a local Git repository.

Unfortunately, just about anything that I do end up in one error or another.

E:\rt\rt>git --version
git version 1.6.4.msysgit.0

E:\rt\rt>cd..

E:\rt>git svn clone https://rhino-tools.svn.sourceforge.net/svnroot/rhino-tools rt.git
Initialized empty Git repository in e:/rt/rt.git/.git/
RA layer request failed: PROPFIND request failed on '/svnroot/rhino-tools': PROP
FIND of '/svnroot/rhino-tools': could not connect to server (https://rhino-tools
.svn.sourceforge.net) at C:\Program Files\Git/libexec/git-core/git-svn line 1699

E:\rt>svn list https://rhino-tools.svn.sourceforge.net/svnroot/rhino-tools
BDSLiB/
branches/
contrib/
deprecated/
experiments/
tags/
trunk/

I even tried doing this on my Mac, which worked, but only until revision 2085, after which it refuses to do anything.

I know that I haven’t done much with Git so far, but I don’t think that I am doing something that wrong.

FUTURE POSTS

  1. RavenDB and Gen AI Security - 16 hours from now
  2. RavenDB & Distributed Debugging - 4 days from now
  3. RavenDB & Ansible - 7 days from now

There are posts all the way to Jul 21, 2025

RECENT SERIES

  1. RavenDB 7.1 (7):
    11 Jul 2025 - The Gen AI release
  2. Production postmorterm (2):
    11 Jun 2025 - The rookie server's untimely promotion
  3. Webinar (7):
    05 Jun 2025 - Think inside the database
  4. Recording (16):
    29 May 2025 - RavenDB's Upcoming Optimizations Deep Dive
  5. RavenDB News (2):
    02 May 2025 - May 2025
View all series

Syndication

Main feed ... ...
Comments feed   ... ...
}