I think one could say that I'm somewhat infatuated with Python. It's a wonderful language really. The language is easy to use, powerful, and developing things in it gets done so much faster than in most other languages I've used. It's also pretty trivial to pickup, and once you're past the basic strangeness of white-spacing based scope resolution, you quickly will find yourself making what would be severely complex applications in it with but a few lines.

Now, at this point, most people would be pointing out the functional origins of many of Python's capabilities, and then point to the many functional languages that have many of the same capabilities. Or perhaps they would point to the dynamic typing, and how that makes development so much more flexible... a few would probably point out that many bugs won't show up till compile-time that a statically typed language would find immediately.

But none of that really matters to me, because the biggest thing that I find with Python is that it encourages readable code. Now, we're not talking C# readable code, which while it can be made readable still tends to be mixed in with a great deal of language jargon that can confuse the casual reader. Nor am I talking about C++ readable code, which just simply doesn't exist. No, I'm talking about code that you can sit down, and almost read out loud in a sensible manner. Code that you can look at, and without having to filter out many of the little niggling bits, can simply understand what it does.

items = [
    {'name' : "Bronze Sword", 'value' : 50, 'diceCount' : 1, 'diceSides' : 4},
    {'name' : "Steel Sword", 'value' : 100, 'diceCount' : 2, 'diceSides' : 4},
    {'name' : "Adamantium Sword", 'value' : 200, 'diceCount' : 1, 'diceSides' : 10}
]

result = [item for item in items if item['value'] > 50]
print result

Now, looking over the above code what immediately comes to mind is that items is an array containing objects, with properties. Pretty cool in my opinion. A similar C# example could be done, but then you would have to rely on either an explicit Item object, anonymous types, or a dictionary of objects (which would involve typecasting, and other nasty behavior)...

using System;
using System.Collections.Generic;
using System.Linq;

static class Program {
    public static void ForEach<T>(this IEnumerable<T> cont, Action<T> action) {
        foreach (var t in cont)
            action(t);
    }

    static void Main(string[] args) {
        var items = new[] {
            new { Name = "Bronze Sword", Value = 50, DiceCount = 1, DiceSides = 4 },
            new {Name = "Steel Sword", Value = 100, DiceCount = 2, DiceSides = 4},
            new {Name = "Adamantium Sword", Value =200, DiceCount = 1, DiceSides = 10}
        };

        var result = from item in items where item.Value > 50 select item;
        result.ForEach(
            (item) =>
                Console.WriteLine(
                    "{{Name : {0}, Value : {1}, DiceCount : {2}, DiceSides : {3}}}",
                    item.Name, item.Value, item.DiceCount, item.DiceSides
                )
        );
    }
}

Now, think about how much longer that is. Heck, read through it a bit and try and easily understand it. It's not that difficult to do, but you're filtering out a lot of useless language garbage. Things like "new" or "var" just get in the way. Heck, look at what we had to do to easily print out the list in a single line! While we could have certainly embedded the foreach explicitly into the main function, the ability to apply that functionality to any query is just too useful to not define an extension method for it.

A C++ version, which I will not provide here, is even worse since it lacks many of the language niceties of C#. This means you end up spending more time doing all that lovely low down dirty work just to get a simple list of items to perform some queries on and then print out the results of.

But how does it encourage readability? Well, first and foremost, there's the scoping. Since scoping is based on indentation, you have to make sure your code is properly indented. The worst experience one can have is to open someone's source code and find a lack of, or haphazard indenting. It completely ruins the flow when reading the code. Then there's the lack of keywords, which require you to interpret them within the context in which they are used, as some keywords in many languages behave differently depending on the context. A trivial example of this is new in C# when allocating a value type versus allocating a reference type. Last, but certainly not least, are all the libraries. These allow you to rapidly build entire applications without having to worry about all the low level nitty gritty stuff. You just get in there, and go.

I've noticed an interesting trend with the different development fields, especially in the game development field. Take a reasonably old idea, but one they haven't encountered before, toss it at them, and wait to see how long it takes before they are exclaiming that it's revolutionary and amazing!

No, seriously.

CODs been done in the enterprise world for a while now, it has a niche there for which it fits very nicely. But when you toss it in a field that hasn't encountered the idea before, two things tend to happen: Firstly, most developers will dismiss it as being useless or too much overhead, often times without even giving it a shot. The second thing to happen is some small group of developers will pick up on it, and run with the idea till they jump off a cliff. The cliff being the case of taking something too far, such as turning your entire game into a set of interacting components.

It takes a while before the general common sense that has built up in other fields settles in, and then you start to see actual adoption and advancement of the ideas within a field (usually using old information that's been republished as new and revolutionary, or by actually increasing the set of knowledge about a methodology by introducing new information). This is typically A Good Thing™. Remember OO? Yeah, me too, except I remember doing it back when game developers still took C to be the king of all languages, with embedded assembly coming in a close second (hell, there are still game developers who think with that mentality, which is not to say that C doesn't have its place, embedded systems for instance, but most of those support C++ now since they are built via GCC).

So back on the component front, it's the new and wonderful thing I see being all the rage these days. Well, not exactly, the ideas are fairly old (hell, COM is a component oriented system), but it seems a lot of naïve implementations of COD are running around as people try and wrap their heads around the rather mundane (from my perspective) concepts. So here are a few small pointers that hardly mean anything at all, but should get you pointed in what is hopefully a reasonable direction.

Components communicate using interfaces. This allows multiple components to be viewed in a similar fashion, but to behave differently behind the scenes. As an example, let us imagine a system that allows us to query for all messages sent on a particular date. Two particular implementations would be say a SQL database with the messages and the date sent, and a set of directories named by date containing files that are the messages that were sent on that date. Ideally our interface allows us to use both of these methods interchangeably (see LSP).

Components can publish interfaces. How exactly this is done is really up to the implementer, for instance you might have them simply exposed in a header file (for C++). This allows for the client and the component to talk to each other, as the component now has an interface that it can pass messages to the client with, and the client has the components interface (as per the previous paragraph), to send messages to the component with.

Components don't have to be local. One of the most common misconceptions is that components are simply objects used within the scope of one process (or even thread) on a single application. But that's not actually true; a component can reside on a separate process, or entirely separate device. Components can publish themselves as providing services (via say CORBA and other object brokers like .Net remoting), and clients can request those services and obtain instances of those components that they can call via the component's exposed interfaces.

The biggest and most important thing to stress though is: Don't go overboard. Be reasonable with your implementations and you won't have much trouble. Refactor frequently, and think before you implement.

Something that's been bugging me for a while now are developers who appear to be wearing blinders.

I don't necessarily mean physically (although that is entirely possible), but mentally. They are confronted with a problem, and when a solution is suggested they are unable to see how to apply it, or come up with non-existent limitations on the solution, because they are unable to see how to apply it in a bigger view. Now, that doesn't mean they are wearing blinders all the time, although I frequently find that they are, but it does suggest an endemic issue with developers being unable to focus on anything more than the problem at hand.

Architects have to be able to view the design from both a high vantage point and lower, more problem oriented vantage points. This duality is what makes architects so special. As such when they encounter a problem, the solution they will devise tends to fit both the problem at hand, and within the design as a whole. However, this ability should also be present within developers (who will hopefully become architects), as you should be able to look beyond your current problem and its limitations and see how a solution can fit as a whole.

A simple example, let's take the instance of submitting high scores to a server for a single player game, assume the game is something like DDR. Submitting high scores then must be done in some manner,, such as a web-post to the server. So the question is, how can we deter cheating? Some ideas were thrown around, including storing the high score in multiple places in memory, using MD5 to ensure that it hadn't changed, and a few other solutions. The problems with these various solutions should be fairly obvious, as an example the first one doesn't stop me from using a packet analyzer to examine the sent data and determining a way of submitting my own scores, nor does the MD5 method, as I can just rehash my high score. A more complex method of submitting high scores was suggested; send the moves made to the server. Let the server run the moves through (it could do it quite fast for a DDR based game, since it would just need the exact key time indices, and then an error margin).

The solution was initially rejected because the author of said game couldn't see how users might submit more than one high score. After it was pointed out that the same method that allows a person to submit multiple high scores would work anyways, he rejected it for other spurious reasons (such as "what happens if they quit the game part way through?"), all of which the entire solution would have suffered from. He was wearing blinders, and couldn't see how the solution fit within the whole architecture.

Now, admittedly, the solution isn't perfect, and it would take some work on the server side as well to properly recalculate the high score. It also doesn't prevent cheating, just makes it quite a bit harder, since they would have to submit the key strokes with the proper timing, and assuming some sort of a random seed, that can be harder to automate.

So the next time you find yourself having issues with a solution, stop. Take a step back, and look at the solution from a few more angles. Perhaps it fits, just not how you expected it to.

As noted previously there are some cases where the performance of unmanaged code can beat that of the managed JIT. In the previous case it was the matrix multiplication function. We do have some other possible performance benefits we can give to our .NET code, specifically, we can NGEN it. NGEN is an interesting utility, it can perform heavy optimizations that would not be possible in the standard runtime JIT (as we shall see). The question before us is: Will it give us enough of a boost to be able to surpass the performance of our unmanaged matrix multiplication?

An Analysis of Existing Code

We haven't looked at the current code that was produced for our previous tests yet, so I feel that it is time we gave it a look and see what we have. To keep this shorter we'll only look at the inner product function. The code produced for the matrix multiplication suffers from the same problems and benefits from the same extensions. For the purposes of this writing we'll only consider the x64 platform. First up we'll look at our unmanaged matrix multiplication, which as we may recall is an SSE2 version. There some things we should note: this method cannot be inlined into the managed code, and there are no frame pointers (they got optimized out).

00000001`800019c3 0f100a          movups  xmm1,xmmword ptr [rdx]
00000001`800019c6 0f59c8          mulps   xmm1,xmm0
00000001`800019c9 0f28c1          movaps  xmm0,xmm1
00000001`800019cc 0fc6c14e        shufps  xmm0,xmm1,4Eh
00000001`800019d0 0f58c8          addps   xmm1,xmm0
00000001`800019d3 0f28c1          movaps  xmm0,xmm1
00000001`800019d6 0fc6c11b        shufps  xmm0,xmm1,1Bh
00000001`800019da 0f58c1          addps   xmm0,xmm1
00000001`800019dd f3410f1100      movss   dword ptr [r8],xmm0
00000001`800019e2 c3              ret

The code used to produce the managed version shown below has undergone a slight modification. No longer does the method return a float, instead it has an out parameter to a float, which ends up holding the result of the operation. This change was made to eliminate some compilation issues in both the managed and unmanaged versions. In the case of the managed version below, without the out parameter the store operation (at 00000642`801673b3) would have required a conversion to a double and back to a single again, the new versions are shown at the end of this post. Examining the managed inner product we get a somewhat worse picture:

00000642`8016732f 4c8b4908        mov     r9,qword ptr [rcx+8]
00000642`80167333 4d85c9          test    r9,r9
00000642`80167336 0f8684000000    jbe     00000642`801673c0
00000642`8016733c f30f104110      movss   xmm0,dword ptr [rcx+10h]
00000642`80167341 488b4208        mov     rax,qword ptr [rdx+8]
00000642`80167345 4885c0          test    rax,rax
00000642`80167348 7676            jbe     00000642`801673c0
00000642`8016734a f30f104a10      movss   xmm1,dword ptr [rdx+10h]
00000642`8016734f f30f59c8        mulss   xmm1,xmm0
00000642`80167353 4983f901        cmp     r9,1
00000642`80167357 7667            jbe     00000642`801673c0
00000642`80167359 f30f105114      movss   xmm2,dword ptr [rcx+14h]
00000642`8016735e 483d01000000    cmp     rax,1
00000642`80167364 765a            jbe     00000642`801673c0
00000642`80167366 f30f104214      movss   xmm0,dword ptr [rdx+14h]
00000642`8016736b f30f59c2        mulss   xmm0,xmm2
00000642`8016736f f30f58c1        addss   xmm0,xmm1
00000642`80167373 4983f902        cmp     r9,2
00000642`80167377 7647            jbe     00000642`801673c0
00000642`80167379 f30f105118      movss   xmm2,dword ptr [rcx+18h]
00000642`8016737e 483d02000000    cmp     rax,2
00000642`80167384 763a            jbe     00000642`801673c0
00000642`80167386 f30f104a18      movss   xmm1,dword ptr [rdx+18h]
00000642`8016738b f30f59ca        mulss   xmm1,xmm2
00000642`8016738f f30f58c8        addss   xmm1,xmm0
00000642`80167393 4983f903        cmp     r9,3
00000642`80167397 7627            jbe     00000642`801673c0
00000642`80167399 f30f10511c      movss   xmm2,dword ptr [rcx+1Ch]
00000642`8016739e 483d03000000    cmp     rax,3
00000642`801673a4 761a            jbe     00000642`801673c0
00000642`801673a6 f30f10421c      movss   xmm0,dword ptr [rdx+1Ch]
00000642`801673ab f30f59c2        mulss   xmm0,xmm2
00000642`801673af f30f58c1        addss   xmm0,xmm1
00000642`801673b3 f3410f114040    movss   dword ptr [r8+40h],xmm0
.
.
.
00000642`801673bd f3c3            rep ret
00000642`801673bf 90              nop
00000642`801673c0 e88b9f8aff      call    mscorwks!JIT_RngChkFail (00000642`7fa11350)

Wow! Lots of conditionals there, it's not vectorized either, but we don't expect it to be, automatic vectorization is a hit and miss type of deal with most optimizing compilers (like the Intel one). Not to mention, vectorizing in the runtime JIT would take up far too much time. This method is inlined for us (thankfully), but we see that it is littered with conditionals and jumps. So where are they jumping to? Well, they are actually ending up just after the end of the method. Note the nop instruction that causes the jump destination to be paragraph aligned, that is intentional. As you can probably guess based on the name from the jump destination, those conditionals are checking the array bounds, stored in r9 and rax, against the indices being used. Those jumps aren't actually that friendly for branch prediction, but for the most part they won't hamper the speed of this method much, but they are an additional cost. Unfortuantly, they are rather problematic for the matrix version, and tend to cost quite a bit in performance.

We also can see that in x64 mode the JIT will use SSE2 for floating point operations. This is quite nice, but does have some interesting consequences, for instance comparing floating point numbers generated using the FPU and those using SSE2 will actually more than likely fail, EVEN IF you truncate them to their appropriate sizes. The reason for this is that the XMM registers (when using the single versions of the instructions and not the double ones) store the floating point values as exactly 32 bit floats. The FPU however will expand them to 80 bit floats, which means that operations on those 80 bit floats before truncating them can affect the lower bits of the 32 bit result in a manner that will result in them differing in the lower portions. If you are wondering when this might become an issue, then you can imagine the problems of running a managed networked game where you have 64bit and 32 bit clients all sending packets to the server. This is just another reason why you should be using deltas for comparison of floats. Other things to note is that with the addition of SSE2 support came the ability to use instructions that save us loads and stores, such as the cvtss2sd and cvtsd2ss instructions, which perform single to double and double to single conversions respectively.

Examining the Call Stack

Of course, there is also the question of exactly what all does our program go through to call our unmanaged methods. First off, the JIT will have to generate several marshalling stubs (to deal with any non-blittable types, although in this case all of the passed types are blittable), along with the security demands. The total number of machines instructions for these stubs is around 10-30, never the less, they aren't inlinable and end up having to be created at runtime. The extra overhead of these calls can add up to quite a bit. First up we'll look at the pinvoke and the delegate stacks:

000006427f66bd14 ManagedMathLib!matrix_mul
0000064280168b85 mscorwks!DoNDirectCall__PatchGetThreadCall+0x78
0000064280168ccc ManagedMathLib!DomainBoundILStubClass.IL_STUB(Single[], Single[], Single[])+0xb5
0000064280168a0f PInvokeTest!SecurityILStubClass.IL_STUB(Single[], Single[], Single[])+0x5c
000006428016893e PInvokeTest!PInvokeTest.Program+<>c__DisplayClass8.<Main>b__0()+0x1f
0000064280167ca1 PInvokeTest!PInvokeTest.Program.TimeTest(TestMethod, Int32)+0x6e
000006427f66c5e2 PInvokeTest!PInvokeTest.Program.Main(System.String[])+0x591

000006427f66bd14 ManagedMathLib!matrix_mul
0000064280168465 mscorwks!DoNDirectCall__PatchGetThreadCall+0x78
00000642801685c1 ManagedMathLib!DomainBoundILStubClass.IL_STUB(Single[], Single[], Single[])+0xb5
0000064280168945 PInvokeTest!SecurityILStubClass.IL_STUB(Single[], Single[], Single[])+0x51
0000064280167d59 PInvokeTest!PInvokeTest.Program.TimeTest(TestMethod, Int32)+0x75
000006427f66c5e2 PInvokeTest!PInvokeTest.Program.Main(System.String[])+0x649

We can see the two stubs that were created, along with this last method called DoNDirectCall__PatchGetThreadCall that actually does the work of calling to our unmanaged function. Exactly what it does is probably what the name says, although I haven't actually dug in and tried to find out what's going on in the internals of it. One important thing to notice is the PInvokeTest!PInvokeTest.Program+<>c__DisplayClass8.<Main>b__0() call, which is actually a delegate used to call to our unmanaged method (passed in to TimeTest). By using the delegate to call the matrix multiplication function, the JIT was able to eliminate the calls entirely. Other than that, the contents of the two sets of stubs are practically identical. The security stub actually asserts that we have the right to call to unmanaged code, as this is a security demand and can change at runtime, this cannot be eliminated. Calling to our unmanaged function from the manged DLL is up next, and it turns out that this is also the most direct call:

000006427f66bf32 ManagedMathLib!matrix_mul
0000064280169601 mscorwks!DoNDirectCallWorker+0x62
00000642801694ef ManagedMathLib!ManagedMathLib.ManagedMath.MatrixMul(Single[], Single[], Single[])+0xd1
0000064280168945 PInvokeTest!PInvokeTest.Program+<>c__DisplayClass8.<Main>b__3()+0x1f
0000064280167ecf PInvokeTest!PInvokeTest.Program.TimeTest(TestMethod, Int32)+0x75
000006427f66c5e2 PInvokeTest!PInvokeTest.Program.Main(System.String[])+0x7bf

As we can see, the only real work that is done to call our unmanaged method is the call to DoNDirectCallWorker. Digging around in that method we find that it is basically a wrapper that saves registers, sets up some registers and then dispatches to the unmanaged function. Upon returning it restores the registers and returns to the caller. There is no dynamic method construction, nor does this require any extra overhead on our end. In fact, one could say that the code is about as fast as we can expect it to be for a managed to unmanaged transition. Looking at the difference between the original unmanaged inner product call and the new version (which writes takes a pointer to the destination float), being made from the managed DLL, we can see a huge difference:

000006427f66bf32 ManagedMathLib!inner_product
0000064280169bd0 mscorwks!DoNDirectCallWorker+0x62
0000064280169acf ManagedMathLib!ManagedMathLib.ManagedMath.InnerProduct(Single[], Single[], Single ByRef)+0xc0
0000064280168955 PInvokeTest!PInvokeTest.Program+<>c__DisplayClass8.<Main>b__7()+0x1f
00000642801681c5 PInvokeTest!PInvokeTest.Program.TimeTest(TestMethod, Int32)+0x75
000006427f66c5e2 PInvokeTest!PInvokeTest.Program.Main(System.String[])+0xab5

000006427f66bd14 ManagedMathLib!inner_product
0000064280169ca3 mscorwks!DoNDirectCall__PatchGetThreadCall+0x78
0000064280169ba0 ManagedMathLib!DomainBoundILStubClass.IL_STUB(Single*, Single*)+0x43
0000064280169b00 ManagedMathLib!ManagedMathLib.ManagedMath.InnerProduct(Single[], Single[])+0x50
000006428016893e PInvokeTest!PInvokeTest.Program+<>c__DisplayClass8.<Main>b__7()+0x20
00000642801681c5 PInvokeTest!PInvokeTest.Program.TimeTest(TestMethod, Int32)+0x6e
000006427f66c5e2 PInvokeTest!PInvokeTest.Program.Main(System.String[])+0xab5

Notice the second call stack has the marshalling stub (also note the parameters to the stub). Returning value types has all sorts of interesting consequences. By changing the signature to write out to a float (in the case of the managed DLL it uses an out parameter), we eliminate the marshalling stub entirely. This improves performance by a decent bit, but nowhere near enough to make up for the call in the first place. The managed inner product is still significantly faster.

And then came NGEN

So, we've gone through and optimized our managed application, but yet it still is running too slow. We contemplate the necessity of moving some code over to the unmanaged world and shudder at the implications. Security would be shot, bugs abound...what to do! But then we remember that there's yet one more option, NGEN!

Running NGEN on our test executable prejitted the whole thing, even methods that eventually ended up being inlined. So, what did it do to our managed inner product? Well first we'll look at the actual method that got prejitted:

PInvokeTest.Program.InnerProduct2(Single[], Single[], Single ByRef)
Begin 0000064288003290, size b0
00000642`88003290 4883ec28        sub     rsp,28h
00000642`88003294 4c8bc9          mov     r9,rcx
00000642`88003297 498b4108        mov     rax,qword ptr [r9+8]
00000642`8800329b 4885c0          test    rax,rax
00000642`8800329e 0f8696000000    jbe     PInvokeTest_ni!COM+_Entry_Point <PERF> (PInvokeTest_ni+0x333a) (00000642`8800333a)
00000642`880032a4 33c9            xor     ecx,ecx
00000642`880032a6 488b4a08        mov     rcx,qword ptr [rdx+8]
00000642`880032aa 4885c9          test    rcx,rcx
00000642`880032ad 0f8687000000    jbe     PInvokeTest_ni!COM+_Entry_Point <PERF> (PInvokeTest_ni+0x333a) (00000642`8800333a)
00000642`880032b3 4533d2          xor     r10d,r10d
00000642`880032b6 483d01000000    cmp     rax,1
00000642`880032bc 767c            jbe     PInvokeTest_ni!COM+_Entry_Point <PERF> (PInvokeTest_ni+0x333a) (00000642`8800333a)
00000642`880032be 41ba01000000    mov     r10d,1
00000642`880032c4 4883f901        cmp     rcx,1
00000642`880032c8 7670            jbe     PInvokeTest_ni!COM+_Entry_Point <PERF> (PInvokeTest_ni+0x333a) (00000642`8800333a)
00000642`880032ca 41ba01000000    mov     r10d,1
00000642`880032d0 483d02000000    cmp     rax,2
00000642`880032d6 7662            jbe     PInvokeTest_ni!COM+_Entry_Point <PERF> (PInvokeTest_ni+0x333a) (00000642`8800333a)
00000642`880032d8 41ba02000000    mov     r10d,2
00000642`880032de 4883f902        cmp     rcx,2
00000642`880032e2 7656            jbe     PInvokeTest_ni!COM+_Entry_Point <PERF> (PInvokeTest_ni+0x333a) (00000642`8800333a)
00000642`880032e4 483d03000000    cmp     rax,3
00000642`880032ea 764e            jbe     PInvokeTest_ni!COM+_Entry_Point <PERF> (PInvokeTest_ni+0x333a) (00000642`8800333a)
00000642`880032ec b803000000      mov     eax,3
00000642`880032f1 4883f903        cmp     rcx,3
00000642`880032f5 7643            jbe     PInvokeTest_ni!COM+_Entry_Point <PERF> (PInvokeTest_ni+0x333a) (00000642`8800333a)
00000642`880032f7 f30f104a14      movss   xmm1,dword ptr [rdx+14h]
00000642`880032fc f3410f594914    mulss   xmm1,dword ptr [r9+14h]
00000642`88003302 f30f104210      movss   xmm0,dword ptr [rdx+10h]
00000642`88003307 f3410f594110    mulss   xmm0,dword ptr [r9+10h]
00000642`8800330d f30f58c8        addss   xmm1,xmm0
00000642`88003311 f30f104218      movss   xmm0,dword ptr [rdx+18h]
00000642`88003316 f3410f594118    mulss   xmm0,dword ptr [r9+18h]
00000642`8800331c f30f58c8        addss   xmm1,xmm0
00000642`88003320 f30f10421c      movss   xmm0,dword ptr [rdx+1Ch]
00000642`88003325 f3410f59411c    mulss   xmm0,dword ptr [r9+1Ch]
00000642`8800332b f30f58c8        addss   xmm1,xmm0
00000642`8800332f f3410f1108      movss   dword ptr [r8],xmm1
00000642`88003334 4883c428        add     rsp,28h
00000642`88003338 f3c3            rep ret
00000642`8800333a e811e0a0f7      call    mscorwks!JIT_RngChkFail (00000642`7fa11350)
00000642`8800333f 90              nop

Interesting results eh? First off, all of the checks are right up front, and ignoring the stack frames we can see exactly what will be inlined. Some other things to note: This method appears a lot better than before, with all of the branches right up at the top where one would assume branch prediction can best deal with them (the registers never change and are being compared to constants). Never the less there are some oddities in this code, for instance there appear to be some extrenuous instructions like mov eax,3. Yeah, don't ask me. Never the less the code is clearly superior to its previous form, and in fact the matrix version is equally as superior, with the range checks being spaced out significantly more (and a bunch are done right up front as well). Of course, the question now is: How much does this help our performance? First up we'll examine some results from the new code base, and then some from the NGEN results on the same code base.

Count: 50
PInvoke MatrixMul : 00:00:07.6456226 Average: 00:00:00.1529124
Delegate MatrixMul: 00:00:06.6500307 Average: 00:00:00.1330006
Managed MatrixMul: 00:00:05.5783511 Average: 00:00:00.1115670
Internal MatrixMul: 00:00:04.5377141 Average: 00:00:00.0907542
PInvoke Inner Product: 00:00:05.4466987 Average: 00:00:00.1089339
Delegate Inner Product: 00:00:04.5001885 Average: 00:00:00.0900037
Managed Inner Product: 00:00:00.5535891 Average: 00:00:00.0110717
Internal Inner Product: 00:00:02.2694728 Average: 00:00:00.0453894

Count: 10
PInvoke MatrixMul : 00:00:01.5706254 Average: 00:00:00.1570625
Delegate MatrixMul: 00:00:01.2689247 Average: 00:00:00.1268924
Managed MatrixMul: 00:00:01.1501118 Average: 00:00:00.1150111
Internal MatrixMul: 00:00:00.9302144 Average: 00:00:00.0930214
PInvoke Inner Product: 00:00:01.0198933 Average: 00:00:00.1019893
Delegate Inner Product: 00:00:00.8538827 Average: 00:00:00.0853882
Managed Inner Product: 00:00:00.0987369 Average: 00:00:00.0098736
Internal Inner Product: 00:00:00.4287660 Average: 00:00:00.0428766

All in all, our performance changes have helped out the managed inner product a decent amount, although even the unmanaged calls managed to get a bit of a boost. Now for the NGEN results:

Count: 50
PInvoke MatrixMul : 00:00:07.5788052 Average: 00:00:00.1515761
Delegate MatrixMul: 00:00:06.2202549 Average: 00:00:00.1244050
Managed MatrixMul: 00:00:04.0376665 Average: 00:00:00.0807533
Internal MatrixMul: 00:00:04.5778189 Average: 00:00:00.0915563
PInvoke Inner Product: 00:00:05.2785764 Average: 00:00:00.1055715
Delegate Inner Product: 00:00:04.1814388 Average: 00:00:00.0836287
Managed Inner Product: 00:00:00.5579279 Average: 00:00:00.0111585
Internal Inner Product: 00:00:02.2419279 Average: 00:00:00.0448385

Count: 10
PInvoke MatrixMul : 00:00:01.3822036 Average: 00:00:00.1382203
Delegate MatrixMul: 00:00:01.1436108 Average: 00:00:00.1143610
Managed MatrixMul: 00:00:00.7386742 Average: 00:00:00.0738674
Internal MatrixMul: 00:00:00.8427460 Average: 00:00:00.0842746
PInvoke Inner Product: 00:00:00.9507331 Average: 00:00:00.0950733
Delegate Inner Product: 00:00:00.7428082 Average: 00:00:00.0742808
Managed Inner Product: 00:00:00.1005084 Average: 00:00:00.0100508
Internal Inner Product: 00:00:00.4025611 Average: 00:00:00.0402561

So, now we can see that our matrix multiplication doesn't offer any advantages over the managed version, in fact it's actually SLOWER than the managed version! We also can see that the unmanaged invocations also benefitted from the NGEN process, as their managed calls were also optimized somewhat, although the stub wrappers are still there and hence still add their overhead. Other things we note is that the inner product function appears to have slowed down just a bit, this might be nothing, or it might be due to machine load or it might genuinly be slower. I'm tempted to say that it's actually slower now, though.

Conclusion

You may recall that this was all sparked by a discussion I had way back when about comparing managed and unmanaged benchmarks and the disadvantages of just setting the /clr flag. I've gone a bit past that though in looking at our managed resources and optimized unmanaged resources and when it is actually beneficial to call into unmanaged code. It is still beneficial to do so, but only with some operations that are just sufficiently taxing enough to bother with. In this case our matrix code which, while in a pure JIT situation, the native code clearly beat out the JIT produced code, gets beat out by the managed version. So what is sufficiently taxing then? Well, set processing might be taxing enough. That is: applying a set of vectorized operations to a collection of objects. But the reality is, you MUST profile first before you can be sure that optimizations of that sort are anywhere near what you need, as if you just assume it will you're probably mistaken.

On a final note, the x86 version also performs better when NGENed than the native version, although in a surprise jump, the delegates actually cost significantly more:

Count: 50
PInvoke MatrixMul : 00:00:07.9897235 Average: 00:00:00.1597944
Delegate MatrixMul: 00:00:27.2561396 Average: 00:00:00.5451227
Managed MatrixMul: 00:00:03.5224029 Average: 00:00:00.0704480
Internal MatrixMul: 00:00:04.5232549 Average: 00:00:00.0904650
PInvoke Inner Product: 00:00:05.5799834 Average: 00:00:00.1115996
Delegate Inner Product: 00:00:29.5660003 Average: 00:00:00.5913200
Managed Inner Product: 00:00:00.5755690 Average: 00:00:00.0115113
Internal Inner Product: 00:00:01.8218949 Average: 00:00:00.0364378

Exactly why this is I haven't investigated, and perhaps I will next time.

Sources for the new inner product functions:

void __declspec(dllexport) inner_product(float const* v1, float const* v2, float* out) {

        __m128 a = _mm_mul_ps(_mm_loadu_ps(v1), _mm_loadu_ps(v2));

        a = _mm_add_ps(a, _mm_shuffle_ps(a, a, _MM_SHUFFLE(1, 0, 3, 2)));

        _mm_store_ss(out, _mm_add_ps(a, _mm_shuffle_ps(a, a, _MM_SHUFFLE(0, 1, 2, 3))));

}

static void InnerProduct(array<float>^ v1, array<float>^ v2, [Runtime::InteropServices::Out] float% result) {

        pin_ptr<float> pv1 = &v1[0];

        pin_ptr<float> pv2 = &v2[0];

        pin_ptr<float> out = &result;

        inner_product(pv1, pv2, out);

}

public static void InnerProduct2(float[] v1, float[] v2, out float f) {

        f = v1[0] * v2[0] + v1[1] * v2[1] + v1[2] * v2[2] + v1[3] * v2[3];

}

Integrating unmanaged code into the managed platform is one of the problem areas with the managed world. Often times the exact costs of calling into unmanaged code is unknown. This obviously leads to some confusion as to when it is appropriate to mix in unmanaged code to help to improve the performance of our application.

PInvoke

There are three ways to access an unmanaged function from managed code. The first is to use the PInvoke capabilities of the language. In C# this is done by declaring a method with external linkage and indicating (using the DllImportAttribute attribute) in which DLL the method may be found. The second way would be to obtain a pointer to the function (using LoadLibrary/GetProcAddress/FreeLibrary), and marshal that pointer to a managed delegate using Marshal.GetDelegateForFunctionPointer. Finally you can write an unmanaged wrapper around the function, using C++/CLI, and invoke that managed method, which will in turn call the unmanaged method.

For the purposes of this post we'll be using two mathematical sample functions. The first being the standard inner product on R³ (aka the dot product), and the second will be a 4x4 matrix multiplication. We'll be comparing two implementations, the first will be a trivial managed implementation of them, and the second will be a SSE2 optimized version. Thanks must be given to Arseny Kapoulkine for the SSE2 version of the matrix multiplication.

First up are the implementations of the inner product functions, it should be noted that I'll be doing the profiling in x64 mode, however the results are similar (albeit a bit slower) for x86.

public static float InnerProduct2(float[] v1, float[] v2) {
        return v1[0] * v2[0] + v1[1] * v2[1] + v1[2] * v2[2] + v1[3] * v2[3];
}

float __declspec(dllexport) inner_product(float const* v1, float const* v2) {
        float result;
        __m128 a = _mm_mul_ps(_mm_loadu_ps(v1), _mm_loadu_ps(v2));
        a = _mm_add_ps(a, _mm_shuffle_ps(a, a, _MM_SHUFFLE(1, 0, 3, 2)));
        _mm_store_ss(&result, _mm_add_ps(a, _mm_shuffle_ps(a, a, _MM_SHUFFLE(0, 1, 2, 3))));
        return result;
}

Things that should be noted about these implementations is that they both operate soley on arrays of floats. InnerProduct2 is inlineable since it's only 23 bytes long and is taking reference types as parameters. The unmanaged inner product could also be implemented using the SSE3 haddps instruction, however I decided to keep it as processor neutral as possible by using only SSE2 instructions.

The implementations of the matrix multiplication vary quite significantly as well, the managed version is the trivial implementation, but its expansion into machine code is quite long. The unmanaged version is an SSE2 optimized one, the raw performance boost of using it is quite significant.

public static void MatrixMul2(float[] m1, float[] m2, float[] o) {
        o[0] = m1[0] * m2[0] + m1[1] * m2[4] + m1[2] * m2[8] + m1[3] * m2[12];
        o[1] = m1[0] * m2[1] + m1[1] * m2[5] + m1[2] * m2[9] + m1[3] * m2[13];
        o[2] = m1[0] * m2[2] + m1[1] * m2[6] + m1[2] * m2[10] + m1[3] * m2[14];
        o[3] = m1[0] * m2[3] + m1[1] * m2[7] + m1[2] * m2[11] + m1[3] * m2[15];

        o[4] = m1[4] * m2[0] + m1[5] * m2[4] + m1[6] * m2[8] + m1[7] * m2[12];
        o[5] = m1[4] * m2[1] + m1[5] * m2[5] + m1[6] * m2[9] + m1[7] * m2[13];
        o[6] = m1[4] * m2[2] + m1[5] * m2[6] + m1[6] * m2[10] + m1[7] * m2[14];
        o[7] = m1[4] * m2[3] + m1[5] * m2[7] + m1[6] * m2[11] + m1[7] * m2[15];

        o[8] = m1[8] * m2[0] + m1[9] * m2[4] + m1[10] * m2[8] + m1[11] * m2[12];
        o[9] = m1[8] * m2[1] + m1[9] * m2[5] + m1[10] * m2[9] + m1[11] * m2[13];
        o[10] = m1[8] * m2[2] + m1[9] * m2[6] + m1[10] * m2[10] + m1[11] * m2[14];
        o[11] = m1[8] * m2[3] + m1[9] * m2[7] + m1[10] * m2[11] + m1[11] * m2[15];

        o[12] = m1[12] * m2[0] + m1[13] * m2[4] + m1[14] * m2[8] + m1[15] * m2[12];
        o[13] = m1[12] * m2[1] + m1[13] * m2[5] + m1[14] * m2[9] + m1[15] * m2[13];
        o[14] = m1[12] * m2[2] + m1[13] * m2[6] + m1[14] * m2[10] + m1[15] * m2[14];
        o[15] = m1[12] * m2[3] + m1[13] * m2[7] + m1[14] * m2[11] + m1[15] * m2[15];
}

void __declspec(dllexport) matrix_mul(float const* m1, float const* m2, float* out)
{
        __m128 r;

        __m128 col1 = _mm_loadu_ps(m2);
        __m128 col2 = _mm_loadu_ps(m2 + 4);
        __m128 col3 = _mm_loadu_ps(m2 + 8);
        __m128 col4 = _mm_loadu_ps(m2 + 12);

        __m128 row1 = _mm_loadu_ps(m1);

        r = _mm_add_ps(_mm_mul_ps(_mm_shuffle_ps(row1, row1, _MM_SHUFFLE(0, 0, 0, 0)), col1),
               _mm_add_ps(_mm_mul_ps(_mm_shuffle_ps(row1, row1, _MM_SHUFFLE(1, 1, 1, 1)), col2),
               _mm_add_ps(_mm_mul_ps(_mm_shuffle_ps(row1, row1, _MM_SHUFFLE(2, 2, 2, 2)), col3),
               _mm_mul_ps(_mm_shuffle_ps(row1, row1, _MM_SHUFFLE(3, 3, 3, 3)), col4))));

        _mm_storeu_ps(out, r); 

        __m128 row2 = _mm_loadu_ps(m1 + 4);

        r = _mm_add_ps(_mm_mul_ps(_mm_shuffle_ps(row2, row2, _MM_SHUFFLE(0, 0, 0, 0)), col1),
               _mm_add_ps(_mm_mul_ps(_mm_shuffle_ps(row2, row2, _MM_SHUFFLE(1, 1, 1, 1)), col2),
               _mm_add_ps(_mm_mul_ps(_mm_shuffle_ps(row2, row2, _MM_SHUFFLE(2, 2, 2, 2)), col3),
               _mm_mul_ps(_mm_shuffle_ps(row2, row2, _MM_SHUFFLE(3, 3, 3, 3)), col4))));

        _mm_storeu_ps(out + 4, r); 

        __m128 row3 = _mm_loadu_ps(m1 + 8);

        r = _mm_add_ps(_mm_mul_ps(_mm_shuffle_ps(row3, row3, _MM_SHUFFLE(0, 0, 0, 0)), col1),
               _mm_add_ps(_mm_mul_ps(_mm_shuffle_ps(row3, row3, _MM_SHUFFLE(1, 1, 1, 1)), col2),
               _mm_add_ps(_mm_mul_ps(_mm_shuffle_ps(row3, row3, _MM_SHUFFLE(2, 2, 2, 2)), col3),
               _mm_mul_ps(_mm_shuffle_ps(row3, row3, _MM_SHUFFLE(3, 3, 3, 3)), col4))));

        _mm_storeu_ps(out + 8, r); 

        __m128 row4 = _mm_loadu_ps(m1 + 12);

        r = _mm_add_ps(_mm_mul_ps(_mm_shuffle_ps(row4, row4, _MM_SHUFFLE(0, 0, 0, 0)), col1),
               _mm_add_ps(_mm_mul_ps(_mm_shuffle_ps(row4, row4, _MM_SHUFFLE(1, 1, 1, 1)), col2),
               _mm_add_ps(_mm_mul_ps(_mm_shuffle_ps(row4, row4, _MM_SHUFFLE(2, 2, 2, 2)), col3),
               _mm_mul_ps(_mm_shuffle_ps(row4, row4, _MM_SHUFFLE(3, 3, 3, 3)), col4))));

        _mm_storeu_ps(out + 12, r);
}

It is trivially obvious that the managed version of the matrix multiplication cannot be inlined. The overhead of the function call is really the least of your worries though (it is the smallest cost of the entire method really). The unmanaged version is a nicely optimized SSE2 method, and requires only a minimal number of loads and stores from main memory, and the loads and stores are reasonably cache friendly (P4 will prefetch 128 bytes of memory).

PInvoke

Of course, the question is, how do these perform against each other when called from a managed application. The profiling setup is quite simple. It simply runs the methods against a set of matricies and vectors (randomly generated) a million times. It repeats those tests several more times (100 in this case), and averages the results. Full optimizations were turned on for both the unmanaged and managed tests. The Internal calls are made from a managed class that directly calls to the unmanaged methods. Both the managed wrapper and the unmanaged methods are hosted in the same DLL (source for the full DLL at the end of this entry).

PInvoke MatrixMul : 00:00:15.0203285 Average: 00:00:00.1502032
Delegate MatrixMul: 00:00:13.1004306 Average: 00:00:00.1310043
Managed MatrixMul: 00:00:10.2809715 Average: 00:00:00.1028097
Internal MatrixMul: 00:00:08.8992407 Average: 00:00:00.0889924
PInvoke Inner Product: 00:00:10.6779944 Average: 00:00:00.1067799
Delegate Inner Product: 00:00:09.3359882 Average: 00:00:00.0933598
Managed Inner Product: 00:00:01.3460812 Average: 00:00:00.0134608
Internal Inner Product: 00:00:05.6842336 Average: 00:00:00.0568423

The first thing to note is that the PInvoke calls for both the matrix multiplication and inner product were the slowest. The delegate calls were only slightly faster than the PInvoke calls. As we move into the managed territory we find the the results begin to diverge. The managed matrix multiplication is slower than the internal matrix multiplication, however the managed inner product is several times faster than the internal one.

Part of the reason behind this divergance is a result of the invocation framework. There is a cost to calling unmanaged methods from managed code, as each method must be wrapped to perform operations such as fixing any managed resources, performing marshalling for non-blittable types, and finally calling the actual native method. After returning the method further marshalling of the return type may be required, along with checks on the condition of the stack and exception checks (SEH exceptions are caught and wrapped in the SEHException class). Even the internal calls to the unmanaged method require some amount of this, although the actual marshalling requirements are avoided, as are some of the other costs. The result is that the costs add up over time, and in the case of the inner product the additional cost overrode the complexity requirements of the method (which is fairly trivial). The case, on the average, is different for the matrix multiplication. The additional costs of the call do not add a significant amount overhead compared to that of the body of the method, which executes faster than that of the managed matrix multiplication due to vectorization.

Performing further testing with counts at 50 and 25 reveal similar results, however the managed matrix multiplication begins to approach the performance of the internal one. However, even at a count of 1 (that's one million matrix multiplications), the internal matrix multiplication is faster than the managed version.

Count = 50
PInvoke MatrixMul : 00:00:07.4730356 Average: 00:00:00.1494607
Delegate MatrixMul: 00:00:06.4519274 Average: 00:00:00.1290385
Managed MatrixMul: 00:00:05.1662482 Average: 00:00:00.1033249
Internal MatrixMul: 00:00:04.3371530 Average: 00:00:00.0867430
PInvoke Inner Product: 00:00:05.3891030 Average: 00:00:00.1077820
Delegate Inner Product: 00:00:04.7625597 Average: 00:00:00.0952511
Managed Inner Product: 00:00:00.6791549 Average: 00:00:00.0135830
Internal Inner Product: 00:00:02.6719175 Average: 00:00:00.0534383

Count = 25
PInvoke MatrixMul : 00:00:03.7432932 Average: 00:00:00.1497317
Delegate MatrixMul: 00:00:03.2074834 Average: 00:00:00.1282993
Managed MatrixMul: 00:00:02.6200096 Average: 00:00:00.1048003
Internal MatrixMul: 00:00:02.2144342 Average: 00:00:00.0885773
PInvoke Inner Product: 00:00:02.8778559 Average: 00:00:00.1151142
Delegate Inner Product: 00:00:02.0178957 Average: 00:00:00.0807158
Managed Inner Product: 00:00:00.3385675 Average: 00:00:00.0135427
Internal Inner Product: 00:00:01.4391529 Average: 00:00:00.0575661

Count = 5
PInvoke MatrixMul : 00:00:00.7642981 Average: 00:00:00.1528596
Delegate MatrixMul: 00:00:00.6407667 Average: 00:00:00.1281533
Managed MatrixMul: 00:00:00.5231416 Average: 00:00:00.1046283
Internal MatrixMul: 00:00:00.4458765 Average: 00:00:00.0891753
PInvoke Inner Product: 00:00:00.5702666 Average: 00:00:00.1140533
Delegate Inner Product: 00:00:00.4122217 Average: 00:00:00.0824443
Managed Inner Product: 00:00:00.0683842 Average: 00:00:00.0136768
Internal Inner Product: 00:00:00.2899304 Average: 00:00:00.0579860

Count = 1
PInvoke MatrixMul : 00:00:00.1476958 Average: 00:00:00.1476958
Delegate MatrixMul: 00:00:00.1337818 Average: 00:00:00.1337818
Managed MatrixMul: 00:00:00.1155993 Average: 00:00:00.1155993
Internal MatrixMul: 00:00:00.0919538 Average: 00:00:00.0919538
PInvoke Inner Product: 00:00:00.1155769 Average: 00:00:00.1155769
Delegate Inner Product: 00:00:00.0906768 Average: 00:00:00.0906768
Managed Inner Product: 00:00:00.0155480 Average: 00:00:00.0155480
Internal Inner Product: 00:00:00.0653527 Average: 00:00:00.0653527

Conclusion

Clearly we should reserve unmanaged operations for longer running methods where the cost of the managed wrappers is negligible compared to the cost of the method. Even heavily optimized methods cost significantly in the wrapping code, and so trivial optimizations are easily overshadowed by that cost. It is best to use unmanaged operations wrapped in a C++/CLI wrapper (and preferably the wrapper will be part of the library that the operations are in). Next time we'll look at the assembly produced by the JIT for these methods under varying circumstances.

Source for Managed DLL:

#pragma managed(push, off)

#include <intrin.h>

extern "C" {

        float __declspec(dllexport) inner_product(float const* v1, float const* v2) {

               float result;

               __m128 a = _mm_mul_ps(_mm_loadu_ps(v1), _mm_loadu_ps(v2));

               a = _mm_add_ps(a, _mm_shuffle_ps(a, a, _MM_SHUFFLE(1, 0, 3, 2)));

               _mm_store_ss(&result, _mm_add_ps(a, _mm_shuffle_ps(a, a, _MM_SHUFFLE(0, 1, 2, 3))));

               return result;

        }

void __declspec(dllexport) matrix_mul(float const* m1, float const* m2, float* out)

{

        __m128 r;

        __m128 col1 = _mm_loadu_ps(m2);

        __m128 col2 = _mm_loadu_ps(m2 + 4);

        __m128 col3 = _mm_loadu_ps(m2 + 8);

        __m128 col4 = _mm_loadu_ps(m2 + 12);

        __m128 row1 = _mm_loadu_ps(m1);

        r = _mm_add_ps(_mm_mul_ps(_mm_shuffle_ps(row1, row1, _MM_SHUFFLE(0, 0, 0, 0)), col1),

               _mm_add_ps(_mm_mul_ps(_mm_shuffle_ps(row1, row1, _MM_SHUFFLE(1, 1, 1, 1)), col2),

               _mm_add_ps(_mm_mul_ps(_mm_shuffle_ps(row1, row1, _MM_SHUFFLE(2, 2, 2, 2)), col3),

               _mm_mul_ps(_mm_shuffle_ps(row1, row1, _MM_SHUFFLE(3, 3, 3, 3)), col4))));

        _mm_storeu_ps(out, r); 

        __m128 row2 = _mm_loadu_ps(m1 + 4);

        r = _mm_add_ps(_mm_mul_ps(_mm_shuffle_ps(row2, row2, _MM_SHUFFLE(0, 0, 0, 0)), col1),

               _mm_add_ps(_mm_mul_ps(_mm_shuffle_ps(row2, row2, _MM_SHUFFLE(1, 1, 1, 1)), col2),

               _mm_add_ps(_mm_mul_ps(_mm_shuffle_ps(row2, row2, _MM_SHUFFLE(2, 2, 2, 2)), col3),

               _mm_mul_ps(_mm_shuffle_ps(row2, row2, _MM_SHUFFLE(3, 3, 3, 3)), col4))));

        _mm_storeu_ps(out + 4, r); 

        __m128 row3 = _mm_loadu_ps(m1 + 8);

        r = _mm_add_ps(_mm_mul_ps(_mm_shuffle_ps(row3, row3, _MM_SHUFFLE(0, 0, 0, 0)), col1),

               _mm_add_ps(_mm_mul_ps(_mm_shuffle_ps(row3, row3, _MM_SHUFFLE(1, 1, 1, 1)), col2),

               _mm_add_ps(_mm_mul_ps(_mm_shuffle_ps(row3, row3, _MM_SHUFFLE(2, 2, 2, 2)), col3),

               _mm_mul_ps(_mm_shuffle_ps(row3, row3, _MM_SHUFFLE(3, 3, 3, 3)), col4))));

        _mm_storeu_ps(out + 8, r); 

        __m128 row4 = _mm_loadu_ps(m1 + 12);

        r = _mm_add_ps(_mm_mul_ps(_mm_shuffle_ps(row4, row4, _MM_SHUFFLE(0, 0, 0, 0)), col1),

               _mm_add_ps(_mm_mul_ps(_mm_shuffle_ps(row4, row4, _MM_SHUFFLE(1, 1, 1, 1)), col2),

               _mm_add_ps(_mm_mul_ps(_mm_shuffle_ps(row4, row4, _MM_SHUFFLE(2, 2, 2, 2)), col3),

                _mm_mul_ps(_mm_shuffle_ps(row4, row4, _MM_SHUFFLE(3, 3, 3, 3)), col4))));

        _mm_storeu_ps(out + 12, r);

}