MS Detours 3.0 and Direct3DCreate9()

Nautilus 103 Apr 19, 2012 at 16:11


Lately I’ve taken an interest in hooks and hacks. Don’t think bad, my purpose is noble. The thing is, I’ve run into a bit of a problem with a small 3rd party app I’ve grown fond of. It won’t accept command line arguments, making itself useless in many cases. Unable to work around the problem, and unable to contact the original programmer, the option left is to rewrite such app. Which does a simple thing, and quite honestly I think I can do the same - provided that I get past one obstacle.

This app made use of the Microsoft Detours API library. So I though I’d do the same (what better way to emulate someone than using the same tools he did with?). Today we’re at version 3.0. But the old app was almost certainly made with version 1.5, which has been long retired by Microsoft <_<
I’ve been trying to use Detours 3.0 to hook into the Direct3DCreate9() function to no avail. I’m following the API setup process to the letter, and there’s just no room to get creative and invent mistakes. Yet it’s failing. The funny thing is that if I try to hook some other API function, like the Win32 ::InvalidateRect(), it’s sweet success at the first try.

Normally I’d question my own code, especially since I’m toying with something I don’t understand. But I got proof that the Detours API is acting strangely, which is easily observed by inspecting the addresses of the functions you want to place a detour on.

To setup a detour is a very simple mechanism (the whole point of using the Detours API):

// 1) Prepare a function pointer for the API function you wish to detour.
BOOL (WINAPI *fp_FooFunc) (int, int) = FooFunc;

// 2) Write a user function to be called in place of the original API to detour.
BOOL WINAPI Hook_FooFunc (int param1, int param2)
    // (optional) Do stuff pre-call.
    // ...

    // (optional) Call the real FooFunc.
    BOOL bRet = (*fp_FooFunc) (param1, param2);

    // (optional) Do more stuff post-call.
    // ...

    // Return accordingly.
    return bRet;

// 3) Use the Detours API functions to place the detour.
// (in my case this is to be done from a DllMain. Here I just show the relevant code)
DetourTransactionBegin ();
DetourUpdateThread (::GetCurrentThread ());
DetourAttach ((void**) &fp_FooFunc, (void*) Hook_FooFunc);
DetourTransactionCommit ();

Like I said, there’s no room for creativity. Either you follow this setup, or it won’t work.

The four above Detour*() functions are returning success. Always.
If I detour a Win32 API, such as InvalidateRect(), all is fine.
If I detour the Direct3DCreate9() function, the detour just doesn’t work.

Inspecting the 3 addresses of the functions involved in the above described setup, I observe suspect behavior when attempting to detour Direct3DCreate9(). It is clearly bugging it.

Here is a sample of the typical working scenario:

// Note : Fictional addresses.
// ---------------------------
// Pre-Detour.
FooFunc   addr = 12345678
fp_FooFunc   addr = 12345678
Hook_FooFunc addr = 654321

// Post-Detour.
FooFunc   addr = 4321
fp_FooFunc   addr = 12345678
Hook_FooFunc addr = 654321

// The detour works.

And here is a sample of the failing scenario:

// Note : Fictional addresses.
// ---------------------------
// Pre-Detour.
FooFunc   addr = 12345678
fp_FooFunc   addr = 12345678
Hook_FooFunc addr = 654321

// Post-Detour.
FooFunc   addr = 137
fp_FooFunc   addr = 9988776655
Hook_FooFunc addr = 654321

// The detour does not work.

Do you see it? fp_FooFunc’s stored address gets changed as well. This is not supposed to happen. That function pointer exists specifically to allow my program to make calls to the original un-detoured FooFunc API. I have no idea where it is pointing with its new address, but that’s not the only problem. The new FooFunc address is also bugged, because wherever it is pointing to, it’s not resolving to my Hook_FooFunc - I know this because the hooked program is continuing to call the original FooFunc it was linked to, completely ignoring the new address imposed by the detour.

My program isn’t crashing. At least that! And upon closing, the Detours’ cleanup code correctly reverts the modified function addresses to their original state. Always.

This happens when using Detours 3.0.
I’ve googled around for help, and have found a website from some chinese guy who [claimed to have] succesfully used Detours 2.1 to hook into the blobs.exe demo of DirectX9. I downaloded his full source code, along with the very source of Detours 2.1 (you have to build the detours.lib yourself after installation). Then I have literally ripped his code to see if it’d work. And it doesn’t.
At this point I have uninstalled Detours 3.0.
I have installed Detours 2.1.
And again his code doesn’t work.

Taking a closer look, I notice differences between my Detours 2.1 source code and his Detours 2.1 source code (he had a much beefer version of it, somehow).
So I decide to substitute his Detours source code to mine. But again it won’t work.

Googling further for help, I found a number of websites explaining various methods to setup various versions of Detours and get them working with DirectX9, and hook into the various methods… but none of them is using Detours 3.0.
And none of them was following the procedure explained by the detours.chm.
The only 1 guy I found showcasing working code with Detours 3.0, wasn’t using it like the detours.chm doc explain - for a change.
He was clearly knowing his business as he was doing obscure pointers/offset magic (all uncommented, go figure) to get what he wanted. But in any case he wasn’t detouring the Direct3DCreate9() function… so all his magic is useless to me.

Just to make sure I left nothing untried, I have recompiled my program using the Detours 2.1 library. Again the pointers get modified the wrong way and don’t work, and again only if I try to detour the Direct3DCreate9() function. With anything else it shall work at the first try.

At this point one could think that maybe the D3D9 program I’m trying to mess with is enacting some practice to prevent this kind of hooking. Not the case. I’m hooking an app I wrote myself to serve as sandbox. This app simply starts, creates a window, and waits. When the user presses Return, it creates Direct3D9, instances a Devide to work in windowed mode, then begins rendering an empty scene as fast as it can. Nothing fancy.

I’m at my wits end with this.
Anybody has ideas?

Detours looks bugged to me. Both versions 2.1 and 3.0 are.
And I can’t fix the bug(s) myself, or I wouldn’t be here.

Are there alternatives that won’t require me to become a nerd of PE, assembly and whatnot?
Possibly C++ alternatives, as I don’t know other languages nor am interested in learning them.
This had to be a simple thing…

If you wish to help and have specific questions, I’ll post my source code. It’s very small.
As for the Detours source code, just download the package (\~600 KB) from the microsoft website. Once you install it, you have the \src available, waiting to be built into a static lib.

3 Replies

Please log in or register to post a reply.

Nautilus 103 Apr 19, 2012 at 22:51

[ Success ]

Looks like the Detours 3.0 API is fine, after all :wub:
The documentation of it is not, though. It is missing the pivotal bit of knowledge that I’ve been searching for. I wonder if it was intentional. Oh it doen’t matter. I’m happy.

Man, I’m crying over a piece of software - how stupid am I? Hehehe.
But I can finally adjust it :lol:

Um… given the audience of this website… Is it okay if I share?
I mean, won’t this twitch some nose? (just making sure)
Else it can stay unmentioned. No problem.

__________Smile_ 101 Apr 19, 2012 at 23:14

In good old days I wrote hooking code myself (overwriting import table, WinAPI have structures/functions for that, no ASM required). Now we have special API just for that…

Nautilus 103 Apr 19, 2012 at 23:45

@}:+()___ (Smile)

In good old days I wrote hooking code myself (overwriting import table, WinAPI have structures/functions for that, no ASM required). Now we have special API just for that…

Hey, I know. I perfectly understand.
I’m the kind of programmer that would like to do everything by himself. Not because my code would beat yours, no. it’s about personal satisfaction. The personal satisfaction I gain from *knowing* how to do something - without external tools.

I’ve had loads of such satisfaction. But in a world where abstraction is ever on the rampage (we got frameworks to abstract just about everything today) the time required to get under the hood and learn how to do it yourself is going to make you fall behind, more and more.

By the time you learn how to assemble a Biplane from the ground up, you look about you and there’s a crowd of ignorant kids speeding on their flashy UFO! Would they know how to assemble a Biplane, or even make minor repairs to their UFO? Certainly not. Still, *they* are speeding on a UFO, while *you* are stuck on a Biplane. Heh.

I had to give up on my “I want to learn” aptitude :( I was remaining so behind that it wasn’t fun anymore.