0
Lately I've taken an interest in hooks and hacks. Don't think bad, my purpose is noble. The thing is, I've run into a bit of a problem with a small 3rd party app I've grown fond of. It won't accept command line arguments, making itself useless in many cases. Unable to work around the problem, and unable to contact the original programmer, the option left is to rewrite such app. Which does a simple thing, and quite honestly I think I can do the same - provided that I get past one obstacle.
This app made use of the Microsoft Detours API library. So I though I'd do the same (what better way to emulate someone than using the same tools he did with?). Today we're at version 3.0. But the old app was almost certainly made with version 1.5, which has been long retired by Microsoft
I've been trying to use Detours 3.0 to hook into the Direct3DCreate9() function to no avail. I'm following the API setup process to the letter, and there's just no room to get creative and invent mistakes. Yet it's failing. The funny thing is that if I try to hook some other API function, like the Win32 ::InvalidateRect(), it's sweet success at the first try.
Normally I'd question my own code, especially since I'm toying with something I don't understand. But I got proof that the Detours API is acting strangely, which is easily observed by inspecting the addresses of the functions you want to place a detour on.
To setup a detour is a very simple mechanism (the whole point of using the Detours API):
// 1) Prepare a function pointer for the API function you wish to detour.
BOOL (WINAPI *fp_FooFunc) (int, int) = FooFunc;
// 2) Write a user function to be called in place of the original API to detour.
BOOL WINAPI Hook_FooFunc (int param1, int param2)
{
// (optional) Do stuff pre-call.
// ...
// (optional) Call the real FooFunc.
BOOL bRet = (*fp_FooFunc) (param1, param2);
// (optional) Do more stuff post-call.
// ...
// Return accordingly.
return bRet;
}
// 3) Use the Detours API functions to place the detour.
// (in my case this is to be done from a DllMain. Here I just show the relevant code)
DetourTransactionBegin ();
DetourUpdateThread (::GetCurrentThread ());
DetourAttach ((void**) &fp_FooFunc, (void*) Hook_FooFunc);
DetourTransactionCommit ();
Like I said, there's no room for creativity. Either you follow this setup, or it won't work.The four above Detour*() functions are returning success. Always.
If I detour a Win32 API, such as InvalidateRect(), all is fine.
If I detour the Direct3DCreate9() function, the detour just doesn't work.
Inspecting the 3 addresses of the functions involved in the above described setup, I observe suspect behavior when attempting to detour Direct3DCreate9(). It is clearly bugging it.
Here is a sample of the typical working scenario:
// Note : Fictional addresses. // --------------------------- // Pre-Detour. FooFunc addr = 12345678 fp_FooFunc addr = 12345678 Hook_FooFunc addr = 654321 // Post-Detour. FooFunc addr = 4321 fp_FooFunc addr = 12345678 Hook_FooFunc addr = 654321 // The detour works.
And here is a sample of the failing scenario:
// Note : Fictional addresses. // --------------------------- // Pre-Detour. FooFunc addr = 12345678 fp_FooFunc addr = 12345678 Hook_FooFunc addr = 654321 // Post-Detour. FooFunc addr = 137 fp_FooFunc addr = 9988776655 Hook_FooFunc addr = 654321 // The detour does not work.
Do you see it? fp_FooFunc's stored address gets changed as well. This is not supposed to happen. That function pointer exists specifically to allow my program to make calls to the original un-detoured FooFunc API. I have no idea where it is pointing with its new address, but that's not the only problem. The new FooFunc address is also bugged, because wherever it is pointing to, it's not resolving to my Hook_FooFunc - I know this because the hooked program is continuing to call the original FooFunc it was linked to, completely ignoring the new address imposed by the detour.
My program isn't crashing. At least that! And upon closing, the Detours' cleanup code correctly reverts the modified function addresses to their original state. Always.
This happens when using Detours 3.0.
I've googled around for help, and have found a website from some chinese guy who [claimed to have] succesfully used Detours 2.1 to hook into the blobs.exe demo of DirectX9. I downaloded his full source code, along with the very source of Detours 2.1 (you have to build the detours.lib yourself after installation). Then I have literally ripped his code to see if it'd work. And it doesn't.
At this point I have uninstalled Detours 3.0.
I have installed Detours 2.1.
And again his code doesn't work.
Taking a closer look, I notice differences between my Detours 2.1 source code and his Detours 2.1 source code (he had a much beefer version of it, somehow).
So I decide to substitute his Detours source code to mine. But again it won't work.
Googling further for help, I found a number of websites explaining various methods to setup various versions of Detours and get them working with DirectX9, and hook into the various methods... but none of them is using Detours 3.0.
And none of them was following the procedure explained by the detours.chm.
The only 1 guy I found showcasing working code with Detours 3.0, wasn't using it like the detours.chm doc explain - for a change.
He was clearly knowing his business as he was doing obscure pointers/offset magic (all uncommented, go figure) to get what he wanted. But in any case he wasn't detouring the Direct3DCreate9() function... so all his magic is useless to me.
Just to make sure I left nothing untried, I have recompiled my program using the Detours 2.1 library. Again the pointers get modified the wrong way and don't work, and again only if I try to detour the Direct3DCreate9() function. With anything else it shall work at the first try.
At this point one could think that maybe the D3D9 program I'm trying to mess with is enacting some practice to prevent this kind of hooking. Not the case. I'm hooking an app I wrote myself to serve as sandbox. This app simply starts, creates a window, and waits. When the user presses Return, it creates Direct3D9, instances a Devide to work in windowed mode, then begins rendering an empty scene as fast as it can. Nothing fancy.
I'm at my wits end with this.
Anybody has ideas?
Detours looks bugged to me. Both versions 2.1 and 3.0 are.
And I can't fix the bug(s) myself, or I wouldn't be here.
Are there alternatives that won't require me to become a nerd of PE, assembly and whatnot?
Possibly C++ alternatives, as I don't know other languages nor am interested in learning them.
This had to be a simple thing...
If you wish to help and have specific questions, I'll post my source code. It's very small.
As for the Detours source code, just download the package (~600 KB) from the microsoft website. Once you install it, you have the \src available, waiting to be built into a static lib.
I was remaining so behind that it wasn't fun anymore.
