[DirectX / C++] Runtime Check Failure #0

D619d95cddb1edb227f51ef539d15cdc
0
Nautilus 103 Apr 23, 2013 at 13:43

My DirectX9 wrapper won’t work with a specific game, which -from what I gather- was compiled against Dx9.0c Aug2008 release (what’s found on the game DVD at least). The game will startup and immediately shutdown. No sound, no popup, no error message, no log written nowhere, no Application Error in the system’s events log, not even a crash dump. Nothing.

My wrapper was compiled against a slightly older version yet (Mar2008), so I figured it was time to update the SDK and rebuild the wrapper. For reasons of my own I chose to update to Dx9.0c Mar2009 release.
I recompiled my wrapper. And to ensure that I didn’t break anything in this new version I have first run tests on games that I knew were working fine with it in the past (these range from indie to AAA productions). The new wapper passed all tests.

Proud and confident that it’d work with this latest game as well, I tried it… and again the game would startup then silently shutdown.
So I insert a Sleep() call inside my dll and rebuild it with debug info. Then I launch the game and connect the debugger to it. Stepping line-by-line, at some point visual studio halts with this message:

Run-Time Check Failure #0 - The value of ESP was not properly saved across a function call. This is usually a result of calling a function declared with one calling convention with a function pointer declared with a different calling convention.

Basically it’s saying that the stack pointer is corrupted (on return from a function call the pointer is different from what was before the call), am I correct?
So my code is somehow stepping on forbidden ground. But I believe that the calling conventions hinted to by the message have nothing to do with this. My dll is built to double as a proxy (for when a launcher isn’t an option) and if I really got the calling conventions wrong it would never work, not just this time.

Searching around the web for possible causes of the problem, I find this sample code which triggers the above error message:

class A
{
public:
    virtual int doStuff (int a, int B) { return (a + B); }
};

class B
{
public:
    virtual int doStuff (int a) { return a; }
};

void main (void)
{
    A a;
    int Stuff = ((B*) &a)->doStuff (1); // <-- Runtime Check Failure #0
}

The text accompanying the sample said that I could be using headers and libs that are out of sync (either newer header with older lib, or vice versa). I’m inclined to think this is the case. I upgraded my project from one SDK version to another (Mar2008 -> Mar2009), after all, and haven’t touched a single line of code. I just recompiled.

As the error occurs on a call to IDirect3D9::GetAdapterIdentifiers(), the relevant code (I think) is wholly inside d3d9.h and d3d9.lib. So what do I do, I take a pair of such files from both SDK versions and make a binary comparison of them. Surprise: both pairs of .h and .lib files are identical.

At this point I don’t know what to search for.
Any ideas?

6 Replies

Please log in or register to post a reply.

B5262118b588a5a420230bfbef4a2cdf
0
Stainless 151 Apr 23, 2013 at 14:29

The api must have changed, whats happening is that the code is expecting a function with one form and you are calling it with a different set of parameters.

What’s the line of code causing the error?

D619d95cddb1edb227f51ef539d15cdc
0
Nautilus 103 Apr 23, 2013 at 15:01

I have massively simplified the code in the wrapper, yet the error persists.
I have my H_IDirect3D9 class that inherits from IDirect3D9.

Here is it, from my D3D9Wrapper.h header:
(sorry - the code tag messes up my indentation. Check all the way >> right >>, in the code window. Most methods just forward the call to the underlying D3D9 object)

class H_IDirect3D9 : public IDirect3D9
{
    private:
        IDirect3D9* p;

    public:
        ULONG r;
        H_IDirect3D9 (IDirect3D9* ptr, const ULONG ref) : p (ptr), r (ref) {}
        virtual ~H_IDirect3D9 () {}

    public:
        // IUnknown methods:
        // -----------------

        STDMETHOD(QueryInterface)(THIS_ REFIID riid, void** ppvObj) { return this->p->QueryInterface (riid, ppvObj); }

        // Implemented in: D3D9Wrapper.cpp
        STDMETHOD_(ULONG,AddRef)(THIS);
        STDMETHOD_(ULONG,Release)(THIS);

        // IDirect3D9 methods:
        // -------------------

        STDMETHOD(RegisterSoftwareDevice)(THIS_ void* pInitializeFunction)                                                                                                                                { return this->p->RegisterSoftwareDevice (pInitializeFunction); }
        STDMETHOD_(UINT, GetAdapterCount)(THIS)                                                                                                                                                          { return this->p->GetAdapterCount (); }
        STDMETHOD(GetAdapterIdentifier)(THIS_ UINT Adapter,DWORD Flags,D3DADAPTER_IDENTIFIER9* pIdentifier)                                                                                              { return this->p->GetAdapterIdentifier (Adapter, Flags, pIdentifier); }
        STDMETHOD_(UINT, GetAdapterModeCount)(THIS_ UINT Adapter,D3DFORMAT Format)                                                                                                                        { return this->p->GetAdapterModeCount (Adapter, Format); }
        STDMETHOD(EnumAdapterModes)(THIS_ UINT Adapter,D3DFORMAT Format,UINT Mode,D3DDISPLAYMODE* pMode)                                                                                                    { return this->p->EnumAdapterModes (Adapter, Format, Mode, pMode); }
        STDMETHOD(GetAdapterDisplayMode)(THIS_ UINT Adapter,D3DDISPLAYMODE* pMode)                                                                                                                        { return this->p->GetAdapterDisplayMode (Adapter, pMode); }
        STDMETHOD(CheckDeviceType)(THIS_ UINT Adapter,D3DDEVTYPE DevType,D3DFORMAT AdapterFormat,D3DFORMAT BackBufferFormat,BOOL bWindowed)                                                              { return this->p->CheckDeviceType (Adapter, DevType, AdapterFormat, BackBufferFormat, bWindowed); }
        STDMETHOD(CheckDeviceFormat)(THIS_ UINT Adapter,D3DDEVTYPE DeviceType,D3DFORMAT AdapterFormat,DWORD Usage,D3DRESOURCETYPE RType,D3DFORMAT CheckFormat)                                            { return this->p->CheckDeviceFormat (Adapter, DeviceType, AdapterFormat, Usage, RType, CheckFormat); }
        STDMETHOD(CheckDeviceMultiSampleType)(THIS_ UINT Adapter,D3DDEVTYPE DeviceType,D3DFORMAT SurfaceFormat,BOOL Windowed,D3DMULTISAMPLE_TYPE MultiSampleType,DWORD* pQualityLevels)                  { return this->p->CheckDeviceMultiSampleType (Adapter, DeviceType, SurfaceFormat, Windowed, MultiSampleType, pQualityLevels); }
        STDMETHOD(CheckDepthStencilMatch)(THIS_ UINT Adapter,D3DDEVTYPE DeviceType,D3DFORMAT AdapterFormat,D3DFORMAT RenderTargetFormat,D3DFORMAT DepthStencilFormat)                                      { return this->p->CheckDepthStencilMatch (Adapter, DeviceType, AdapterFormat, RenderTargetFormat, DepthStencilFormat); }
        STDMETHOD(CheckDeviceFormatConversion)(THIS_ UINT Adapter,D3DDEVTYPE DeviceType,D3DFORMAT SourceFormat,D3DFORMAT TargetFormat)                                                                    { return this->p->CheckDeviceFormatConversion (Adapter, DeviceType, SourceFormat, TargetFormat); }
        STDMETHOD(GetDeviceCaps)(THIS_ UINT Adapter,D3DDEVTYPE DeviceType,D3DCAPS9* pCaps)                                                                                                                { return this->p->GetDeviceCaps (Adapter, DeviceType, pCaps); }
        STDMETHOD_(HMONITOR, GetAdapterMonitor)(THIS_ UINT Adapter)                                                                                                                                      { return this->p->GetAdapterMonitor (Adapter); }

        // Implemented in: D3D9Wrapper.cpp
        STDMETHOD(CreateDevice)(THIS_ UINT Adapter,D3DDEVTYPE DeviceType,HWND hFocusWindow,DWORD BehaviorFlags,D3DPRESENT_PARAMETERS* pPresentationParameters,IDirect3DDevice9** ppReturnedDeviceInterface);
};

What I observe is that after creating _and_ releasing the D3D9 object for 4 times in a row (leaving no outstanding reference counts), the game creates D3D9 a 5th time, and then proceeds with calling his IDirect3D9::GetAdapterIdentifier() method which, inside the layer of my wrapper, translates to H_IDirect3D9::GetAdapterIdentifier().
For your reading convenience here’s what the call I receive becomes:

virtual HRESULT H_IDirect3D9::GetAdapterIdentifier (UINT Adapter, DWORD Flags, D3DADAPTER_IDENTIFIER9* pIdentifier)
{
    return this->p->GetAdapterIdentifier (Adapter, Flags, pIdentifier);
}

At this point I suspect that *any* call made to D3D9’s methods would produce the very same error. The ESP disaster must be configuring somewhere earlier. But I can’t spot it.

B5262118b588a5a420230bfbef4a2cdf
0
Stainless 151 Apr 23, 2013 at 16:49

hmmm aren’t all com interfaces defined as pure?

 STDMETHOD (QueryInterface)(THIS_ REFIID riid, void **ppv) PURE;
 STDMETHOD_(ULONG,AddRef)(THIS) PURE;
 STDMETHOD_(ULONG,Release)(THIS) PURE;
D619d95cddb1edb227f51ef539d15cdc
0
Nautilus 103 Apr 24, 2013 at 12:13

@Stainless

hmmm aren’t all com interfaces defined as pure?

STDMETHOD (QueryInterface)(THIS_ REFIID riid, void **ppv) PURE;
STDMETHOD_(ULONG,AddRef)(THIS) PURE;
STDMETHOD_(ULONG,Release)(THIS) PURE;

They are. But pure virtual is only for the interface (the abstract class). The moment you want to inherit from an abstract class in order to spawn an instance, the implementation (my H_IDirect3D9 class) can’t be pure anymore.

D619d95cddb1edb227f51ef539d15cdc
0
Nautilus 103 Apr 24, 2013 at 12:18

(forgive the double post - merely to keep things separate)

More findings from yesterday night…

If I debug line-by-line, slowly, when I inspect the DirectX objects I see their __vfptr being called __vfptr inside the visual studio debugger. Also, the game will [C]reate and [R]elease IDirect3D9 several times in this order: C, R, C, R, C, R, C, R, C, then the Runtime Check Failure #0.

BUT!

If after I’m in with the debugger, I let the code run freely (not line-by-line) till the first breakpoint inside my Direct3DCreate9() function…
… then the __vfptr pointer becomes lpVtbl inside the visual studio debugger. Moreover, the game will [C]reate and [R]elease IDirect3D9 in this order: C, R, C, R, C, C, R, C, R, then an Access Violation first-hand exception occurs (and not the Runtime Check Failure #0).

Anybody knows what I’m talking about? Because I do not.

In the latter scenario something else occurs.
When the game creates IDirect3D9 twice in a row, my code expects to find a reference count equal to 2 the second time… instead DirectX reports 1. This would imply that a call to IDirect3D9::Release() occurred outside of my wrapper’s layer.
It gets better. After the game has made the last [known to me] call to IDirect3D9::Release() it proceeds at calling IDirect3D9::GetAdapterIdentifier(), point at which I get the Access Violation.
Inspecting my wrapper in that moment I notice that the wrapper class is instanced, but the underlying IDirect3D9 object is having a lpVtbl pointing -> to junk, just as if the D3D9 object was released. The problem is that the game was able to use my wrapper pointer (D3D9 to him), meaning that he retained a reference to it because he knew it was valid (meaning that there had to be 1 outstanding pointer despite my believing that there was 0).

Summing up: on one hand I have a case in which I expect to have 2 outstanding pointers to IDirect3D9, but it turns out that there is only 1. Shortly thereafter the opposite happens: when I believe to be 0 outstanding pointers around, it turns out that there was [at least] 1 instead.

This is making no sense… unless the game is somehow calling Direct3DCreate9() and IDirect3D9::Release() outside of my wrapper’s layer. That is, I suspect that this is happening: C, R, C, R, C, R, C, R, C, R, C. The calls in red would occur out of my wrapper’s control.

I’m kinda disoriented. Any suggestions?

D619d95cddb1edb227f51ef539d15cdc
0
Nautilus 103 Apr 30, 2013 at 12:41

Sweet : )

Problem solved. Sneaky game was getting creative with the IUnknown::QueryInterface() method. I must say, it’s the first game I see making any use of that. Yet another demonstration that one can’t make assumptions…

Seeing what truly happens now, I confirm that the order of operations during initialization makes no sense. It creates and releases D3D many times, obtaining several duplicate pointers from different sources in the process, then releasing only a part of them while continuing to ask for more pointers in the canonic way. I suspect it’s a sort of anti-cheat measure, since most cheating softwares exploit the d3d dll to invade a game’s process. Maybe I’m wrong. But it’s a fact that this game ships with dedicated anti-cheat [3rd party] software on the DVD, and the user is prompted whether to install it along with the game (to make online play safer).
Be as it may, I myself know of solid ways (and their counters) to detect alien modules in a game’s address space. I mean, if really this was anti-cheat, it was a weak attempt.

Anyway, in my simplification of the code (see the H_IDirect3D9 wrapper class I posted) I had stripped QueryInterface() of the instructions to sniff into the parameters passing through. That in itself wasn’t causing the error, but had I left it in place I’d have spot the problem sooner.

Remains the mistery of how the speed at which I debug my dll code drives to either the Runtime Check Failure #0 error (slow) or the Access Violation first-hand exception (fast). That, and the __vfptr pointer changing name within the VS debugger (__vfptr if slow, lpVtbl if fast).
If somebody someday has the answer, post away. I’ll make sure to read it.

I’m off to refine my code into a proper release build.

Ciao ciao : )