Sunday, October 20, 2013

Generating x86 Function Calls Dynamically

Introduction

A few years ago I was considering creating my own byte-code interpreted language.  One of the goals for the language was to be able to call native code as directly as possible from the byte-code.  I think the idea was that the byte-code compiler would setup generic call instructions, and the interpreter would determine whether the destination address was inside the byte-code memory block.  If it wasn't, it would use the CNativeInvoke class, which I am now sharing with all of you - perhaps someone can take this and build the byte-code interpreted language that I didn't write.

CNativeInvoke

CNativeInvoke exposes methods for defining whether a generated call is __cdecl or __stdcall, setting a "this" pointer for C++ class/interface calls, adding parameters, and executing the calls (with or without return values).  CNativeInvoke makes use of the Windows API VirtualAlloc() to allocate memory for the dynamic code block and VirtualProtect() to mark that memory executable.

The comments in the code show you which assembly instructions are being written into memory.  I don't claim to be an assembly expert at all, so it took me some time to figure out which opcodes and such to use.  Visual Studio's mapping from C++ to ASM was helpful during this research.

NativeInvoke.h

#pragma once

#define    SIZE_NATIVE_INVOKE_PAGE        1024
#define    MAX_NATIVE_PARAMS            63

class CNativeInvoke
{
private:
    LPBYTE m_pbPage;
    INT m_nWritePtr;
    INT m_cParams;
    BOOL m_fStackCleanup;
    DWORD_PTR m_dwThisPtr;

public:
    CNativeInvoke (BOOL fStackCleanup = TRUE /* Defaulted for __cdecl */, DWORD_PTR dwThisPtr = 0);
    ~CNativeInvoke ();

    HRESULT Initialize (VOID);
    VOID SetStackCleanup (BOOL fStackCleanup);
    VOID SetThisPtr (DWORD_PTR dwThisPtr);
    VOID Reset (VOID);

    HRESULT AddParam8 (BYTE bParam);
    HRESULT AddParam16 (WORD wParam);
    HRESULT AddParam32 (DWORD dwParam);

    HRESULT Call (DWORD_PTR dwPtr);
    HRESULT Call (DWORD_PTR dwPtr, DWORD* pdwReturn);
    HRESULT Call (DWORD_PTR dwPtr, DWORDLONG* pdwlReturn);

protected:
    VOID EmitCall (DWORD_PTR dwPtr);
    VOID EmitOpCode (BYTE bOpCode, DWORD dwValue);
    VOID EmitOpCode (BYTE bOpCode, BYTE bOperand, DWORD dwValue);
    HRESULT Execute (VOID);
};

NativeInvoke.cpp

#include <windows.h>
#include "Assert.h"    // Change this to include your own Assert(x) macro
#include "NativeInvoke.h"

#define    DWORDPTR(p)        (DWORD)(DWORD_PTR)(p)

CNativeInvoke::CNativeInvoke (BOOL fStackCleanup, DWORD_PTR dwThisPtr)
{
    m_pbPage = NULL;
    m_nWritePtr = 0;
    m_cParams = 0;
    m_fStackCleanup = fStackCleanup;
    m_dwThisPtr = dwThisPtr;
}

CNativeInvoke::~CNativeInvoke ()
{
    if(m_pbPage)
        VirtualFree(m_pbPage,SIZE_NATIVE_INVOKE_PAGE,MEM_RELEASE);
}

HRESULT CNativeInvoke::Initialize (VOID)
{
    HRESULT hr;

    Assert(NULL == m_pbPage);

    m_pbPage = (LPBYTE)VirtualAlloc(NULL,SIZE_NATIVE_INVOKE_PAGE,MEM_COMMIT | 
                         MEM_RESERVE,PAGE_EXECUTE_READWRITE);
    if(m_pbPage)
    {
        Reset();
        hr = S_OK;
    }
    else
        hr = HRESULT_FROM_WIN32(GetLastError());

    return hr;
}

VOID CNativeInvoke::SetStackCleanup (BOOL fStackCleanup)
{
    // TRUE means the caller cleans up the stack, and this means adding a value to esp
    // to cover for the passed parameters.  If FALSE (or if there are no parameters,
    // it is assumed that the callee will clean up the stack.
    //
    // If using __cdecl or __thiscall (with variable parameter lists), you must set
    // the stack cleanup to TRUE.
    //
    // If using __stdcall or __thiscall (with a fixed parameter list), you must set
    // the stack cleanup to FALSE.
    m_fStackCleanup = fStackCleanup;
}

VOID CNativeInvoke::SetThisPtr (DWORD_PTR dwThisPtr)
{
    m_dwThisPtr = dwThisPtr;
}

VOID CNativeInvoke::Reset (VOID)
{
    m_pbPage[0] = 0x55;                                    // push ebp
    m_pbPage[1] = 0x8B;                                    // mov ebp, esp
    m_pbPage[2] = 0xEC;
    m_nWritePtr = 3;
    m_cParams = 0;
}


Read more: Codeproject
QR: Inline image 1