Saturday, August 31, 2013

Kernel-Mode Debugging in a VM using Visual Studio 2012


This article is about how to setup Kernel-Mode debugging in a VM using Visual Studio 2012. This article will be helpful for people new to driver development that wish to create, build, deploy, and debug a simple "HelloKernel". project. 

The setup process can be divided to three parts:  

  • Setting up the Virtual Machine 
  • Setting up Visual Studio 2012 
  • Setting up your HelloKernel Project 
Each part will be explained step by step.  

How to Setup the Virtual Machine 

The Virtual Machine setup was tested on VMWare 9.0 running Windows 7 x64 OS.

  • Create a new VM, use default settings (tested on VmWare, Win7 x64)
  • Add a serial port to the VM, follow these steps:
    • Right click VM -> Settings -> Add -> Serial Port -> Output to named pipe
    • Named pipe = \\.\pipe\com_2
    • First Combobox = The end is the server
    • Second Combobox = The other end is an application
    • Connect at power on = Checked
    • Finish
    • Yield CPU on poll = Checked
  • In the VM, in an elevated Command Prompt window, enter the following commands:
    • bcdedit /debug on
    • bcdedit /dbgsettings serial debugport:2 baudrate:115200
  • In Windows, search for "Manage advanced sharing settings":
    • -Set "Turn on file and printer sharing"
    • -Set "Turn off password protected sharing"
  • In Windows, search for "Computer Management":
    • -Local Users and Groups -> Users:
    • -Administrator -> Right Click -> Properties -> Uncheck Account is disabled
    • -Administrator -> Right Click -> Set Password -> Set your password
  • Reboot the VM

How to Setup Visual Studio 2012  
  • Install Visual Studio 2012 and WDK 8.
  • On the main toolbar, press Driver-> Test -> Configure Computers -> Add New Computer:
    • Computer name = VM name
    • Select Provision computer and choose debugger settings
    • Next
    • Connection Type = Serial
    • Pipe = Checked
    • Reconnect = Checked
    • Port = \\.\pipe\com_2
    • Target Port = com2
    • Next -> Finish
(Don't worry if Status is Undetermined)

Read more: Codeproject