Debugging a classic ReaderWriterLock deadlock with SOSex.dll

I was helping out on an issue the other day where the process would stall if they added enough users in their load tests.  Btw, serious kudos to them for making load tests, so much nicer to work with a problem in test rather than when it is getting critical on a production machine.

We gathered some memory dumps with debug diag of the asp.net process (w3wp.exe) and found that most of the threads were waiting in this type of callstack:

NOTE:  I have changed a lot of function names and code snippets since it is not relevant for the post.

0:071> !clrstack

OS Thread Id: 0x21ac (71)

ESP       EIP    
1c8ceb88 7c8285ec [HelperMethodFrame_1OBJ: 1c8ceb88] System.Threading.ReaderWriterLock.AcquireReaderLockInternal(Int32)

1c8cebe4 1c880d72 DataStore.get_SomeItem()

...

So they were sitting in DataStore.get_SomeItem, waiting to acquire a ReaderLock for some resource.  A ReaderWriterLock is used to synchronize access to resources, where you want to allow multiple readers at the same time, or one writer.  

Read more: If broken it is, fix it you should

Posted via email from jasper22's posterous