Using ADPlus to create dumps on a specific .NET exception
Sometimes an application throws an exception and you need to have a look at the environment to be able to find the cause of the exception. To do this, it is convenient if it would be possible to capture a snapshot of the process precisely at the point when the exception is thrown.
The example at hand is an issue where a customer reported that the application threw an ArgumentException:
ArgumentException: '', hexadecimal value 0x1B, is an invalid character.]
System.Xml.XmlEncodedRawTextWriter.InvalidXmlChar(Int32 ch, Char* pDst, Boolean entitize) +1702938
System.Xml.XmlEncodedRawTextWriter.WriteElementTextBlock(Char* pSrc, Char* pSrcEnd) +493
System.Xml.XmlEncodedRawTextWriter.WriteString(String text) +88
System.Xml.XmlWellFormedWriter.WriteString(String text) +87
System.Xml.XmlWrappedWriter.WriteString(String text) +15
System.ServiceModel.Syndication.TextSyndicationContent.WriteContentsTo(XmlWriter writer) +62
System.ServiceModel.Syndication.SyndicationContent.WriteTo(XmlWriter writer, String outerElementName, String outerElementNamespace) +430
System.ServiceModel.Syndication.Atom10FeedFormatter.WriteItemContents(XmlWriter dictWriter, SyndicationItem item, Uri feedBaseUri) +840
System.ServiceModel.Syndication.Atom10ItemFormatter.WriteItem(XmlWriter writer) +42
System.ServiceModel.Syndication.Atom10ItemFormatter.WriteTo(XmlWriter writer) +58
EPiServer.Search.IndexItemBase.ToSyndicationItemXml() +4104
EPiServer.Search.IndexRequestItem.ToSyndicationItemXml() +139
EPiServer.Search.Data.SearchFactory.AddToQueue(IndexRequestItem item, String namedIndexingService) +117
EPiServer.Search.SearchHandler.UpdateIndex(IndexRequestItem item, String namedIndexingService) +70
EPiServer.Community.Search.SearchHandler.UpdateIndex(Int32 entityId, IEntity entity, IndexAction action) +312
EPiServer.Community.Search.SearchHandler.AddEntitiesToIndex(GetCollection getCollection, Boolean overwrite) +233
EPiServer.Community.Search.SearchHandler.AddCommunityEntitiesToIndex(Type type, Boolean overwrite) +1536
EPiServer.Community.Search.SearchHandler.AddCommunityEntitiesToIndex(Boolean overwrite) +20
Interpreting the exception, we know that the problem occurs during the indexing of some community entity. However, the stack trace leaves us no hint of which entity, nor the field that contain the 0x1B character that the XML writer class is unable to handle.
Again, Debugging Tools for Windows comes to the rescue. This time, I'll show you how to use ADPlus to attach to a process and perform a dump each time an ArgumentException is thrown.
Create a file, let's call it "dump-on-argumentexception.cfg", with the following content:
<ADPlus><!-- Configuring ADPlus to log only exceptions we're interested in -->
<Exceptions><Config><!-- This is for the CLR exception -->
<Code> clr </Code><Actions1> Log </Actions1><CustomActions1> .loadby sos mscorwks; !StopOnException System.ArgumentException 1; j ($t1 = 1) '.dump /ma /u ArgumentException.dmp; gn' ; 'gn'</CustomActions1><ReturnAction1> VOID </ReturnAction1><Actions2> Log </Actions2></Config></Exceptions></ADPlus>
Note that when you're troubleshooting a .NET 4 process, you need to replace ".loadby sos mscorwks" with ".loadby sos clr" in the CustomActions1-element of the configuration above.
Then, invoke ADPlus with the following set of parameters.
C:\Program Files (x86)\Debugging Tools for Windows (x86)>cscript adplus.vbs -crash –p <PID> -FullOnFirst –c dump-on-argumentexception.cfg
Replace <PID> with the process ID you want to monitor, for example the w3wp.exe. ADPlus will launch, attach a console debugger to the process and take a memory dump each time the process throws an ArgumentException, even if it's handled (try/catched).
Debugging the created memory dump(s)
Open up WinDbg, and select File->Open Crash Dump... Navigate to one of the created dumps.
Load the .NET debugging extension:
0:020> .loadby sos mscorwks
Note the 0:020? This is WinDbg's syntax of telling us which process:thread is currently active. In dumps with an active exception on one of its threads, WinDbg will automatically switch to the thread with the exception.
Now, to see the stack trace including all available parameters and local variables, issue the following command:
0:020> !clrstack –a
OS Thread Id: 0xdd1c (20)
ESP EIP
0b8ff600 66083e1c System.Web.HttpRuntime.ProcessRequestInternal(System.Web.HttpWorkerRequest)
PARAMETERS:
this = 0x02611d4c
...
...
Given this information, you may call SOS' !DumpObject command ("!do" for short) to examine the contents of 'this' or any other parameter that are still available (release mode binaries often optimize them away), and !DumpStackObjects ("!dso" for short) to enumerate all objects on the stack.
Unfortunately, I don't have a memory dump that is perfectly suitable to demonstrate this. An out-of-context example would be this examination of the LanguageManager's FileSystemWatcher instance on the current thread.
0:020:x86> !dso
OS Thread Id: 0x1524 (20)
ESP/REG Object Name
000000000f16a624 000000000358c1d4 System.IO.FileSystemWatcher
000000000f16a67c 000000000358c1d4 System.IO.FileSystemWatcher
000000000f16a690 000000000358c1d4 System.IO.FileSystemWatcher
000000000f16a694 000000000358c1d4 System.IO.FileSystemWatcher
000000000f16a6f8 000000000358c1d4 System.IO.FileSystemWatcher
000000000f16a714 000000000358c1d4 System.IO.FileSystemWatcher
000000000f16a730 000000000358c1d4 System.IO.FileSystemWatcher
000000000f16a740 000000000358c1d4 System.IO.FileSystemWatcher
000000000f16a750 000000000358c030 EPiServer.Core.LanguageManager
000000000f16a75c 000000000358c030 EPiServer.Core.LanguageManager
000000000f16a768 000000000358bf28 System.Object
000000000f16a7a0 000000000357f0f4 EPiServer.Web.InitializeEngine+ActionState
000000000f16a7a4 000000000357ec8c EPiServer.Web.InitializeEngine
0:020:x86> !do 000000000358c1d4
Name: System.IO.FileSystemWatcher
MethodTable: 000000000a2de9a4
EEClass: 000000000adb82b0
Size: 92(0x5c) bytes
(C:\windows\assembly\GAC_MSIL\System\2.0.0.0__b77a5c561934e089\System.dll)
Fields:
MT Field Offset Type VT Attr Value Name
0000000060ad078c 400018a 4 System.Object 0 instance 0000000000000000 __identity
000000000a2dc594 40008e0 8 ...ponentModel.ISite 0 instance 0000000000000000 site
000000000ae0b3a4 40008e1 c ....EventHandlerList 0 instance 0000000000000000 events
0000000060ad078c 40008df 108 System.Object 0 static 000000001365be64 EventDisposed
0000000060ad0b70 400315f 10 System.String 0 instance 000000000358c044 directory
0000000060ad0b70 4003160 14 System.String 0 instance 000000000358c098 filter
0000000060aceb3c 4003161 18 ...es.SafeFileHandle 0 instance 0000000000000000 directoryHandle
000000000a2de708 4003162 34 System.Int32 1 instance 81 notifyFilters
0000000060aa463c 4003163 40 System.Boolean 1 instance 0 includeSubdirectories
0000000060aa463c 4003164 41 System.Boolean 1 instance 0 enabled
0000000060aa463c 4003165 42 System.Boolean 1 instance 0 initializing
0000000060ad2dbc 4003166 38 System.Int32 1 instance 8192 internalBufferSize
000000000a2de93c 4003167 48 ...tForChangedResult 1 instance 000000000358c21c changedResult
0000000060aa463c 4003168 43 System.Boolean 1 instance 0 isChanged
00000000084dd30c 4003169 1c ...SynchronizeInvoke 0 instance 0000000000000000 synchronizingObject
0000000060aa463c 400316a 44 System.Boolean 1 instance 0 readGranted
0000000060aa463c 400316b 45 System.Boolean 1 instance 1 disposed
0000000060ad2dbc 400316c 3c System.Int32 1 instance 2 currentSession
000000000a2deb5c 400316d 20 ...ystemEventHandler 0 instance 0000000000000000 onChangedHandler
000000000a2deb5c 400316e 24 ...ystemEventHandler 0 instance 0000000000000000 onCreatedHandler
000000000a2deb5c 400316f 28 ...ystemEventHandler 0 instance 0000000000000000 onDeletedHandler
000000000a2dec40 4003170 2c ...namedEventHandler 0 instance 0000000000000000 onRenamedHandler
000000000a2dfd74 4003171 30 ...ErrorEventHandler 0 instance 0000000000000000 onErrorHandler
0000000060aa463c 4003172 46 System.Boolean 1 instance 1 stopListening
0000000060aa463c 4003173 47 System.Boolean 1 instance 0 runOnce
0000000060ad17a0 4003174 bc0 System.Char[] 0 static 000000001366d540 wildcards
0000000060ad2dbc 4003175 b08 System.Int32 1 static 383 notifyFiltersValidMask
0:020:x86> !do 000000000358c044
Name: System.String
MethodTable: 0000000060ad0b70
EEClass: 000000006088d66c
Size: 82(0x52) bytes
(C:\windows\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
String: C:\EPiServer\Sites\composer\lang
Fields:
MT Field Offset Type VT Attr Value Name
0000000060ad2dbc 4000096 4 System.Int32 1 instance 33 m_arrayLength
0000000060ad2dbc 4000097 8 System.Int32 1 instance 32 m_stringLength
0000000060ad1850 4000098 c System.Char 1 instance 43 m_firstChar
0000000060ad0b70 4000099 10 System.String 0 shared static Empty
>> Domain:Value 0000000000798d20:00000000025b1198 0000000007f817e0:00000000025b1198 0000000007f81ca8:00000000025b1198 <<
0000000060ad17a0 400009a 14 System.Char[] 0 shared static WhitespaceChars
>> Domain:Value 0000000000798d20:00000000025b1788 0000000007f817e0:000000000266e0e0 0000000007f81ca8:00000000035a0794 <<
Just as a side note, the following command can be useful for identifying the PIDs of your IIS7 worker processes:
%windir%\system32\inetsrv\appcmd list wp
As a straight note I think this is a nice little crash course in debugging dump files.