Prerequisites

  • .NET 4.5 or higher
  • To record data
    • Windows 10 or higher
    • Pre Windows 10
      • Windows Performance Toolkit of Windows 10 (works also on Windows 7) Please download from the Windows 10 SDK.
      • On Windows 7 you need after the WPT installation to execute
        • wpr -disablepagingexecutive
        • And a reboot to enable full stack walking.
    <!--EndFragment-->
  • To analyze data
    • Windows Performance Toolkit of Windows 10 SDK to load the trace data into WPA
    • But you can configure also another trace viewer like PerfView if you like in the Configuration dialog.
  • Caution
    • ETWController hooks into your keyboard and mouse event handlers. If something goes wrong your machine appears react very slowly. In that case you should close ETWController and not try to debug it because debugging will not get you anywhere. Creating a memory dump and sending it to me is the better option.

Local Recording

When you start it you can first configure the settings to connect to a remote instance. If you want to use it only locally  you do not need to configure anything extra. Mouse and keyboard capturing is enabled along with screenshots by default. All settings are saved when you exit and remembered when you start the tool next time if you prefer a different setup.

image

To start profiling by default the xxwpr.cmd script is configured which is  a simple wrapper around wpr.exe. It supports exactly the same command line options as wpr.

image

Which Commands are Supported?

By default wpr.exe is used as trace collection executable. This means that you need only to pass the command line options of wpr into the start/stop/cancel text boxes.

You can execute any script you like by prepending the command line with :: to tell ETWController that the command line should be treated as custom script.

WPR is by default installed with Windows 10 which makes it a good choice

  • But for more sophisticated recording sessions wpr with its xml logger inheritance is not the best option in my opinion.
  • The good old xperf tool which is part of the Windows Performance Toolkit is still the most flexible tool out there.
  • tracelog from the Driver development kit is the only tool which supports (most) enhanced ETW features which were introduced with Windows 8 and 10.

xxWPR.cmd

xxWPR is a wrapper around WPR. You can pass the same command line arguments to it like WPR.
The only difference is that for the -stop command you can pass as output file name not only .ETL but also .7z or .zip file names.
    -stop xxx.7z [ScreenshotDir] will generate a 7z file from the generated etl file and compress the etl, ngen and optional screenshot folder into
                                 into the archive file. If all goes well the input files are deleted and only the compressed file is kept.

If you configure as output file name .7z or .zip all output data is compressed and stored in one file which makes it a perfect choice if you want to let others record data on their machines so they can easily transfer it to you.

Stopping a Trace        

Press the Stop button and wait until the trace state Stopping stops flashing. If an error happens a message box will pop up. You can examine the script output by pressing the Show Output button.

image

That will create a nice 7z file which contains the etl file, managed pdbs and the saved screenshots

image

If you specify as output file an ETL file you can open the resulting file directly within ETWController by clicking on the Open Trace button. This will execute the configured trace viewer which is currently WPA with a custom profile.

You can configure the used tool in the Configuration - Configuration menu

image

If you want to open a ETL file like ETWController you can apply in WPA this profile by using

image

and select from the ETWController folder Simple.wpaProfile

image

To open any ETL file with that profile you can save it as default profile in WPA

image

This works but has the drawback that if you open another ETL file with ETWController which also loads a profile you will load basically SimpleProfile two times which results in a lot of redundant views. I normally close all tabs and save a empty view as startup profile to work around that. When you need a specific profile you can use the Apply Recently Used Profile to load an interesting one.

The next thing to do is to load the symbols so you are able to make most use of the supplied stacktags which mainly cover .NET related issues. Check out the file ETW\default.stacktags for more information and my blog at

image

 

Distributed Profiling

First you need to deploy and start ETWController on both machines.

Important: Windows Firewall will ask you if you want to give it network access. If you press No you will have to manually fiddle around with your network settings. I have seen interesting issues like that the keyboard events work but the WCF service won´t  work for some reason.

Both instances will automatically start a local WCF service which can receive keyboard and mouse events from remote machines.

Now you need to configure on your "client" machine the remote machine in the Configuration - Configuration menu. There you need to enter the name or IP of the remote host and press OK.

image

To send data over the wire check the "Start Sending" check box.

image

To start a distributed profiling session go to Trace Collection and check x Enable for the local and remote machine. The default output file name contains %COMPUTERNAME% for exactly this reason. If you collect the traces from both machines and copy the data into one directory you will not overwrite the files from the other machines and as added bonus you can see in the file name on which machine it was collected.

image

When you press Start/Stop/Cancel the ETW recording session on both machines will be started/stopped or cancelled at the same time. The recorded data on both machine will get HookTracer events for all mouse and keyboard events which should make it easy to correlate the user input and the reaction of the remote machine.

 

Network Analysis

If you suspect the network it can help to execute your use case with a network sniffer attached to make sense of the collected data. If you detect some glitches in your app you can press either the Log Slow Event button to insert a custom message you can search later for or you can press the Slow button to define a hot key which writes the Slow message every time to the ETW and network stream when you press the hotkey.

After the trap is set you can search for interesting events in the network stream. Below is an example of a network issue I found this way. There was a packet lost in the tcp stream. We see the "[TCP Previous segment not capture]" ACK message followed by another ACK which is the basic error handling in TCP to force the server to retransmit the missing packet/s. This takes time nearly 300ms as you can see which resulted in noticeable hiccups in an application.

image_thumb[5]

Now you only need to learn how to read network traces. It is now much easier to create a setup where the client/network and server is fully under observation and you only need one repro to have enough data to pinpoint the issue.

Last edited May 29, 2016 at 8:45 AM by Alois, version 4