The Case of the System Process CPU Spikes (by Mark Russinovich)

As you’ve probably surmised by my blog posts and other writings, I like knowing exactly what my systems are doing. I want to know if a process is running away with the CPU, causing memory pressure, or hitting the disk. Besides keeping my computers running smoothly, my vigilance sometimes helps me spot performance and reliability problems in Windows and third-party code.

The main way I keep tabs on things is to configure Process Explorer to run automatically when I log in. Whenever I configure a new computer, I add a shortcut to Process Explorer to my profile’s Start directory that includes the /t (minimize) switch. Process Explorer runs otherwise hidden with tray icon that shows a small historical view of CPU activity level. Because I want access to detailed information about system processes, as well as my own, I also specify the /e option on Vista, which causes Windows to present a UAC prompt on logon that allows me to grant Process Explorer administrative rights.

Because I keep an eye out for CPU spikes in Process Explorer’s tray icon, which show up as green or red for user-mode (application) and kernel-mode (operating system and drivers) CPU usage, respectively, I’ve identified several application bugs over the last few months. In this post, I’ll share how I used both Process Explorer and another tool, Kernrate, to identify a problem with a third-party driver and followed the problem through to a fix by the vendor.

Not long after I got a new laptop several months ago, I noticed that the system sometimes felt sluggish. Process Explorer’s tray icon corroborated my perception by displaying a mini-graph of red CPU activity. The icon opens a tooltip that reports the name of the process consuming the most CPU when you move the mouse over it, and in this case the tooltip showed the System process as being responsible:

The first few times I noticed the problem, it resolved itself shortly after and I didn’t have a chance to troubleshoot. However, I could see by opening Process Explorer’s System Information dialog that the CPU spikes were significant:

The System process is special because it doesn’t host an executable image like other processes. It exists solely to host operating system threads for the memory manager, cache manager, and other subsystems, as well as device driver threads. These threads execute entirely in kernel mode, which is why System process CPU usage shows up as red in Process Explorer’s graphs.

I suspected that a third-party device driver was the cause of the problem, so the first step in my investigation was to figure out which thread was using CPU, which would hopefully point me at the guilty party. I watched vigilantly for signs of trouble every time I switched networks and jumped the first time I saw one. Process Explorer shows the threads running in a process on the Threads page of the Process Properties dialog, so I double-clicked on the System process and switched to the Threads page the next time I noticed the CPU spike:

The “ntkrnlpa.exe” prefix on each thread’s start address identified the ones I saw at the top of the CPU usage sort order as operating system threads (Ntkrnlpa.exe is the version of the kernel loaded on 32-bit client systems that have no execute memory protection or server systems that need to address more than 4GB of memory). Because I had previously configured Process Explorer to retrieve symbols for operating system images from the Microsoft public symbol server, the thread list also showed the names of the thread start functions. The most active threads began in the ExpWorkerThread function, which means that they were worker threads that perform work on behalf of the system and device drivers. Instead of creating dedicated threads that consume memory resources, the system and drivers can throw work at the shared pool of operating system worker threads.

Unfortunately, knowing that worker threads were causing the CPU usage didn’t get me any closer to solving identifying a root cause. I really needed to know what functions the worker threads were calling, because the functions would be inside the device driver or operating system component on whose behalf the threads were running. One way to look inside a thread’s execution is to look at the thread’s stack with Process Explorer. The stack is a memory region that stores function invocations and Process Explorer will show you a thread’s stack when you select the thread press the Stack button or double-click on the thread. On Vista, however, you get this error when you try and look at the stack for threads in the System process:

The System process is a special type of process on Vista called a “protected process” that doesn’t allow any access to its threads or memory. Protected processes were introduced to support Digital Rights Management (DRM) so that hi-definition content providers can store content encryption keys with a reduced risk of an administrative user using DRM-stripping tools to reach into the process and read the keys.

That approach foiled, I had to find another way to see what the worker threads were doing. For that, I turned to KernRate, a command-line profiling tool that’s a free download from Microsoft. KernRate can profile user-mode processes and kernel-mode threads. It uses the sample-based profiling facility that was introduced in the first release of Windows NT, which records the unique addresses at which the CPU is executing when the profiling interval timer fires. When you stop a profile capture, Kernrate retrieves the information from the kernel, maps the addresses to the loaded device drivers into which the fall, and can even use the symbol engine to report the names of functions.

I wouldn’t need symbols if the trace identified a device driver, so I ran Kernrate without passing it any arguments. Despite the fact that there’s no officially supported version of Kernrate for Vista, the version for Windows XP, Kernrate_i386_XP.exe, works on Vista 32-bit (you can also use the recently-released xperf tool to perform similar profiling - xperf requires Vista or Server 2008, but works on 64-bit versions). I let the profile run through heavy bursts of CPU and then hit Ctrl+C to print the results to the console window:

In first place were hits in the kernel, but in second was a driver that I didn’t recognize, b57nd60x. Most driver files are located in the %systemroot%\system32\drivers directory, so I could have opened that folder and viewed the file’s properties in Explorer, but I had Process Explorer open so a quicker way to check the driver’s vendor and version was to open the DLL view for the System process. The DLL view shows the DLLs and files mapped into the address space of user-mode processes, but for the System process it shows the kernel modules, including drivers, loaded on the system. The DLL view revealed that the driver was for my laptop’s NIC, was from Broadcom, and was version 10.10:

Now that I knew that the Broadcom driver was causing the CPU usage, the next step was to see if there was a newer version available. I went to Dell’s download page for my system, but didn’t find anything. Suspecting that what I noticed might not be a known issue, I decided to notify Broadcom. I used contacts on the hardware ecosystem team here at Microsoft to find the Broadcom driver representative and email him a detailed description of the symptoms and my investigation. He forwarded my email to the driver developer, who acknowledged that they didn’t know the cause and within a few days sent me a debug version of the driver with symbols so that I could capture a Kernrate profile that would tell them what functions in the driver were active during the spikes. The problem reoccurred a few days later and I sent back the kernrate output with function information.

The developer explained that my trace revealed that the driver didn’t efficiently interact with the PCIe bus when processing specific queries and the problem seemed to be exacerbated by my particular hardware configuration. He gave me new driver for me to try and after a few weeks of monitoring my laptop closely for issues, I confirmed that the problem appeared to be resolved. The updated driver has not yet been posted to Dell’s support site, but I expect it to show up there in the near future. Another case closed, this time with Process Explorer, Kernrate, and a helpful Broadcom driver developer.

If you like these troubleshooting blog posts, you’ll enjoy the webcast of my “Case of the Unexplained…” session from TechEd/ITforum. Its 75 minutes are packed with real-world troubleshooting examples, including the one written up in this post and others, as well as some that I haven’t documented. At the end of the session I ask the audience to send me screenshots, log files and descriptions of their own troubleshooting success stories, in return for which I’ll send back a signed copy of Windows Internals. The offer stands, so remember to document your investigation and you can get a free book. I’ve gotten a number of great examples and my next blog post will be a guest post by someone that watched the webcast and used Process Monitor to solve a problem with their web server.

Finally, if you want to see me speak live, come to TechEd US/IT Pro in June in Orlando where I’ll be delivering “The Case of the Unexplained…”, “Windows Server 2008 Kernel Advances”, and “Windows Security Boundaries”. Hope to see you there!

Wireless Java and J2ME Tutorials

Exploring J2ME: Building an expense tracker The Java 2 Micro Edition (J2ME), coupled with the CLDC and MIDP specifications, provides a platform that enables developers to build Java applications that run on PDAs and BlackBerry-style e-mail pagers and that interface with corporate software "back home" via a wireless connection. This article kicks off a series that will introduce you to development on the MIDP platform.

Exploring J2ME: Building an expense detail form the detail form used to edit an existing expense item or add a new one and let you get your feet wet with the Item family of components.

Exploring J2ME: Using the Record Management System creating a record store for the ExpensesApp MIDlet suite using the MIDP’s Record Management System (RMS).

Exploring J2ME: Sorting records the RMS’s record-sorting API, the RecordEnumeration class and RecordComparator interface, both of which you find in the javax.microedition.rms package.

Exploring J2ME: Being a good mobile citizen polish up ExpensesApp a bit by reducing its memory requirements and correcting some data problems using the record change notification API.

Java Programming for Wireless devices using J2ME/CLDC/MIDP
J2ME Overview, J2ME Building Blocks: Virtual Machine, Configurations and Profiles, Connected Limited Device Configuration (CLDC), Mobile Information Device Profile (MIDP), MIDP Applications (MIDlets) and MIDlet Lifecycle, Writing a simple MIDP Application...

Running J2ME Applications on Palm-powered Devices Learn how to migrate your Enterprise J2ME applications to Palm-Powered devices including the Treo 650 and Palm LifeDrive.

J2ME for the Java Developer How does Sun's Java 2 Micro Edition (J2ME) platform for smaller devices compare to the more familiar Java 2 Standard Edition? Let's dig into the details a little...

Deploy Your J2ME Application on a Sprint Phone Web server configuration and caching problems are just two of the obstacles you'll need to overcome when you deploy Java applications on cell phones. This article provides the workarounds and solutions you'll need to overcome these bumps.

Don's Wireless Journal Java ME programming tutorials, mostly geared towards game programming.

BREWing Java code Can Java code be enhanced by good BREW in wireless development? Recently IBM introduced a single JVM (Java Virtual Machine), which acts as an extension to BREW. Java developers can take advantage of this JVM to write Java applications once and run them on all BREW-enabled devices as long as they have sufficient memory footprints, regardless of manufacturer. The BREW platform is standardized, so porting applications from one device to another is simplified. BREW also makes getting Java applications onto handsets much easier, and enables upgrading and recalling JVMs over the air.

Java 2 Micro Edition Basics (PDF) A Very Brief History of Java, Architecture of Java, Java 2, Java Community Process, Java 2 Micro Edition, Profiles, Connected Limited Device Configuration (CLDC), Supported J2SE Classes, Generic Connection Framework...

Programming Strategies for Small Devices (PDF) If in Doubt, Do Not Use Java, Move Computation to the Server, Simplify the Application, Build Smaller Applications, Remove the Public Members, Use Less Memory at Run Time, Use Scalar Types, Do Not Depend on the Garbage Collector, Help the Garbage Collector, Use Lazy Instantiation, Release Resources Early, Reuse Objects, Avoid Exceptions, Code with Performance in Mind, Use Local Variables, Avoid String Concatenation, Use Threads, but Avoid Synchronization, Separate the Model...

Java 2 Micro Edition (J2ME) Specifications (PDF) Introducing the Micro Edition, A New Virtual Machine, New and Changed Classes, Configurations and Profiles, Building Blocks, The KVM, Related Technologies,

Dealing With Dates and Times in MIDP Applications Parsing and displaying dates and times is often complicated because of formatting and locale issues. J2SE provides several classes to simplify date and time handling -- classes such as java.util.Calendar, java.util.Date, java.util.TimeZone, and java.text.DateFormat. By comparison, MIDP defines only subsets of the Calendar, Date and TimeZone classes, and does not include any form of DateFormat...

Developing Custom Components for the Mobile Information Device Profile A question that novice MIDP programmers often ask is "How can I extend the MIDP user interface classes to define my own custom components?" You might want to extend the TextField class, for example, to support more kinds of constraints or alternate forms of input. Or you might want to define a completely new kind of UI component...

Using Passwords to Protect Your MIDlets Security is always a concern when you are writing an application that deals with sensitive data. It's especially so on handheld devices, which are more apt to be lost or stolen than a desktop computer. A handheld device is likely to hold truly personal information that you don't want strangers to know, things such as important phone numbers and addresses. Safeguarding that information should always be a priority...

Adapting Images for MIDP Devices The MIDP supports the display of bitmapped images, but not with the flexibility of J2SE. The MIDP specification only requires support for a single format, the PNG format. PNG was developed primarily as a replacement for the older GIF, and has the support of the World Wide Web consortium. Modern web browsers and image editors can load and save images in PNG format..

Making HTTP Connections Using Background Threads All user interface frameworks share some basic concepts. This is true no matter if the UI framework is the LCDUI classes defined by the MIDP, or the AWT and Swing classes defined by J2SE. One of the basic concepts is that the user interface must always be responsive to the user's interaction...

Using MIDP Gauges The MIDP defines a number of high-level user interface components. Most of these components are designed to be placed on a top-level window called a form. Such components are referred to as items, because they all extend the javax.microedition.lcdui.Item class. The form itself is an instance of the javax.microedition.lcdui.Form class, and acts primarily as a container for items...

Building Splash Screens for MIDlets A splash screen is an informational screen that presents the name and version of an application, along with any necessary legal information such as the copyright and trademark notices. It's something you might want your MIDlets to display on startup. Adding a splash screen to a MIDlet is easy to do, but there are a few things to watch out for...

Client-Server Communication over HTTP using MIDP and Servlets Many, if not most, MIDP applications have a server-side piece to them. In other words, a client-server paradigm is used to offload complicated work from the limited capabilities of the MIDP device to the more capable server environment. Because the MIDP 1.0 specification only mandates support for HTTP, all client-server communication must be done through an HTTP gateway...

Drawing Flicker-Free Graphics with the MIDP The term "double buffering" refers to a common technique used in computer graphics. If you consider a device's display to be a memory buffer into which drawing primitives write (the drawing primitives are the basic drawing methods such as drawLine and drawArc), with double buffering you draw into a second, offscreen memory buffer and then copy the entire contents of the second buffer into the display buffer...

Parsing XML in CLDC-based Profiles XML is a portable, text-based way of representing structured data. More and more applications are using XML to exchange information, and at some point your CLDC-based applications are going to need to process XML documents. Over time, even HTML pages will likely migrate to an XML-based format called XHTML...

Using the MIDP List Component The MIDP groups its user interface components into two sets of classes: the high-level API and the low-level API. With the high-level API you define your user interface using a set of platform-independent abstractions and let the MIDP implementation decide the look and behavior of the user interface...

Enumerating, Filtering, and Sorting MIDP Record Stores The RMS, part of the MIDP, lets you store arbitrary byte arrays as records in a database. The set of records is referred to as a record store. The RMS assigns a record ID to each record when it is created. The record ID is an integer greater than or equal to 1. The application uses this ID to get and set the content of a record...

The Connected Device Configuration and the Foundation Profile Much of what's been written about J2ME talks about the Connected Limited Device Configuration. The CLDC defines an extremely small subset of the Java Platform for very limited devices -- devices, like cellular telephones, that do not have enough memory or enough processing power to handle a full-blown Java implementation. There is, however, another configuration available that is midway between the CLDC and a full J2SE implementation: the Connected Device Profile...

Handling Multiple Simultaneous MIDP Alerts If you write a multithreaded MIDP application it's easy to trigger two or more alerts simultaneously. In other words, while one alert is being displayed, a second alert is triggered. Dealing with this situation is not as simple as you might think. Alerts are displayed using the two-argument form of the Display object's setCurrent method...

Using Timers One of the improvements in version 1.3 of J2SE are classes that make it simpler to schedule tasks for execution by a background thread. The MIDP also includes these new classes, so J2ME developers also benefit...

An Introduction to the High-Level User Interface API: Alerts and Tickers The MIDP includes both a low-level UI API and a high-level UI API. The low-level API gives you complete access to a device's screen and to raw key and pointer events. However, no user interface controls are available with the low-level API -- your application must explicitly draw buttons and other familiar controls. This is the price you pay for the flexibility of the low-level API...

Using Record Stores Efficiently The data in a record store is stored in chunks called records. A record is a set of arbitrary binary data, basically an array of bytes. You always read or write an entire record at once, using the getRecord, addRecord and setRecord methods of the RecordStore class...

Using the MIDP Low-Level User Interface API With J2ME, profiles are responsible for defining the user interface (UI) API. The MIDP defines two such APIs, referred to as the high-level and low-level APIs. The high-level API requires you to use task-oriented abstractions to define what the user interface does. You have no real control over what gets drawn on the screen -- the implementation selects the best approach for the device...

Making Connections With the CLDC As you can tell from its name, the Connected Limited Device Configuration (CLDC) is designed for connected devices. A connected device has the ability to connect to the outside world, whether the connection is serial, infrared, wireless, or something else entirely. How then do you take advantage of this connectivity in your Java applications? The answer is: through a set of classes called the Generic Connection framework, or GCF for short.

Record Management System Basics The Mobile Information Device Profile (MIDP) defines a set of classes for storing and retrieving data. These classes are called the Record Management System (RMS). With the RMS, it's possible to make data persist across invocations of a MIDlet. Different MIDlets in the same MIDlet suite can also use the RMS to share data...

Writing World-Aware J2ME Applications Well-written applications are both internationalized and localized. An application is internationalized, if it can correctly handle different encodings of character data. An application is localized, if it formats and interprets data (dates, times, timezones, currencies, messages and so on) according to rules specific to the user's locale. An application that is both internationalized and localized, can be referred to as "world-aware."...

Dealing With Missing J2SE Classes A problem you might run into when writing J2ME applications is how to deal with missing classes, that is, classes that are available in J2SE but not in J2ME. Note that J2ME configurations and profiles include subsets of the J2SE core classes. This means that specific methods or classes that you would normally use when writing a J2SE application are not always available. Ideally, you want to maximize the amount of code that is portable between your J2ME and J2SE applications...

The J2ME Development Cycle J2ME is a set of technologies and specifications that enable Java applications to run on devices such as cell phones, Personal Digital Assistants, or set top boxes, that are unable to accommodate the J2SE runtime environment. These devices have severe resource constraints when compared to desktop or server computers, limitations that kept the Java Platform off these devices until now...

Making HTTP Connections with MIDP HTTP connectivity is one of the requirements of the MIDP. A MIDP-enabled device must be able to interact with a web server through conventional HTTP requests. If the network doesn't directly support HTTP, then the device must route its requests through a gateway. But this is all transparent to the application developer...

Getting Started with Java on PDAs (2/02) PDAs are becoming a permanent fixture in the everyday lives of consumers and business people. There's no question that we have Palm to thank for bringing us a small, pen-based, easy-to-use organizer to help keep our busy lives on track. At the same time, Microsoft has been trying to hit one out of the park for years with its Windows CE operating system running on a variety of handheld devices. J2ME Tutorial, Part 1: Creating MIDlets Java 2 Micro Edition (J2ME) combines a resource-constrained JVM and a set of Java APIs for developing applications for mobile devices. This article is the first in a series. This time, after a quick introduction to J2ME, I will provide a step-by-step guide to creating J2ME applications, also known as MIDlets, using a simple example. This will cover how to test and deploy these MIDlets as well. Finally, I will round out this installment with a look at the lifecycle of a MIDlet. JavaMobiles.com Do you need a mobile telephone with Java? Are you searching for Java virtual machine for your PDA? You want to know what is this wireless Java about? You will find answers to all these questions here, at JavaMobiles.com!

Develop wireless applications using J2ME Update on Java support for mobile phones, two-way pagers, and related consumer products. Learn more about programming mobile devices and consumer gadgets of all sorts in Bill Day's report from JavaOne on the Java 2 Platform, Micro Edition.

Track wireless sessions with J2ME/MIDP (4/02) Every e-commerce application must support session tracking. Unfortunately, MIDP (Mobile Information Device Profile), a J2ME (Java 2 Platform, Micro Edition) technology, supports only the standard HTTP protocol, which is stateless. In this article, Michael Juntao Yuan and Ju Long explore ways to add session support into the current MIDP network API framework...

SyncML and Micro Java Development When I heard the crunch of my phone underneath the tire of the car I had two concurrent thoughts blare in my head: 1. You Idiot! Why didn't you make sure that you left your phone in the car when you got out to push it out of the snow bank? 2. I wish all that info I spent hours keying into my phone was wirelessly synchronized to my office network. Until mobile devices have the ability to be constantly connected to a network, and are made of the same substance as the famous "black box", synchronization is an integral component of the wireless application architecture.

Introduction To Midlets And The Display A MIDlet is a Java application designed to run on a mobile device. There are two core Java libraries that make up the develop-ment environment for these mobile applications, the Connected, Limited Device Configuration (CLDC) and the Mobile Information Device Profile (MIDP).

Event Handling Through the Item Object Event processing is essential to most every MIDlet. It's hard to imagine an application where there is no user interaction what-so-ever, even if this interaction is as simple as requesting to start or stop a MIDlet. In this article we'll learn the specifics of event handling using the MIDP Command object.

Nokia MIDP Example Applications Animated Example MIDlet, Networked MIDlet Example, Workflow Application Example, Game MIDlet Example Using The Nokia UI API, Nokia UI API Sound Example...

Developer Platforms: Guidelines For Testing J2ME Applications v1.0 This document lists fundamental issues that must be taken into account when testing J2ME™ MIDP applications.

Porting BREW Games to Java MIDP v1.0 This document assists developers who want to port BREW games to the J2ME platform, and who want to plan the development of BREW titles to make it easier to port them.

Java 2ME and MIDP Development Welcome to the world of MIDlets. A MIDlet is a Java application that conforms to the specifications set out by the Connected, Limited Device Configuration (CLDC) and the Mobile Information Device Profile (MIDP)...

Your First Micro Java Game (12/02) Mobile phone manufacturers have embraced Java in a way that not even PC manufacturers have. Java is clearly one of the platforms of choice for mobile devices, and an ideal language for throwing together mobile games. The Mobile Information Device Profile (MIDP) of J2ME is a subset of the standard Java you know and love, with a few minor modifications. Writing a basic MIDlet game is simple as apple pie. This article will show you how to start cookin'.

Personal Basis Profile vs. Personal Profile: What's the Difference? (5/03) For a long while, all the excitement in J2ME was centered on the Connected Limited Device Configuration (CLDC), released in final form in May, 2000. The first CLDC-based profile, the MIDP, followed two months later. MIDP garnered much interest in the Java community because it defined a new application model, as well as classes for user interface and persistence. In other words, it provided a complete environment for deploying and running interactive applications.

J2ME Low-Level Network Programming with MIDP 2.0 (4/03) Version 1.0 of the Mobile Information Device Profile (MIDP) lacks low-level networking support for TCP/IP sockets and UDP/IP datagrams, but the MIDP 2.0 specification (JSR 118) has responded to the needs of the 2.5G and 3G networks now being deployed by adding support for sockets and datagrams, thus providing mobile applications more capable networking interfaces.

Wireless Software Design Techniques What every wireless software developer should know (5/02) Developing applications for devices with limited resources requires skillful developers who are willing to learn and understand how the wireless environment operates, and able to make intelligent decisions regarding software architecture and performance issues. The purpose of this article is to enlighten you about concerns you should keep in mind when developing J2ME applications.

MIDP Code Samples This page provides links to a variety of code samples that use the MIDP APIs. M-Commerce, Alerts and Timers, Event Handling, Internationalization, Network Connections, Persistent Storage, Getting Started, Servlets, User Interface...

J2ME for Home Appliances and Consumer Electronic Devices (1/03) Describes the CDC, related profiles, and how to develop applications for consumer and embedded devices.

The Internet Developer's Guide to J2ME One of your jobs as a Web developer is to track up-and-coming technologies that affect how you write and deploy applications. You're probably familiar with Java as a server-side development language even though most people turn to DHTML instead of Java for building client applications. However, a new kind of client-based Java is on the horizon. It's the Java 2 Micro Edition, or J2ME for short. J2ME has the chance to succeed where applets failed.

Network Programming with J2ME Wireless Devices The wireless devices such as cell phones and two-way pagers keep their owners connected to the outside world at anytime from anywhere. They offer great connectivity that other types of devices couldn’t offer. Application development for these wireless devices is going to be in great demand for the next couple years. Network programming plays an important role in wireless application development to take advantage of the connectivity these devices have to offer

Wisdom works everywhere

An old man lived alone in Minnesota . He wanted to spade his potato garden, but it was very hard work. His only son, who would have helped him, was in prison. The old man wrote a letter to his son and mentioned his situation:

Dear Son,

I am feeling pretty bad because it looks like I won't be able to plant my potato garden this year. I hate to miss doing the ga rden, because your mother always loved planting time.

I'm just getting too old to be digging up a garden plot. If you were here, all my troubles would be over. I know you would dig the plot for me, if you weren't in prison.

Love,

Dad

Shortly, the old man received this telegram:

For Heaven's sake, Dad, don't dig up the garden!! That's where I buried the GUNS!!

At 4 a.m. the next morning, a dozen police officers showed up and dug up the entire garden without finding any guns.

Confused, the old man wrote another note to his son telling him what happened, and asked him what to do next.

His son's reply was:

'Go ahead and plant your potatoes, Dad, It's the best I could do for you from here.'