Back in my developer days when I was building DLLs and OCXs, I was constantly registering and unregistering DLLs using the command line (find me a dev who isn't a command line junkie - I still have two prompts open as I type this). As time's gone on, I submitted to using Windows Explorer more and more to do things and (this is going back many years), someone told me how to setup a shell extension to DLLs to allow you to register and unregister them directly. I guess I kept that file around all these years and it's useful to find it again now I'm installing test builds regularly. Hopefully, it's obvious how this works when you see the registry file below. Simply save as xyzzy.reg and merge it into the registry.

REGEDIT4
[HKEY_CLASSES_ROOT\.dll]
@="dllfile"
[HKEY_CLASSES_ROOT\dllfile\shell]
@="WIN_SYS"
[HKEY_CLASSES_ROOT\dllfile\shell\REG_32]
@="Register"
[HKEY_CLASSES_ROOT\dllfile\shell\REG_32\command]
@="regsvr32.exe %1"
[HKEY_CLASSES_ROOT\dllfile\shell\Unregister]
@=""
[HKEY_CLASSES_ROOT\dllfile\shell\Unregister\command]
@="regsvr32.exe /u %1"
[HKEY_CLASSES_ROOT\.ocx]
@="dllfile"

Once you click on a DLL in Windows Explorer, you'll see two new options to Register and Unregister DLLs. Normal disclaimer applys about messing up your registry if you choose to use this etc.

Today I would like to do a quick walkthrough of the VMMap tool from Windows Sysinternals.

a brief discussion of a very useful tool. You may have had the need to run a Performance Monitor log at one time or another, but noticed that a needed counter was missing from the list. This can occur if the counter has been disabled intentionally, or if the counter has been disabled by the operating system itself due to it faulting 3 times. When a counter is disabled, a simple registry value is set that tells the OS to ignore it. This value is set under the Performance key of the service in question. If the Disable Performance Counters entry is set to 1 then the counter is disabled, like so:

 

Now, you can always reenable a performance counter manually by editing the registry, but they are not always easy to find depending on the service name they are listed under. We have a tool that can come in handy in these situations, the Extensible Counter List tool, or Exctrlst.exe. This tool shows a list of all installed performance counters, sorted either by DLL name, service or Counter ID. With this, you can easily see if a counter is enabled or disabled, and set them however you wish. This utility is part of the Windows 2000 resource kit, and can also be downloaded from here:

 

https://download.microsoft.com/download/win2000platform/exctrlst/1.00.0.1/nt5/en-us/exctrlst_setup.exe

 

Once downloaded, start the installation and follow the installation wizard. The default install location for this will be:

 

C:\Program Files\Resource Kit

 

 

 

To launch the utility, just double click the executable - Exctrlst.exe.

 

NOTE: On Windows Vista and above, this must be run elevated, so right-click and select the Run As Administrator option.

 

When it opens, you should see a window very similar to this:

 

 

Now look through the list for the performance counter which you are missing. For example, if the disk counters are missing, search for PerfDisk. Once you click the performance counter you are interested in, verify if the Performance Counters Enabled box is checked or not. If the box is not checked, then the performance counter is disabled. To reenable it, all you have to do is place a check mark in the box. The registry change happens immediately, so all you have to do is close the tool when you are done.

This tool can be run on any operating system version from Windows 2000 to Windows Server 2008 R2.

I would also suggest that you have a look at the below articles as well.

https://support.microsoft.com/kb/300956

https://technet.microsoft.com/en-us/library/cc784382(WS.10).aspx

That’s all for now.

As system memory increases, and applications and roles of the server become more important, access to the disk becomes more competitive between applications and the operating system.  As a result, you can start to see the Event ID 333 logged, and you are likely seeing other performance issues including server hangs, pauses, sluggish response times and more.  So today we're going to discuss what the Event ID 333 is, possible causes and common solutions for both 32-bit and 64-bit systems.

The first thing to understand is what exactly an Event ID 333 is.  The event ID 333 is a System event error log that occurs when the registry is unable to complete a flush operation to the disk.  There are several reasons that this can fail and we'll discuss them below.  So, what does the Event ID 333 look like?  In the system event log, you see at least one, and more likely multiple instances of the following:

Event ID 333’s are new to Windows Server 2003 Service Pack 1 and are written to the system event log if the Operating System is not able to flush out or write to the registry hive.  The symptoms that accompany an Event ID 333 can vary between server hangs, “Insufficient resources exist to complete the requested service” errors, SQL Databases stopping and starting, database queries are slow and sluggish operating system performance.  The apparent slow server performance is why the Performance team is engaged on the majority of the issues.  As mentioned previously, the Event ID 333 can occur on both x86 and x64 systems.

Now that we have a little background on what the Event ID 333 is and why it exists, we’re going to take a look at the causes. There are a number of different reasons for why a server is logging the Event ID 333's - the majority of the issues are caused by one of the following:

• The disk hosting the system partition is not keeping up with the load (high disk queue lengths).  This can be determined using a Performance Monitor capture.  This can occur on both x64 and x86 systems.
• There is memory pressure.  In many cases there will also be issues with low Paged (Event 2020) or NonPaged (Event 2019) pool memory or low System PTE's.  Using Performance Monitor can help to determine if you are low on System PTE's.  This is more likely to occur on x86 systems dues to the memory limitations of the 32-bit address space
• A filter driver is keeping the registry from being flushed.  This can occur on both x64 and x86 systems
• A User account granted the “Lock system pages in memory” user right.  This can occur on both x64 and x86 systems

So - how do we go about resolving the Event ID 333 problem?  Depending on the particular scenario that is causing your issue, then you should perform one of the following:

Disk issues:

1. Enable Enhanced Performance Options for the physical disks that host the Operating System itself
2. Update your disk controller firmware and drivers.  Engage your hardware vendor to ensure that you are running the latest supported version and also to verify if there are any known issues.
3. Determine what process is taxing the disk by using Performance Monitor logging.  Examine the Process Object and review each processes I/O Data bytes/sec.  If a process exhibits an unusual amount of I/O compared to your server's baseline performance, then investigate that process.  For non-Microsoft processes, contact the software vendor.

Maximize Kernel Memory and System PTE's:

1. If you are using the /3GB switch on a 32-bit system, do you really need it?  As we discussed in our post on /3GB this switch may not be necessary. 
   • Some software vendors recommend implementing the switch for their applications - verify with them that they have the IMAGE_FILE_LARGE_ADDRESS_AWARE flag set in their image header.  If they do not, then the application is only able to use the standard 2GB of user-mode memory - but the kernel is now limited to 1GB for no reason. 
   • You can also check the virtual memory of the executable that is believed to take advantage of /3GB.  If the process is not exceeding 1.8GB of Virtual Memory then it is unlikely to be able to take advantage of the extra virtual memory.
   • If the /3GB switch is required (for example Exchange Servers that are configured as Mailbox Servers) then add the /USERVA=2970 switch to give a portion of the User-mode memory to the Kernel to help guard against PTE Depletion.  Microsoft KB Article 316739 discusses the use of the /USERVA switch.
2. Adjust the allocation and use of PagedPool memory by making some registry changes that are outlined in Microsoft KB Article 312362.
3. Disable the Hot Add Memory feature to re-allocate reserved Paged Pool memory back to the Operating System.  When the Hot Add Memory feature is enabled, the operating system pre-allocates kernel resources to handle any future memory that may be added to the computer.  Kernel resources are allocated based on the capabilities of the computer instead of on the RAM that is actually installed.  The kernel may allocate significant resources to RAM that may never be installed.  Therefore, the Hot Add Memory feature may cause the maximum size of the paged pool to be much smaller than expected.

A filter driver is preventing the registry from being flushed:

A quick note on filter drivers - removing or disabling filter drivers without understanding the impact to the system can result in unexpected system behaviors including system hangs or bugchecks.  Examples of common programs that use filter and kernel drivers include Anti-virus software, Backup Software (including Microsoft Volume Shadow Copy) and Version-2 printer drivers.  For information on how to temporarily disable filter drivers, refer to Microsoft KB Article 816071.  Before disabling the filter drivers however, check to see if some of the 3rd party drivers are simply outdated.  The easiest way to check on these 3rd party filter drivers is to use our MPS Reporting utility to capture this information.

1. Download the MPSRPT_SETUPPerf.exe file from the Microsoft Product Support Reporting Tools page to the local machine.
2. Run the MPS Reports executable
3. Once the report has finished, you can extract the PStat.txt file from the .CAB file that is created and review the list of 3rd party filter / kernel drivers to see which ones may be outdated.  Contact the vendor for the filter driver to get the latest supported version and also to discuss any potential issues with the driver.

"Lock Pages in Memory" user right:

On x64 systems, the likelihood of a server running out of kernel memory or System PTE's is far less than on an x86 system.  However, we have seen the Event 333 error occur on x64 systems as well.  Using the "!vm" debugger command when reviewing a memory dump of the server may indicate that the server is low on physical memory, even though performance monitor data indicates that there is plenty of available memory.  The sample output below illustrates this:

15: kd> !vm
*** Virtual Memory Usage ***
Physical Memory: 2095394 ( 8381576 Kb)
Page File: \??\C:\pagefile.sys
Current: 4249600 Kb Free Space: 4154172 Kb
Minimum: 4249600 Kb Maximum: 4249600 Kb
Available Pages: 868200 ( 3472800 Kb)
ResAvail Pages: 250 ( 1000 Kb)

********** Running out of physical memory **********

To work around this behavior on x64 platforms or on servers with 4GB or less of physical RAM use the following steps:

1. Click on Start --> Run
2. Type Secpol.msc and click OK
3. Expand "Local Policies" then click on "User Rights Assignment"
4. Double click on "Lock pages in memory"
5. Highlight any user accounts listed and click on "Remove"
6. Click OK after all accounts have been removed
7. Reboot the system

NOTE: By default, the Windows Operating system does not grant this user right to any accounts.  The “Lock pages in memory” right is granted to the account used for SQL Services by the SQL 2005 RTM/SP1 Enterprise Edition install on 32bit systems.  If you are using SQL Enterprise on 32-bit servers with more than 4GB of RAM, the Lock Pages in Memory right is needed.  To help reduce the occurrence of the Event ID 333's on these systems, ensure the user account you are using for the SQL services is only used for SQL. Check the access right for “Lock pages in memory” and only list the account used for SQL.  If System, or any other accounts are listed, remove them.  For x64 systems, remove all accounts listed.

Well that does it for this post. 

First of all - what are Pool Resources?  When a machine boots up, the Memory Manager creates two dynamically sized memory pools that kernel-mode components use to allocate system memory. These two pools are known as the Paged Pool and NonPaged Pool.  Each of these pools start at an initial size that is based upon the amount of physical memory present in the system.  Pool memory is a subset of available memory and is not necessarily contiguous.  If necessary, these pools can grow up to a maximum size that is determined by the system at boot time.

So - what distinguishes Paged Pool and NonPaged Pool memory?  The first difference is that Paged Pool is exactly what its name implies - it can be paged out.  The NonPaged Pool cannot be paged out.  Drivers use the NonPaged Pool for many of their requirements because they can be accessed at any Interrupt Request Level (IRQL).  The IRQL defines the hardware priority at which a processor operates at any given time (there's a link to a document covering Scheduling, Thread Context and IRQL's in the Additional Resources section at the end of this post).

Getting back to our Pool Resources, it is important to remember that these resources are finite.  The table below outlines some sample maximum values for Paged / NonPaged Pool on x86 systems that are not configured with the /3GB switch in the system's boot.ini file.  We'll cover /3GB and its effects on memory in a future post.  We'll also cover Kernel Changes to Windows Vista separately.  It's important to note that x64 systems don't suffer from the same Virtual Address Space limitations! 

Windows 2000

 * If Terminal Services is installed on Windows 2000, Paged Pool is lowered down to 160 MB unless a registry change is made to the server to set the Paged Pool Size to its maximum value (see below).

Windows 2003 SP1

On Windows 2003 systems, Terminal Services are enabled by default. 

On both Windows 2000 and Windows 2003, the HKLM\System\CurrentControlSet\Control\Session Management\Memory Management\PagedPoolSize value can be set to 0xFFFFFFFF (or resetting the value to 0) to ensure that the Virtual Address Space used for Paged Pool is maximized.

Also - here's the theoretical maximums for pre-Vista Operating Systems:

 * depends on whether or not /3GB is enabled

Now that we know what the maximum value ranges should look like, here's how to verify what those values look like on your own system using Process Explorer:

1. Download Process Explorer from the Microsoft Sysinternals Site.
2. Unzip the contents of the Zip file to C:\ProcessExplorer
3. Run ProcExp.exe
4. The first thing you'll want to do is configure the Symbols and the dbghelp.dll path.  If you leave the default path alone (see the image below), you'll get an error message that you haven't configured symbols - even if you have the Symbol path filled in (the Procexp.chm file in the ProcessExplorer folder provides instructions).
   
5. To get the most out of Process Explorer however, you'll need to install the Microsoft Debugging Tools (we'll be needing these later anyway when we start getting into troubleshooting).  This is important because you can specify the proper version of dbghelp.dll.  Once you have Process Explorer fully configured (see the image below) for the settings, you're ready to check out your Pool Resources:
   
6. Within Process Explorer, select View ... System Information ... and look at the Kernel Memory Section.  The Paged Limit and NonPaged Limit show the current maximum values for the system you are examining.
   

Now, let's examine what type of items reside in each of these pools.  Within the NonPaged pool, you would find handles that are used by applications in the user-mode space as well as Kernel-Mode drivers (typically ending in a .sys file extension).  Examples of Paged Pool items are Token Objects, Kernel-Mode drivers and the Registry.

Here's where things get really interesting - what happens to a system when these Pool Resources get depleted?  Some of the most common symptoms exhibited are:

• the machine becomes sluggish
• users can no longer log on to the machine
• console access is sluggish
• users cannot connect to file shares or other shared resources
• system hangs including the console itself being unresponsive

Symptoms such as this are usually the first indication that there is something causing an issue with the machine. 

If the NonPaged Pool on a server has become depleted, the machine will log an Event in the System Log as shown below:

Event ID 2019
Event Type: Error
Event Source: Srv
Event Category: None
Event ID: 2019
Description: The server was unable to allocate from the system NonPaged pool because the pool was empty.

Paged Pool Depletion is logged as an Event 2020:

Event ID 2020
Event Type: Error
Event Source: Srv
Event Category: None
Event ID: 2020
Description: The server was unable to allocate from the system paged pool because the pool was empty.

What are these error messages telling us beyond the fact that there is an issue with Pool Depletion?  A common misunderstanding of this message is that the problem is being caused by the Server Service (srv.sys).  Usually the Server Service is the first component to experience the issue because it is trying to satisfy a request and cannot allocate the appropriate Pool Memory

Our new blog has been launched today.

Hi all. Today I would like to bring to your attention an issue we have been seeing lately that very well may effect those of you in a corporate environment. McAfee has recently released information about this issue on their web site.

The issue is that one or multiple servers may become unresponsive or start failing in any of their installed roles. Some of the possible symptoms are:

· Slow file access
· Slow read/writes from an application
· Server unresponsive/hangs
· Slow SQL Server performance
· IIS Hangs
· Inability to connect remotely via RDP 
 

Further investigation may reveal that any number of processes are running high CPU or memory, or all combined are depleting the system of resources. It may not be evident what is causing the issue; just that many processes combined are most likely involved.

This can occur if McAfee Access Protection and Buffer Overflow Protection are installed. There is a known issue where severe performance degradation may occur during the scanning or monitoring of the following processes:

iexplore.exe
msimn.exe
svchost.exe 
explorer.exe 
mapisp32.exe 
ftp.exe
services.exe 
frameworkservice.exe 
lsass.exe 
inetinfo.exe 
outlook.exe 
wmplayer.exe 
mplayer2.exe 
rpcss.exe 
msmsgs.exe 
winword.exe 
excel.exe 
mstask.exe 
powerpnt.exe 
msaccess.exe 
visio32.exe 
wuauclt.exe 
sqlservr.exe 
dllhost.exe 
VSEBOTest.exe 
w3wp.exe 
EventParser.exe 
NaiMServ.exe 
SrvMon.exe
naPrdMgr.exe

Disabling the services does not actually remove the drivers, so you may see the issue even if you turn off the suspect functionality. The two drivers involved are:

a. MFEAPFK.SYS McAfee, Inc. Access Protection Filter Driver

b. MFEBOPK.SYS McAfee, Inc. Buffer Overflow Protection Driver

Due to the overhead placed on some applications by McAfee Access Protection and Buffer Overflow Protection, McAfee recommends disabling and removing  these to resolve performance issues. This hotfix will remove the filter drivers and disable the associated services.

For more info, please see the following articles on McAfee’s web site:

List of Processes Protected by Buffer Overflow Protection

https://kc.mcafee.com/corporate/index?page=content&id=KB58007

Access Protection and Buffer Overflow Protection drivers remain loaded when disabled

https://kc.mcafee.com/corporate/index?page=content&id=KB65820

VirusScan Enterprise and Buffer Overflow Protection (Master Article)

https://kc.mcafee.com/corporate/index?page=content&id=KB67733

 RSS feed.