Serious memory leak in nonpaged pool (Irp tag)

This post may not be very closely related to Sysinternals utilities, but perhaps some real Windows guru with extensive knowledge of kernel mode operations and debugging techniques will be able to help me.

I will not provide a lot of details initially, but if somebody knows the stuff inside out and is willing to help me I will share what I have done so far to troubleshoot the issue.

In short I have a serious memory leak in non paged pool which thanks to the great poolmon utility I was able to narrow down to Irp packets. I tried Driver Verifier, but it doesn't even come close to identifying the offending driver.

Here is the output from the poolmon without Driver Verifier active after about 2 days of system uptime.

 Memory:16767752K Avail: 7331916K  PageFlts:145103   InRam Krnl:14628K P:463504K
 Commit:9227568K Limit:41931724K Peak:9473216K            Pool N:6756892K P:518196K
 System pool information
 Tag  Type     Allocs            Frees            Diff       Bytes                  Per Alloc

 Irp  Nonp   14889335 ( 217)   7337908 ( 112)  7551427  5578491328 (      74240)         738
 RaME Nonp          1 (   0)         0 (   0)        1    12435456 (          0)    12435456
 File Nonp   57721607 (1579)  57690746 (1577)    30861    10268176 (        672)         332
 NDCM Nonp    1304741 (  14)   1304379 (  20)      362     7392800 (      -4016)       20422
 Ntfx Nonp     122926 (   8)     99672 (   0)    23254     7257664 (       2816)         312
 MmCa Nonp     749973 ( 116)    728471 ( 118)    21502     5459152 (       -416)         253
 PDSt Nonp     549580 (  24)    529858 (  16)    19722     4733280 (       1920)         240
 FMsl Nonp     119134 (   8)     95994 (   0)    23140     4442880 (       1536)         192
 VoSm Nonp        194 (   0)       129 (   0)       65     4097088 (          0)       63032
 Pool Nonp         15 (   0)        10 (   0)        5     3315280 (          0)      663056
 MmCi Nonp       6846 (   0)        54 (   0)     6792     3211968 (          0)         472
 SaSc Nonp     155105 (  22)    140028 (  14)    15077     3136016 (       1664)         208
 KETR Nonp       8205 (   0)         2 (   0)     8203     2803792 (          0)         341
 Tef2 Nonp        303 (   0)         0 (   0)      303     2410032 (          0)        7953
 Mm   Nonp      13177 (   0)     13163 (   0)       14     2169888 (          0)      154992
 WPSd Nonp    1264054 (  28)   1262757 (  29)     1297     2133664 (        -96)        1645
 Thre Nonp     338536 (  40)    337152 (  32)     1384     1788160 (      10368)        1292
 Even Nonp   83902172 (3738)  83892491 (3730)     9681     1247072 (       1024)         128
 MINI Nonp      22986 (  99)     22655 ( 134)      331     1208816 (      -8016)        3652
 TdxC Nonp      10525 (   0)      8750 (   0)     1775     1164400 (          0)         656
 Vad  Nonp    4680344 ( 330)   4672272 ( 334)     8072     1162368 (       -576)         144
 Hal  Nonp   54098135 ( 895)  54097814 ( 895)      321     1146320 (          0)        3571
 Mdl  Nonp      62158 (  72)     57264 (  10)     4894     1083712 (      12896)         221
 CcSc Nonp     412518 (  13)    410535 (  11)     1983     1047024 (       1056)         528
 Devi Nonp    2707976 (  30)   2707331 (  30)      645      933520 (          0)        1447
 CDmp Nonp         29 (   0)        16 (   0)       13      898160 (          0)       69089
 ALPC Nonp     188028 (   8)    186603 (  14)     1425      756048 (      -3168)         530
 ReTa Nonp      10921 (   0)       751 (   0)    10170      652608 (          0)          64
 ViMm Nonp     153056 ( 530)    151980 ( 668)     1076      645984 (     -10480)         600
 Sema Nonp     248559 ( 622)    243640 ( 622)     4919      629728 (          0)         128
 NDpp Nonp        171 (   0)         0 (   0)      171      617088 (          0)        3608
 EtwB Nonp        524 (   0)       484 (   0)       40      612400 (          0)       15310
 EtwR Nonp     103611 (   1)    100402 (   1)     3209      609920 (          0)         190
 HTab Nonp        813 (   0)       514 (   0)      299      591136 (          0)        1977
 NDnd Nonp      39645 (   0)     38585 (   0)     1060      562784 (          0)         530

There is 5.5 GB (decimal) in nonpaged Irp allocations with about 7.5 thousand Irps that were allocated and not freed. So, clearly Irp leak is the obvious problem here, but which driver is leaking them I haven't been able to pinpoint myself.

There have been a series of blogs on how to debug this type of issue on http://blogs.msdn.com/b/ntdebugging/

OK. Here is the whole story.

The system is Win7 Ultimate x64 which I primarily use to host some VirtualBox VMs. Around beginning of May I noticed that the box started to lock up and it was rock solid for almost a year before then. I discovered that the reason for lock ups is a memory leak that east up its 16 GB of memory in a few days. I started digging deeper and figured out that the nonpaged memory grows from the typical 500 MB or so to a few GBs in a couple of days. The poolmon tool pointed to Irp tag as the clear offender.

I thought it could be some update or new driver that got installed, so I restored the system from an earlier image. I went back to images as far back as November, October and September last year, but nothing helped. The system was working fine till about end of April, so I was very surprised that reverting back to the time the system was working OK did not fix the problem.

Anyway, further troubleshooting with driver verifier (log file analysis) did not point to any driver with suspicious amount of allocated memory. So the next step was to force a memory dump with driver verifier running and use !verifier kernel debugger extension to see the memory allocated.

First of all, driver verifier puts a lot of load on the system, so it stops responding after few hours with CPUs pegged at 100%. Also, the system is much slower when driver verifier is running and the memory leak happens at the much slower rate.

I recently forced a memory dump after about 8 hours of system uptime. The nonpaged memory was about 915 MB, so I'd expected a clear indication of what driver had plenty of memory allocated. Unfortunately not so. I perform the tests when the system is basically idle (no VMs are running), so the typical nonpaged memory utilization in this state is about 150 MB.

The !verifier 1 provided this output:

Verify Level 418 ... enabled options are:
    All pool allocations checked on unload
    Io subsystem checking enabled
    IRP Logging

Summary of All Verifier Statistics

RaiseIrqls                             0x0
AcquireSpinLocks                       0x18819b3d
Synch Executions                       0x7b1a30
Trims                                  0x0

Pool Allocations Attempted             0x474127c6
Pool Allocations Succeeded             0x474127c6
Pool Allocations Succeeded SpecialPool 0x47589c
Pool Allocations With NO TAG           0xa
Pool Allocations Failed                0x0
Resource Allocations Failed Deliberately   0x0

Current paged pool allocations         0x18a9a for 08CA3440 bytes
Peak paged pool allocations            0x27a61 for 0A99EE20 bytes
Current nonpaged pool allocations      0x1916e for 039C57D0 bytes
Peak nonpaged pool allocations         0x19500 for 03AECF70 bytes

Driver Verification List

Entry     State           NonPagedPool   PagedPool   Module

fffffa800cb8d880 Loaded           00036530       00000090    hal.dll
fffffa800cb91740 Loaded           00000000       00000000    kdcom.dll
fffffa800cafd200 Loaded           00000000       00000000    mcupdate.dll
fffffa800cafd040 Loaded           00000000       00000000    PSHED.dll
fffffa800cb90740 Loaded           000196c0       000f2710    CLFS.SYS
fffffa800cb90580 Loaded           00000000       00305a80    CI.dll
fffffa800cb9af50 Loaded           00064e90       00003660    Wdf01000.sys
fffffa800cb9ad90 Loaded           00000650       000002d0    WDFLDR.SYS
fffffa800cb9abd0 Loaded           00090bc0       000050a0    ACPI.sys
fffffa800cb9aa10 Loaded           00000000       00000000    WMILIB.SYS
fffffa800cb9a820 Loaded           00000000       00000000    msisadrv.sys
fffffa800cb9a660 Loaded           0000e110       00015830    pci.sys
fffffa800cb9a470 Loaded           00000000       00000000    vdrvroot.sys
fffffa800cb9a290 Loaded           00004ec0       00000080    partmgr.sys
fffffa800cb9a1b0 Loaded           00000000       00000000    compbatt.sys
fffffa800cba3e50 Loaded           000001a0       00000130    BATTC.SYS
fffffa800cba3c70 Loaded           00000110       00000500    volmgr.sys
fffffa800cba3a90 Loaded           00008140       00004050    volmgrx.sys
fffffa800cba38b0 Loaded           00000000       00000000    pciide.sys
fffffa800cba36d0 Loaded           00000000       00000050    PCIIDEX.SYS
fffffa800cba3510 Loaded           000003d0       00000000    jraid.sys
fffffa800cba3320 Loaded           00007350       00000190    SCSIPORT.SYS
fffffa800cba3130 Loaded           00000000       00002110    mountmgr.sys
fffffa800cba4f40 Loaded           00000000       00000000    vmbus.sys
fffffa800cba4d60 Loaded           000020e0       00000000    winhv.sys
fffffa800cba4b80 Loaded           00000000       00000000    atapi.sys
fffffa800cba49a0 Loaded           00008ec0       00000000    ataport.SYS
fffffa800cba47e0 Loaded           00000000       00000000    amdxata.sys
fffffa800cba4600 Loaded           00847040       00ae4440    fltmgr.sys
fffffa800cba4410 Loaded           00000ba0       00000620    fileinfo.sys
fffffa800cba4230 Loaded           0001abf0       00000a80    stcvsm.sys
fffffa800cba5fa0 Loaded           004980f0       019f7d10    Ntfs.sys
fffffa800cba5dc0 Loaded           00012250       0000b2b0    msrpc.sys
fffffa800cba5bb0 Loaded           000002a0       00004560    ksecdd.sys
fffffa800cba59d0 Loaded           00010ae0       00000070    cng.sys
fffffa800cba57f0 Loaded           00000000       00002570    pcw.sys
fffffa800cba5610 Loaded           00000020       00000000    Fs_Rec.sys
fffffa800cba5430 Loaded           005015a0       00000420    ndis.sys
fffffa800cba5250 Loaded           00125a50       00000000    NETIO.SYS
fffffa800cba5070 Loaded           00000070       00000b00    ksecpkg.sys
fffffa800cba6f40 Loaded           00109730       00000000    tcpip.sys
fffffa800cba6d50 Loaded           000002e0       000030b0    fwpkclnt.sys
fffffa800cba6b60 Loaded           00000000       00000000    vmstorfl.sys
fffffa800cba6980 Loaded           0043c6e0       000001f0    volsnap.sys
fffffa800cba67a0 Loaded           00000000       00000000    spldr.sys
fffffa800cba6580 Loaded           000325c0       00000000    rdyboost.sys
fffffa800cba63a0 Loaded           00000000       00000000    NBVol.sys
fffffa800cba61c0 Loaded           00000000       00000000    NBVolUp.sys
fffffa800cba7fa0 Loaded           00000f20       00000410    mup.sys
fffffa800cba7db0 Loaded           00000000       00000000    hwpolicy.sys
fffffa800cba7bd0 Loaded           000003e0       00000000    fvevol.sys
fffffa800cba7a10 Loaded           000000c0       00000060    disk.sys
fffffa800cba77f0 Loaded           00035680       00000200    CLASSPNP.SYS
fffffa800cba7600 Loaded           00000000       00000000    AtiPcie64.sys
fffffa800cba7410 Loaded           00000000       00000000    ahcix64s.sys
fffffa800cba7220 Loaded           00bdf820       000001b0    storport.sys
fffffa800f9035d0 Loaded&Unloaded  000dd3b0       00000000    crashdmp.sys
fffffa800f5fb900 Loaded&Unloaded  00000000       00000000    dump_storport.sys
fffffa800f906920 Loaded&Unloaded  00000000       00000000    dump_ahcix64s.sys
fffffa800f4bfe70 Loaded&Unloaded  00004010       00000000    dump_dumpfve.sys
fffffa800f958010 Loaded           000137e0       00000030    cdrom.sys
fffffa800fc59150 Loaded           00005560       0320d390    SRTSP64.SYS
fffffa800f92f7f0 Loaded&Unloaded  000016c0       00000000    EX64.SYS
fffffa800f914570 Loaded           00016b90       0000de90    SYMEVENT64x86.SYS
fffffa800f985590 Loaded&Unloaded  00000000       00000000    ENG64.SYS
fffffa800fc4b030 Loaded           00000350       00000260    SRTSPX64.SYS
fffffa800fc60030 Loaded           00000000       00000000    Null.SYS
fffffa800fc62030 Loaded           00000000       00000000    Beep.SYS
fffffa800fb6c030 Loaded           00005180       00000060    watchdog.sys
fffffa800f936560 Loaded           000000c0       00000970    VIDEOPRT.SYS
fffffa800fb5d030 Loaded           00000000       000010f0    vga.sys
fffffa800f9b3480 Loaded           00000000       00000000    RDPCDD.sys
fffffa800f998480 Loaded           00000000       00000000    rdpencdd.sys
fffffa800fc60480 Loaded           00000000       00000000    rdprefmp.sys
fffffa800fb5d490 Loaded           00000370       000005d0    Msfs.SYS
fffffa800fc5b480 Loaded           00000f40       0000ea30    Npfs.SYS
fffffa800fb72490 Loaded           00004870       00000000    TDI.SYS
fffffa800fb6e490 Loaded           00165260       00000000    tdx.sys
fffffa800faf5060 Loaded           00207000       00000000    wpsdrvnt.sys
fffffa800fac7510 Loaded           0009cac0       00005d40    afd.sys
fffffa800fa4b0c0 Loaded           00024590       00000000    netbt.sys
fffffa800fadc8c0 Loaded           000006d0       00000000    wfplwf.sys
fffffa800fb5f030 Loaded           000015b0       00000000    pacer.sys
fffffa800f4407a0 Loaded           00002b30       00000000    vpcnfltr.sys
fffffa800f8f57e0 Loaded           000011f0       00000000    netbios.sys
fffffa800fab28b0 Loaded           00000260       00000000    wanarp.sys
fffffa800fd84040 Loaded           0000a480       00000080    vpcvmm.sys
fffffa800faae500 Loaded           00000000       00000000    VBoxUSBMon.sys
fffffa800fbb2040 Loaded           0000db80       00000000    VBoxDrv.sys
fffffa800fae4510 Loaded           000042c0       00000000    termdd.sys
fffffa800fbc3030 Loaded           00000000       00000000    SCDEmu.SYS
fffffa800fbc5030 Loaded           00000000       00000020    sbmount.SYS
fffffa800fbf3030 Loaded           000064e0       00003e70    rdbss.sys
fffffa800fba94f0 Loaded           00001d10       00000000    nsiproxy.sys
fffffa800fbe20a0 Loaded           00005260       00000080    mssmbios.sys
fffffa800fbb6470 Loaded           00001010       00001cf0    eeCtrl64.sys
fffffa800fb79e30 Loaded           00000f80       0000a9e0    EraserUtilRebootDrv.sys
fffffa800fb795f0 Loaded           00000000       000003f0    discache.sys
fffffa800fada4a0 Loaded           00002070       000027e0    csc.sys
fffffa800fada570 Loaded           000003e0       00000090    dfsc.sys
fffffa800fb2f6c0 Loaded           00000000       00000000    blbdrive.sys
fffffa800fde8030 Loaded           00000000       00000000    AsUpIO.sys
fffffa800fb70490 Loaded           00000000       00000000    AsIO.sys
fffffa800fdfb030 Loaded           000000c0       00000000    tunnel.sys
fffffa800fb61aa0 Loaded           00000bc0       00000000    amdppm.sys
fffffa800fb2f600 Loaded           0000c170       0004a450    atikmpag.sys
fffffa800fe1c8b0 Loaded           00167540       0120cd70    atikmdag.sys
fffffa800fe59030 Loaded           000107b0       01d776a0    dxgkrnl.sys
fffffa800fb964a0 Loaded           000c3f10       000d5ab0    dxgmms1.sys
fffffa800fbbe490 Loaded           0000e860       000000a0    HDAudBus.sys
fffffa800fbe6c60 Loaded           00000000       00000000    USBD.SYS
fffffa800fdd3750 Loaded           0007ebc0       000000f0    nusb3xhc.sys
fffffa800fade480 Loaded           00000000       00000000    usbfilter.sys
fffffa800fe30030 Loaded           00048e80       00000150    USBPORT.SYS
fffffa800fdff490 Loaded           00000000       00000000    usbohci.sys
fffffa800fdbb8b0 Loaded           00000000       00000000    usbehci.sys
fffffa800fe57040 Loaded           00000000       00000000    ASACPI.sys
fffffa800fe26030 Loaded           00002010       00000000    i8042prt.sys
fffffa800fe99030 Loaded           000000d0       00000000    L8042Kbd.sys
fffffa800fe99090 Loaded           00001a60       00000000    kbdclass.sys
fffffa800fe69030 Loaded           000005b0       00000000    L8042mou.Sys
fffffa800fa8f030 Loaded           00000ff0       00000000    LMouKE.Sys
fffffa800fdcc8b0 Loaded           000023c0       00000000    mouclass.sys
fffffa800fe5d030 Loaded           0000da40       00000000    1394ohci.sys
fffffa800feb6030 Loaded           0009bf50       00000000    Rt64win7.sys
fffffa800fdd5490 Loaded           00000030       000000b0    wmiacpi.sys
fffffa800fdbf480 Loaded           00000000       00000000    CompositeBus.sys
fffffa800fe26980 Loaded           00000020       00000000    AgileVpn.sys
fffffa800fe34310 Loaded           00000040       00000000    rasl2tp.sys
fffffa800feac060 Loaded           00000000       00000000    ndistapi.sys
fffffa800fed5030 Loaded           00002830       00000000    ndiswan.sys
fffffa80109b1930 Loaded           00000000       00000000    raspppoe.sys
fffffa800feb4040 Loaded           00000040       00000000    raspptp.sys
fffffa800fe52490 Loaded           000003b0       00000000    rassstp.sys
fffffa800feb0560 Loaded           00000000       000000d0    teamviewervpn.sys
fffffa800fe598b0 Loaded           000003f0       00000000    VBoxNetAdp.sys
fffffa800fe9f4c0 Loaded           00000000       00000000    rdpbus.sys
fffffa800feb09b0 Loaded           00000d80       00000000    VBoxNetFlt.sys
fffffa80109e3970 Loaded           001c3c80       00000000    teefer2.sys
fffffa800feb0e70 Loaded           00001200       00002540    ks.sys
fffffa800ffaa030 Loaded           00000000       00000000    swenum.sys
fffffa800ff15040 Loaded           00000000       00000080    amdiox64.sys
fffffa800ff30040 Loaded           00001a10       00000000    umbus.sys
fffffa800ff5e030 Loaded           000002e0       000004e0    usbrpm.sys
fffffa800ff7f030 Loaded           00000000       00000000    vpcusb.sys
fffffa800ff49040 Loaded           000049e0       00000000    vpchbus.sys
fffffa800ffd7030 Loaded           00000a30       00000080    nusb3hub.sys
fffffa800ffcb030 Loaded           00019430       000001a0    usbhub.sys
fffffa8010ac8030 Loaded           00004e90       00000000    NDProxy.SYS
fffffa8010c46230 Loaded           000000c0       000000b0    drmk.sys
fffffa8010c20890 Loaded           00004930       00007920    portcls.sys
fffffa8010c3cf20 Loaded           0000bf80       00000000    RtHDMIVX.sys
fffffa8010c6b7a0 Loaded           000001d0       00000000    ksthunk.sys
fffffa8010c6b570 Loaded           000294b0       00000e60    viahduaa.sys
fffffa8010ca0480 Loaded           00002760       00000000    61883.sys
fffffa8010d1c550 Loaded           00000ed0       00000000    avc.sys
fffffa8010d71450 Loaded           00000710       00000050    STREAM.SYS
fffffa80111e1040 Loaded           00001920       00000000    msdv.sys
fffffa80116ea030 Loaded           00002e00       00000000    HIDPARSE.SYS
fffffa8010d914d0 Loaded           00006a30       00000060    HIDCLASS.SYS
fffffa800ffa8030 Loaded           00001200       00000000    hidusb.sys
fffffa80116ea280 Loaded           00000590       00000000    AmUStor.SYS
fffffa8011087030 Loaded           00000000       00000000    Dxapi.sys
fffffa8011252030 Loaded           00014520       000000c0    win32k.sys
fffffa80114561a0 Loaded           00000000       00000000    monitor.sys
fffffa8011bd4b90 Loaded&Unloaded  00000000       00000000    TSDDD.dll
fffffa800f96b060 Loaded&Unloaded  00001030       00001010    cdd.dll
fffffa801100baf0 Loaded           00000000       00017dc0    luafv.sys
fffffa80111d18c0 Loaded           00000cc0       00018460    PDFsFilter.sys
fffffa8010ffae30 Loaded           0000eaa0       00000000    WudfPf.sys
fffffa80112437e0 Loaded           00000000       00000000    DefragFS.SYS
fffffa80116c8780 Loaded           00000040       00000000    lltdio.sys
fffffa80115bc520 Loaded           00000000       00000000    pnarp.sys
fffffa8011720af0 Loaded           00000000       00000000    purendis.sys
fffffa80117bf030 Loaded           00000090       00000090    rspndr.sys
fffffa80118bb6c0 Loaded           0001cba0       000024d0    HTTP.sys
fffffa8010907a10 Loaded           00000ce0       000008d0    bowser.sys
fffffa8011520220 Loaded           000000e0       00000000    mpsdrv.sys
fffffa8010968730 Loaded           000060f0       00001120    mrxsmb.sys
fffffa8011711bb0 Loaded           00001050       00000f00    mrxsmb10.sys
fffffa80109cfaf0 Loaded           00000000       00000000    mrxsmb20.sys
fffffa8011c460e0 Loaded           00000000       00000000    AODDriver2.sys
fffffa80104ac220 Loaded&Unloaded  006fa270       00000000    WpsHelper.sys
fffffa80118dc5a0 Loaded           00000000       00000000    cpuz133_x64.sys
fffffa8011d46a40 Loaded           00000000       00000000    cpuz135_x64.sys
fffffa8012276dd0 Loaded           00000000       000000b0    peauth.sys
fffffa8010442180 Loaded           00000000       00000050    secdrv.SYS
fffffa8011c50f00 Loaded           0007ae10       00063220    srvnet.sys
fffffa8012357bb0 Loaded           00006680       00000000    tcpipreg.sys
fffffa8010c19470 Loaded           00012550       00000620    srv2.sys
fffffa8012aef8b0 Loaded           0001a3a0       000018e0    srv.sys
fffffa80123b5c00 Loaded           00000000       00000000    TuneUpUtilitiesDriver64.sys
fffffa800d6aa730 Loaded           00006210       00000000    WUDFRd.sys
fffffa8012b230b0 Loaded           000004a0       00000070    rdpdr.sys
fffffa80116da920 Loaded           00000f60       00000000    tdtcp.sys
fffffa80111febd0 Loaded           00000000       00000000    tssecsrv.sys
fffffa8011186b50 Loaded           000000a0       00002c20    RDPWD.SYS
fffffa80112a59d0 Unloaded         00000000       00000000    spsys.sys
fffffa800dfbd180 Loaded           00000110       00000000    asyncmac.sys
fffffa8012b76010 Loaded           00000000       00000000    myfault.sys

I can't provide the !verifier 3 output, because it's 12 MB, but a script I wrote to provide certain statistics from that output for Irp+ tag listed this:

Driver: CLFS.SYS
NonPagedPool: 104128 bytes
PagedPool: 993040 bytes
Tags found: 10

Driver: Wdf01000.sys
NonPagedPool: 413328 bytes
PagedPool: 13920 bytes
Tags found: 4

Driver: partmgr.sys
NonPagedPool: 20160 bytes
PagedPool: 128 bytes
Tags found: 6

Driver: SCSIPORT.SYS
NonPagedPool: 29520 bytes
PagedPool: 400 bytes
Tags found: 2

Driver: fltmgr.sys
NonPagedPool: 8679488 bytes
PagedPool: 11420736 bytes
Tags found: 1

Driver: volsnap.sys
NonPagedPool: 4441824 bytes
PagedPool: 496 bytes
Tags found: 4

Driver: CLASSPNP.SYS
NonPagedPool: 218752 bytes
PagedPool: 512 bytes
Tags found: 155

Driver: cdrom.sys
NonPagedPool: 79840 bytes
PagedPool: 48 bytes
Tags found: 2

Driver: SRTSP64.SYS
NonPagedPool: 21856 bytes
PagedPool: 52482960 bytes
Tags found: 2

Driver: netbt.sys
NonPagedPool: 148880 bytes
PagedPool: 0 bytes
Tags found: 1

Driver: LMouKE.Sys
NonPagedPool: 4080 bytes
PagedPool: 0 bytes
Tags found: 1

Driver: nusb3hub.sys
NonPagedPool: 2608 bytes
PagedPool: 128 bytes
Tags found: 1

Driver: usbhub.sys
NonPagedPool: 103472 bytes
PagedPool: 416 bytes
Tags found: 7

Driver: avc.sys
NonPagedPool: 3792 bytes
PagedPool: 0 bytes
Tags found: 2

Driver: HIDCLASS.SYS
NonPagedPool: 27184 bytes
PagedPool: 96 bytes
Tags found: 3

Driver: AmUStor.SYS
NonPagedPool: 1424 bytes
PagedPool: 0 bytes
Tags found: 1

Driver: HTTP.sys
NonPagedPool: 117664 bytes
PagedPool: 9424 bytes
Tags found: 2

Driver: mrxsmb.sys
NonPagedPool: 24816 bytes
PagedPool: 4384 bytes
Tags found: 2

Driver: srvnet.sys
NonPagedPool: 503312 bytes
PagedPool: 406048 bytes
Tags found: 101

Driver: srv2.sys
NonPagedPool: 75088 bytes
PagedPool: 1568 bytes
Tags found: 23

Driver: tdtcp.sys
NonPagedPool: 3936 bytes
PagedPool: 0 bytes
Tags found: 12

There are no Irp tags anymore with driver verifier active (well, with the options I've enabled), but Irpt and Irp+, however Irpt tags cannot be found in memory dump at all, but Irp+ tags have the majority of memory allocations anyway based of poolmon output.

So I'm at a loss right now. I've exhausted all the troubleshooting options I knew about and still I have no clue what is the damned thing that is leaking the memory. I had some wild thoughts that it could be something hardware related, but when I boot to an XP maintenance partition there is no memory leak whatsoever.

This is a remote system for me, at least for the time being. I can work with dump files and I tried live debugging, but so far with limited success. Also I have to mention that I'm not a Windows developer, although I did significant amount of C language programming, but that was more than 15 years ago and on different platform. I have some basic understanding of Windows architecture, but I'm a newbie at kernel debugger. That said I tried to configure remote debugging, but I can't get past the error that I've received. I've tried to search for it, but there was no useful hits.

C:\Windows\system32>bcdedit /dbgsettings net hostip:192.168.0.22 port:50000
The debugger type specified is not valid.
Run "bcdedit /?" for command line assistance.
The parameter is incorrect.

With some debugging techniques a live debugging session is needed and I don't know why I get the above error, because the requirements are met.

I'll take a look at xperf and see if it will help. Driver Verifier is supposed to be helpful in my situation, but unfortunately it is not. The Irp+ tag had over 600 MB allocated to it and by my quick estimation the Driver Verifier output does not account for even 10% of it.

Thanks for the tip about xperf. Any tips on the usage of it?

Edited by Russell - 12 October 2012 at 3:12am

ok, PDEngine.exe, stcvsm.sys and SRTSP64.SYS are involved.

PDEngine.exe = Perfect Disk
stcvsm.sys = ShadowProtect
SRTSP64.SYS = Norton

My guess is that ShadowProtect maybe the cause.

Also try to update the AmUStor.sys driver. It also allocates IRP tag.

Edited by MagicAndre1981 - 13 October 2012 at 6:25am