Sysinternals Homepage
Forum Home Forum Home > Windows Discussions > Internals
  New Posts New Posts RSS Feed - NTFS does not seem to release memory pool alloc.
  FAQ FAQ  Forum Search   Events   Register Register  Login Login

NTFS does not seem to release memory pool alloc.
 Post Reply Post Reply Page  12>
Author
Message
  Topic Search Topic Search  Topic Options Topic Options
dirbase View Drop Down
Senior Member
Senior Member
Avatar

Joined: 26 March 2008
Status: Offline
Points: 496 		Post Options Post Options   Thanks (0) Thanks(0)   Quote dirbase Quote  Post ReplyReply Direct Link To This Post Topic: NTFS does not seem to release memory pool alloc.
    Posted: 13 January 2009 at 9:43pm

Hi all,

On two different systems running XP Pro SP3, it seems that NTFS.sys is not releasing the memory pool allocations (both paged and non paged) it makes during heavy file activity.

 I run the following test procedure:

  1. Reboot
  2. Start perfmon with both "memory/pool paged bytes" and "memory/pool non paged bytes" counters
  3. Set perfmon refresh rate at 30 seconds and the vertical scale between 120 and 2000.
  4. Wait some minutes for the levels to stabilize.
  5. Run a string search on the NTFS main partition

I then observe the following trend:



(the red line shows paged pool bytes and the brown line the non paged pool bytes. The vertical scale is in 0.1 Mb units from 120 to 2000. The total time span is 50 minutes.

AFAICT, the driver responsible for making the sharp rise in pool allocations is NTFS.sys (identified via Poolmon with tags such as Ntfr (nonpaged), NtFs (both paged and nonpaged), Ntfn (nonpaged), File (non paged) and Ntf0 (paged); their poolmon "diff" values rising sharply  - this is also confirmed by using verifier / pool track).

NTFS does not seem to release these allocations even after leaving the system idle for several hours.

Further heavy file activity like another string search will again translate in sharp increase in memory pools allocations

(Antivirus have been uninstalled on both systems for these tests since there have been reports of AV inducing increased memory pool demands)

I have two questions: can anyone confirm this behavior (is it "by design")? and is there a way to have NTFS.sys release these allocations ?

Thanks for your time!
Back to Top
molotov View Drop Down
Moderator Group
Moderator Group
Avatar

Joined: 04 October 2006
Status: Offline
Points: 17506 		Post Options Post Options   Thanks (0) Thanks(0)   Quote molotov Quote  Post ReplyReply Direct Link To This Post Posted: 14 January 2009 at 2:40pm
Hi dirbase,

Quote Run a string search on the NTFS main partition
How much data, and how many files, are being sifted through?

Have you tried the experiment on an XP SP2 system?  (I'm in the middle of a test on an SP2 system.)
Daily affirmation:
net helpmsg 4006
Back to Top
dirbase View Drop Down
Senior Member
Senior Member
Avatar

Joined: 26 March 2008
Status: Offline
Points: 496 		Post Options Post Options   Thanks (0) Thanks(0)   Quote dirbase Quote  Post ReplyReply Direct Link To This Post Posted: 14 January 2009 at 9:06pm
Hi Molotov,

Thanks for your interest!
The string search is actually a file search using the Windows Explorer file search wizard, searching for all files containing a given string (I used "Ntfr"Wink) in my C: partition which contains the system. It has about 28 GB used out of 46 GB (representing some 60500 files).

It seems that I was a bit pessimistic in my previous post when I said "Further heavy file activity like another string search will again translate in sharp increase in memory pools allocations". In fact a second and a third identical search have shown that the paged pool used, after a sharp increase during the first search, tends to stabilize at around 175 MB (starting from about 40 MB). The non paged pool used also stabilized during a second and a third identical search at about 40 MB (from a start at 10 MB).

Please have a look at the figure below:



Here is a !vm output prior to the searches:

*** Virtual Memory Usage ***
    Physical Memory:      523771 (   2095084 Kb)
    Page File: \??\C:\pagefile.sys
      Current:   1740800 Kb  Free Space:   1708832 Kb
      Minimum:   1740800 Kb  Maximum:      1740800 Kb
    Available Pages:      427661 (   1710644 Kb)
    ResAvail Pages:       443253 (   1773012 Kb)
    Locked IO Pages:         144 (       576 Kb)
    Free System PTEs:     174674 (    698696 Kb)
    Free NP PTEs:          32766 (    131064 Kb)
    Free Special NP:           0 (         0 Kb)
    Modified Pages:          690 (      2760 Kb)
    Modified PF Pages:       674 (      2696 Kb)
    NonPagedPool Usage:     2580 (     10320 Kb)
    NonPagedPool Max:      65536 (    262144 Kb)
    PagedPool 0 Usage:      6596 (     26384 Kb)
    PagedPool 1 Usage:       654 (      2616 Kb)
    PagedPool 2 Usage:       646 (      2584 Kb)
    PagedPool 3 Usage:       648 (      2592 Kb)
    PagedPool 4 Usage:       660 (      2640 Kb)
    PagedPool Usage:        9204 (     36816 Kb)
    PagedPool Maximum:     92160 (    368640 Kb)
    Shared Commit:          1647 (      6588 Kb)
    Special Pool:              0 (         0 Kb)
    Shared Process:         2656 (     10624 Kb)
    PagedPool Commit:       9204 (     36816 Kb)
    Driver Commit:          3080 (     12320 Kb)
    Committed pages:       60712 (    242848 Kb)
    Commit limit:         919748 (   3678992 Kb)

    Total Private:         42217 (    168868 Kb)

Same after the first search:
*** Virtual Memory Usage ***
    Physical Memory:      523771 (   2095084 Kb)
    Page File: \??\C:\pagefile.sys
      Current:   1740800 Kb  Free Space:   1689316 Kb
      Minimum:   1740800 Kb  Maximum:      1740800 Kb
    Available Pages:      386966 (   1547864 Kb)
    ResAvail Pages:       442468 (   1769872 Kb)
    Locked IO Pages:         147 (       588 Kb)
    Free System PTEs:     174717 (    698868 Kb)
    Free NP PTEs:          32766 (    131064 Kb)
    Free Special NP:           0 (         0 Kb)
    Modified Pages:          741 (      2964 Kb)
    Modified PF Pages:       724 (      2896 Kb)
    NonPagedPool Usage:     9150 (     36600 Kb)
    NonPagedPool Max:      65536 (    262144 Kb)
    PagedPool 0 Usage:     16086 (     64344 Kb)
    PagedPool 1 Usage:      6472 (     25888 Kb)
    PagedPool 2 Usage:      6447 (     25788 Kb)
    PagedPool 3 Usage:      6421 (     25684 Kb)
    PagedPool 4 Usage:      6426 (     25704 Kb)
    PagedPool Usage:       41852 (    167408 Kb)
    PagedPool Maximum:     92160 (    368640 Kb)
    Shared Commit:           986 (      3944 Kb)
    Special Pool:              0 (         0 Kb)
    Shared Process:         3261 (     13044 Kb)
    PagedPool Commit:      41857 (    167428 Kb)
    Driver Commit:          3039 (     12156 Kb)
    Committed pages:      102976 (    411904 Kb)
    Commit limit:         919748 (   3678992 Kb)

    Total Private:         51839 (    207356 Kb)

Same after the third search:
*** Virtual Memory Usage ***
    Physical Memory:      523771 (   2095084 Kb)
    Page File: \??\C:\pagefile.sys
      Current:   1740800 Kb  Free Space:   1685236 Kb
      Minimum:   1740800 Kb  Maximum:      1740800 Kb
    Available Pages:      385222 (   1540888 Kb)
    ResAvail Pages:       440738 (   1762952 Kb)
    Locked IO Pages:         147 (       588 Kb)
    Free System PTEs:     174674 (    698696 Kb)
    Free NP PTEs:          32766 (    131064 Kb)
    Free Special NP:           0 (         0 Kb)
    Modified Pages:          378 (      1512 Kb)
    Modified PF Pages:       364 (      1456 Kb)
    NonPagedPool Usage:    10021 (     40084 Kb)
    NonPagedPool Max:      65536 (    262144 Kb)
    PagedPool 0 Usage:     13960 (     55840 Kb)
    PagedPool 1 Usage:      7154 (     28616 Kb)
    PagedPool 2 Usage:      7182 (     28728 Kb)
    PagedPool 3 Usage:      7137 (     28548 Kb)
    PagedPool 4 Usage:      7137 (     28548 Kb)
    PagedPool Usage:       42570 (    170280 Kb)
    PagedPool Maximum:     92160 (    368640 Kb)
    Shared Commit:           988 (      3952 Kb)
    Special Pool:              0 (         0 Kb)
    Shared Process:         4646 (     18584 Kb)
    PagedPool Commit:      42570 (    170280 Kb)
    Driver Commit:          3080 (     12320 Kb)
    Committed pages:      107265 (    429060 Kb)
    Commit limit:         919748 (   3678992 Kb)

    Total Private:         53980 (    215920 Kb)

Although it does not look like a memory leak since the pool used levels stabilized, I would prefer those ressources to be released!Confused

FYI, here is an output from Poolmon after the three searches: (the box used is a dualcore laptop XP Pro SP3)-sorted by "diff".

 Memory: 2095084K Avail: 1381696K  PageFlts:     0   InRam Krnl: 2124K P:163576K
 Commit: 629396K Limit:3678992K Peak:1113376K            Pool N:41332K     P:170556K
 System pool information
 Tag  Type     Allocs            Frees                  Diff      Bytes         Per Alloc           Mapped_Driver

 Ntfr Nonp       220303 (   0)    129290 (   0)    91013    5825288 (     0)     64       [ntfs.sys     -     ERESOURCE]
 NtFs Paged     195593 (   0)    135517 (   0)    60076    4171832 (     0)     69       [ntfs.sys     -     StrucSup.c]
 MmSt Paged    289274 (   0)    238569 (   0)    50705    55037416 (     0)   1085    [nt!mm        - Mm section object prototype PTEs]
Originally posted by molotov molotov wrote:

Have you tried the experiment on an XP SP2 system?
No, but I could do it at a later time. BTW, could you please indicate if you have seen a different behavior with SP2?

Edited by dirbase - 22 March 2009 at 7:32pm
Back to Top
molotov View Drop Down
Moderator Group
Moderator Group
Avatar

Joined: 04 October 2006
Status: Offline
Points: 17506 		Post Options Post Options   Thanks (0) Thanks(0)   Quote molotov Quote  Post ReplyReply Direct Link To This Post Posted: 19 January 2009 at 1:35pm
Haven't had adequate time or systems that were sufficiently "idle", to work with. Embarrassed Perhaps in the coming days... 
Daily affirmation:
net helpmsg 4006
Back to Top
dirbase View Drop Down
Senior Member
Senior Member
Avatar

Joined: 26 March 2008
Status: Offline
Points: 496 		Post Options Post Options   Thanks (0) Thanks(0)   Quote dirbase Quote  Post ReplyReply Direct Link To This Post Posted: 19 January 2009 at 5:09pm
OK, no problem. Thanks again for your time! I intend to try it myself with SP2 when I get the time, so we can compare the resultsSmile
Back to Top
dirbase View Drop Down
Senior Member
Senior Member
Avatar

Joined: 26 March 2008
Status: Offline
Points: 496 		Post Options Post Options   Thanks (0) Thanks(0)   Quote dirbase Quote  Post ReplyReply Direct Link To This Post Posted: 21 January 2009 at 9:13pm
I finally ran a test on an XP Home SP2 system. The results are similar to those with SP3: a file search on the hard disk (5 Gb used out of 18 Gb) based on a string search leads to a rapid growth in the paged pool and a limited growth in the non paged pool. Both  resources are not released after the test is stopped.
One difference between SP2 and SP3 seems noticeable: The  non paged pool seems to reach a steady state and stay there with SP2, while with SP3 it went on growing in small steps.

Here is a picture from perfmon with SP2 (paged pool used in 0.1 Mb units is the upper red curve ; non paged pool used is the lower brown curve). The time span is 50 minutes.


Here are the values from !vm before the test and one hour after the test was started.

before :
*** Virtual Memory Usage ***
    Physical Memory:      194299 (    777196 Kb)
    Page File: \??\D:\pagefile.sys
      Current:    512000 Kb  Free Space:    506152 Kb
      Minimum:    512000 Kb  Maximum:       512000 Kb
    Available Pages:      131187 (    524748 Kb)
    ResAvail Pages:       143095 (    572380 Kb)
    Locked IO Pages:          58 (       232 Kb)
    Free System PTEs:     239861 (    959444 Kb)
    Free NP PTEs:          32636 (    130544 Kb)
    Free Special NP:           0 (         0 Kb)
    Modified Pages:          706 (      2824 Kb)
    Modified PF Pages:       707 (      2828 Kb)
    NonPagedPool Usage:     1221 (      4884 Kb)
    NonPagedPool Max:      39095 (    156380 Kb)
    PagedPool 0 Usage:     15888 (     63552 Kb)
    PagedPool 1 Usage:       736 (      2944 Kb)
    PagedPool 2 Usage:       770 (      3080 Kb)
    PagedPool Usage:       17394 (     69576 Kb)
    PagedPool Maximum:     40960 (    163840 Kb)
    Shared Commit:           955 (      3820 Kb)
    Special Pool:              0 (         0 Kb)
    Shared Process:         1742 (      6968 Kb)
    PagedPool Commit:      17394 (     69576 Kb)
    Driver Commit:          2021 (      8084 Kb)
    Committed pages:       42622 (    170488 Kb)
    Commit limit:         312199 (   1248796 Kb)

    Total Private:         16898 (     67592 Kb)

one hour later:

*** Virtual Memory Usage ***
    Physical Memory:      194299 (    777196 Kb)
    Page File: \??\D:\pagefile.sys
      Current:    512000 Kb  Free Space:    475468 Kb
      Minimum:    512000 Kb  Maximum:       512000 Kb
    Available Pages:       84648 (    338592 Kb)
    ResAvail Pages:       142901 (    571604 Kb)
    Locked IO Pages:          58 (       232 Kb)
    Free System PTEs:     239904 (    959616 Kb)
    Free NP PTEs:          32636 (    130544 Kb)
    Free Special NP:           0 (         0 Kb)
    Modified Pages:          714 (      2856 Kb)
    Modified PF Pages:       714 (      2856 Kb)
    NonPagedPool Usage:     3064 (     12256 Kb)
    NonPagedPool Max:      39095 (    156380 Kb)
    PagedPool 0 Usage:     16672 (     66688 Kb)
    PagedPool 1 Usage:      2837 (     11348 Kb)
    PagedPool 2 Usage:      2819 (     11276 Kb)
    PagedPool Usage:       22328 (     89312 Kb)
    PagedPool Maximum:     40960 (    163840 Kb)
    Shared Commit:          1092 (      4368 Kb)
    Special Pool:              0 (         0 Kb)
    Shared Process:         1784 (      7136 Kb)
    PagedPool Commit:      22328 (     89312 Kb)
    Driver Commit:          1980 (      7920 Kb)
    Committed pages:       92917 (    371668 Kb)
    Commit limit:         312199 (   1248796 Kb)

    Total Private:         61981 (    247924 Kb)
Back to Top
Vittoriop77 View Drop Down
Newbie
Newbie


Joined: 21 January 2010
Status: Offline
Points: 1 		Post Options Post Options   Thanks (0) Thanks(0)   Quote Vittoriop77 Quote  Post ReplyReply Direct Link To This Post Posted: 21 January 2010 at 6:00am
I experienced the same problem on Windows 2003, do anybody got a solution ?
Thanks
 
 
Vittorio Pavesi
----------------------------
http://www.vittorio.tk
Back to Top
dirbase View Drop Down
Senior Member
Senior Member
Avatar

Joined: 26 March 2008
Status: Offline
Points: 496 		Post Options Post Options   Thanks (0) Thanks(0)   Quote dirbase Quote  Post ReplyReply Direct Link To This Post Posted: 21 January 2010 at 3:51pm
Hi Vittorio,

It may be relevant here to quote the follow-up I wrote on this topic in a comment to Mark's blog on "Pushing the limits of wWindows: paged and nonpaged pool"(look for the comment below the main text dated April 1, 2009):

.
Originally posted by dirbase on April 1, 2009 dirbase on April 1, 2009 wrote:

..about high pool usage following an explorer file search, since the MmSt paged pool tag is on top of the bytes list, I tend to think now that the basic reason is that the many searched files have been cached and the increased MmSt allocations correspond to their PTEs. This would explain why a repeat file search does not increase the pool usage.

Basically, it means that it is a normal behaviour!Stern Smile although the initial steep curve for pool resource usage may seem disturbing!
Back to Top
snoone View Drop Down
Senior Member
Senior Member


Joined: 04 September 2009
Location: Amherst, NH
Status: Offline
Points: 330 		Post Options Post Options   Thanks (0) Thanks(0)   Quote snoone Quote  Post ReplyReply Direct Link To This Post Posted: 21 January 2010 at 3:55pm
This looks "as designed" to me, what you're seeing is a side effect of the file system cache being used. I suspect that whatever application you're using to perform the search is opening each file and performing cached I/O to it. This causes the cache manager to keep the file contents in memory just in case someone comes along looking for them again.
 
The file system cache occupies its own space in memory outside of the pools, so the cache itself isn't causing your rise in pool usage. However, the cache manager relies on structures from the I/O manager (File), the file system (NtFs, Ntfr, etc), and the memory manager (MmSt). None of these structures are going to be freed until you actually boot the data from the cache. The only way that happens is if the cache manager or memory manager decide to evict it to make room for other data or if the volume gets dismounted.
 
As an experiment, you could run the same test except use a seconday storage device. Run the test, check the pool usage, and then dismount the volume:
 
fsutil volume dismount x:
 
(Requires that x: be formated with NTFS)
 
If all goes according to plan, you should see a drop in the pool usage.
 
Note that this isn't a bug, you're suddenly pounding the system with cached file I/O and the system is responding appropriately.
 
-scott
Back to Top
dirbase View Drop Down
Senior Member
Senior Member
Avatar

Joined: 26 March 2008
Status: Offline
Points: 496 		Post Options Post Options   Thanks (0) Thanks(0)   Quote dirbase Quote  Post ReplyReply Direct Link To This Post Posted: 21 January 2010 at 6:59pm
Thanks for these detailed informations, Scott. It seems that your conclusions are close to mine.Smile
Back to Top
 Post Reply Post Reply Page  12>
  Share Topic   

Forum Jump Forum Permissions View Drop Down