4

I have a problem dealing with bad_alloc. It is thrown during a std::vector.push_back when it tries to reloc and allocate 2Mb

The heaps status is

  Heap     Flags   Reserv  Commit  Virt   Free  List   UCR  Virt  Lock  Fast 
                    (k)     (k)    (k)     (k) length      blocks cont. heap 
-----------------------------------------------------------------------------
04e00000 00000002   67704  41088  67704  40735   135   135    0     25   LFH
    External fragmentation  99 % (135 free blocks)
    Virtual address fragmentation  39 % (135 uncommited ranges)
...
0c8f0000 00001002   82216  81384  82216   8247 172196    13    5      3   LFH
...

 0: Heap 04e00000
   Flags          00000002 - HEAP_GROWABLE 
   Reserved memory in segments              67704 (k)
   Commited memory in segments              41088 (k)
   Virtual bytes (correction for large UCR) 58644 (k)
   Free space                               40735 (k) (135 blocks)
   External fragmentation          99% (135 free blocks)
   Virtual address fragmentation   29% (135 uncommited ranges)
   Virtual blocks  0 - total 0 KBytes
   Lock contention 37
   Segments        1

   Low fragmentation heap   04e044f0
       Lock contention        0
       Metadata usage         0 bytes
       Statistics:
           Segments created          0
           Segments deleted          0
           Segments reused           0
       Block cache:

       Buckets info:
  Size   Blocks  Seg  Empty  Aff    Distribution
------------------------------------------------
------------------------------------------------

                    Default heap   Front heap       Unused bytes
   Range (bytes)     Busy  Free    Busy   Free     Total  Average 
------------------------------------------------------------------ 
     0 -   1024      217      0      0      0        487      2
  1024 -   2048        1      0      0      0          1      1
  3072 -   4096        0      1      0      0          0      0
 15360 -  16384        1      0      0      0          1      1
203776 - 204800        0      2      0      0          0      0
220160 - 221184        0      2      0      0          0      0
224256 - 225280        0      1      0      0          0      0
236544 - 237568        0      2      0      0          0      0
240640 - 241664        0      1      0      0          0      0
257024 - 258048        0      1      0      0          0      0
265216 - 266240        0      4      0      0          0      0
273408 - 274432        0      1      0      0          0      0
277504 - 278528        0      2      0      0          0      0
281600 - 282624        0      2      0      0          0      0
285696 - 286720        0      6      0      0          0      0
289792 - 290816        0      4      0      0          0      0
293888 - 294912        0     12      0      0          0      0
297984 - 299008        0     10      0      0          0      0
302080 - 303104        0     13      0      0          0      0
306176 - 307200        0     14      0      0          0      0
310272 - 311296        0     19      0      0          0      0
314368 - 315392        0     15      0      0          0      0
318464 - 319488        0      3      0      0          0      0
326656 - 327680        0      1      0      0          0      0
330752 - 331776        0      3      0      0          0      0
333824 - 334848        1      0      0      0          8      8
351232 - 352256        0      1      0      0          0      0
363520 - 364544        0      2      0      0          0      0
367616 - 368640        0      1      0      0          0      0
396288 - 397312        0      3      0      0          0      0
416768 - 417792        0      1      0      0          0      0
420864 - 421888        0      2      0      0          0      0
424960 - 425984        0      1      0      0          0      0
433152 - 434176        0      1      0      0          0      0
437248 - 438272        0      1      0      0          0      0
466944 - 467968        0      1      0      0          0      0
470016 - 471040        0      1      0      0          0      0
482304 - 483328        0      1      0      0          0      0
------------------------------------------------------------------ 
  Total              220    135      0      0        497      2


 0: Heap 0c8f0000
   Flags          00001002 - HEAP_GROWABLE 
   Reserved memory in segments              82216 (k)
   Commited memory in segments              81384 (k)
   Virtual bytes (correction for large UCR) 81620 (k)
   Free space                               8247 (k) (172196 blocks)
   External fragmentation          10% (172196 free blocks)
   Virtual address fragmentation   0% (13 uncommited ranges)
   Virtual blocks  5 - total 0 KBytes
   Lock contention 3
   Segments        1

   Low fragmentation heap   43230048
       Lock contention        0
       Metadata usage      4096 bytes
       Statistics:
           Segments created        120
           Segments deleted          2
           Segments reused           0
       Block cache:
         5:         4096 bytes (    54,     0)
         6:         8192 bytes (    26,     0)
         7:        16384 bytes (    13,     0)
         8:        32768 bytes (     8,     0)
         9:        65536 bytes (     9,     0)
        10:       131072 bytes (     5,     0)
        11:       262144 bytes (     4,     1)

       Buckets info:
  Size   Blocks  Seg  Empty  Aff    Distribution
------------------------------------------------
    64   25483    53      0  0 (53-25483)
    72    2991    30      0  0 (30-2991)
    80     750    15      0  0 (15-750)
    88      46     1      1  0 (1-46)
    96      42     1      1  0 (1-42)
   104      39     1      1  0 (1-39)
   112      36     1      1  0 (1-36)
   120      33     1      1  0 (1-33)
   128      63     1      1  0 (1-63)
   136      60     1      1  0 (1-60)
   144      56     1      1  0 (1-56)
   152      53     1      1  0 (1-53)
   160      51     1      1  0 (1-51)
   168      48     1      1  0 (1-48)
   200      40     1      1  0 (1-40)
   208      39     1      1  0 (1-39)
   224      36     1      1  0 (1-36)
   280      29     1      1  0 (1-29)
   312      26     1      1  0 (1-26)
  7432      17     1      1  0 (1-17)
  9736      26     1      1  0 (1-26)
 10760      24     1      1  0 (1-24)
 11784      22     1      1  0 (1-22)
------------------------------------------------

                    Default heap   Front heap       Unused bytes
   Range (bytes)     Busy  Free    Busy   Free     Total  Average 
------------------------------------------------------------------ 
     0 -   1024   619437 172194  28854   1067   11637848     17
  1024 -   2048       10      1      0      0        113     11
  3072 -   4096        2      0      0      0         32     16
  4096 -   5120        1      0      0      0         16     16
  5120 -   6144        1      0      0      0         20     20
  7168 -   8192        1      0      0     17         20     20
  9216 -  10240        0      0      0     26          0      0
 10240 -  11264        0      0      0     24          0      0
 11264 -  12288        0      0      0     22          0      0
 12288 -  13312        2      0      0      0         32     16
 15360 -  16384        1      0      0      0          1      1
 16384 -  17408        2      0      0      0         32     16
 20480 -  21504        1      0      0      0          8      8
 21504 -  22528        1      0      0      0         20     20
 40960 -  41984        1      0      0      0          8      8
 43008 -  44032        1      0      0      0         16     16
 48128 -  49152        1      0      0      0         16     16
 54272 -  55296        1      0      0      0          8      8
 58368 -  59392        1      0      0      0         17     17
 72704 -  73728        1      0      0      0         20     20
 83968 -  84992        0      1      0      0          0      0
 96256 -  97280        1      0      0      0         16     16
108544 - 109568        1      0      0      0         16     16
116736 - 117760        1      0      0      0         21     21
139264 - 140288        1      0      0      0         16     16
145408 - 146432        1      0      0      0         16     16
159744 - 160768        1      0      0      0         22     22
163840 - 164864        1      0      0      0         20     20
193536 - 194560        1      0      0      0         20     20
262144 - 263168        3      0      0      0         40     13
327680 - 328704        1      0      0      0         16     16
333824 - 334848        1      0      0      0          8      8
436224 - 437248        1      0      0      0         16     16
------------------------------------------------------------------ 
  Total           619479 172196  28854   1156   11638454     17

My question is what External fragmentation 99% means and why is "heap fragmentation" and not "Virtual Address space fragmentation" ?

What is the difference between "External fragmentation" (99%) and "Virtual address fragmentation" (29%) ?

Creating a new heap, may help ?

PS: I have in the same process a LOT of managed code. The Private Working Set is 1,8Gb.

3
  • Can you show the code where the push_back on the vector happens? And where you try to resize it? Nov 29, 2011 at 19:53
  • In case you have managed code your native and managed heap are both competing for the same virtual memory space which in case on 32 bit app is by default 2 GB. In case you memory gets fragmented also, you should have a look and see why that happens. For .NET managed heap there are a couple of straightforward techniques. See msdn.microsoft.com/en-us/magazine/cc163528.aspx#S6
    – Ghita
    Nov 29, 2011 at 20:02
  • @tony It's a simple std::vector::push_back. The problem is that it goes over the current reserved size and tries to reserve a bigger chunk and failes with std::bad_alloc. My queston is what External fragmentation means (I cannot find any doc about this). From my point of view the only fragmentation that happens is the fragmentation of virtual address space. I an wrong ? Nov 30, 2011 at 9:15

1 Answer 1

2

Basically you have a large number of very small object allocations, which cause the fragmentation. In addition you are mixing native and managed code in the same process and you are making big memory allocations. These approaches require memory management. It would be best to create a dedicated heap for large memory allocations on process startup to avoid this kind of problem. The heap should only be used for large memory allocations of native code.

In addition it would be a good idea to analyze why the managed code needs so much memory.

2
  • I know that managed code is using a managed heap. Managed heap may fragment the memory? is managed heap just a heap from OS point of view ? Dec 1, 2011 at 9:40
  • The managed heap is seen from the OS a a memory allocated by VirtualAlloc, and not as a heap. The .NET runtime manages the contents of this bloc of memory without the OS knowledge. Dec 1, 2011 at 16:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.