3

I wrote a simple application to test memory consumption. In this test application, I created four processes to continually consume memory, those processes won't release the memory unless the process exits.

I expected this test application to consume the most memory of RAM and cause the other application to slow down or crash. But the result is not the same as expected. Below is the code:

 #include <stdio.h>
 #include <unistd.h>
 #include <list>
 #include <vector>

 using namespace std;
 unsigned short calcrc(unsigned char *ptr, int count)
 {
     unsigned short crc;
     unsigned char i;

     //high cpu-consumption code 
     //implements the CRC algorithm
     //CRC is Cyclic Redundancy Code
 }


 void* ForkChild(void* param){
    vector<unsigned char*>  MemoryVector;
    pid_t PID = fork();
    if (PID > 0){
        const int TEN_MEGA = 10 * 10 * 1024 * 1024;
        unsigned char* buffer = NULL;
        while(1){
            buffer  = NULL;
            buffer = new unsigned char [TEN_MEGA];
            if (buffer){
                 try{
                    calcrc(buffer, TEN_MEGA);
                    MemoryVector.push_back(buffer);
                 } catch(...){
                    printf("An error was throwed, but caught by our app!\n");
                    delete [] buffer;
                    buffer = NULL;
                 }
            }
            else{
                 printf("no memory to allocate!\n");
                 try{
                     if (MemoryVector.size()){
                        buffer = MemoryVector[0];
                        calcrc(buffer, TEN_MEGA);
                        buffer = NULL;
                     } else {
                        printf("no memory ever allocated for this Process!\n");
                        continue;
                     }
                 } catch(...){
                    printf("An error was throwed -- branch 2," 
                           "but caught by our app!\n");
                    buffer = NULL;
                 }
             }
         }  //while(1)
    } else if (PID == 0){
    } else {
      perror("fork error");
    }   

    return NULL;
}


int main(){
int children = 4;
    while(--children >= 0){
    ForkChild(NULL);
    };

    while(1) sleep(1);
    printf("exiting main process\n");
    return 0;
 }
  1. TOP command

    PID  USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND           
    2775 steve     20   0 1503m  508  312 R  99.5  0.0   1:00.46 test              
    2777 steve     20   0 1503m  508  312 R  96.9  0.0   1:00.54 test              
    2774 steve     20   0 1503m  904  708 R  96.6  0.0   0:59.92 test              
    2776 steve     20   0 1503m  508  312 R  96.2  0.0   1:00.57 test
    

    Though CPU is high, but memory percent remains 0.0. How can it be possible??

  2. Free command

                      free  shared    buffers     cached          
    Mem:           3083796       0      55996     428296
    

    Free memory is more than 3G out of 4G RAM.

Does there anybody know why this test app just doesn't work as expected?

6
  • 7
    Linux won't actually allocate memory pages unless you write to them. I don't see you writing to your buffer anywhere.
    – user703016
    Sep 20, 2013 at 5:04
  • @한국매미 new[] default-initializes all of the items. Sep 20, 2013 at 5:10
  • 2
    @CoryNelson Yes, and default-initialization of unsigned char is a no-op: no write is involved.
    – user703016
    Sep 20, 2013 at 5:23
  • 1
    @CoryNelson stackoverflow.com/questions/7546620/…
    – keltar
    Sep 20, 2013 at 5:28
  • @한국매미 if you set your comments as an answer to this post, I will accept it.
    – Wallace
    Sep 20, 2013 at 13:04

5 Answers 5

6

Linux uses optimistic memory allocation: it will not physically allocate a page of memory until that page is actually written to. For that reason, you can allocate much more memory than what is available, without increasing memory consumption by the system.

If you want to force the system to allocate (commit) a physical page , then you have to write to it.

The following line does not issue any write, as it is default-initialization of unsigned char, which is a no-op:

buffer = new unsigned char [TEN_MEGA];

If you want to force a commit, use zero-initialization:

buffer = new unsigned char [TEN_MEGA]();
1
  • 5
    -1 for using new?? The question didn't ask for best practices, it asked about memory usage. The answer addresses this concisely -- to introduce anything else would only convolute it. Sep 20, 2013 at 14:55
3

To make the comments into an answer:

  • Linux will not allocate memory pages for a process until it writes to them (copy-on-write).
  • Additionally, you are not writing to your buffer anywhere, as the default constructor for unsigned char does not perform any initializations, and new[] default-initializes all items.
1
  • very brief and helpful! If your comments come earlier I might have taken a different decision :_)
    – Wallace
    Sep 20, 2013 at 14:51
1

fork() returns the PID in the parent, and 0 in the child. Your ForkChild as written will execute all the work in the parent, not the child.

And the standard new operator will never return null; it will throw if it fails to allocate memory (but due to overcommit it won't actually do that either in Linux). This means your test of buffer after the allocation is meaningless: it will always either take the first branch or never reach the test. If you want a null return, you need to write new (std::nothrow) .... Include <new> for that to work.

1
  • yes, you are right, I did make a mistake when judging which one is the child process. Moreover, I also much benefited from your comments on the new operator. Thanks so much for pointing out these!
    – Wallace
    Sep 20, 2013 at 14:36
0

But your program is infact doing what you expected it to do. As an answer has pointed out (@ Michael Foukarakis's answer), memory not used is not allocated. In your output of the top program, I noticed that the column virt had a large amount of memory on it for each process running your program. A little googling later, I saw what this was:

VIRT -- Virtual Memory Size (KiB). The total amount of virtual memory used by the task. It includes all code, data and shared libraries plus pages that have been swapped out and pages that have been mapped but not used.

So as you can see, your program does in fact generate memory for itself, but in the form of pages and stored as virtual memory. And I think that is a smart thing to do

A snippet from this wiki page

A page, memory page, or virtual page -- a fixed-length contiguous block of virtual memory, and it is the smallest unit of data for the following:

  • memory allocation performed by the operating system for a program; and
  • transfer between main memory and any other auxiliary store, such as a hard disk drive.

...Thus a program can address more (virtual) RAM than physically exists in the computer. Virtual memory is a scheme that gives users the illusion of working with a large block of contiguous memory space (perhaps even larger than real memory), when in actuality most of their work is on auxiliary storage (disk). Fixed-size blocks (pages) or variable-size blocks of the job are read into main memory as needed.

Sources:

2
  • oh, I am confused agian. If the program in fact reserved the memory, why I got "free 3083796" from the result of running "shell free" command(RAM of my computer is 4G bytes)?
    – Wallace
    Sep 20, 2013 at 14:28
  • If you are still confused, I have updated my answer plus I found this question on Unix & Linux exchange website which I think the people there will better explain this to you.
    – smac89
    Sep 20, 2013 at 15:11
-1

If you want to gobble up a lot of memory:

int mb = 0;
char* buffer;
while (1) {
    buffer = malloc(1024*1024);
    memset(buffer, 0, 1024*1024);
    mb++;
}

I used something like this to make sure the file buffer cache was empty when taking some file I/O timing measurements.

As other answers have already mentioned, your code doesn't ever write to the buffer after allocating it. Here memset is used to write to the buffer.

4
  • Why allocate one MB at a time? Why not allocate a few hundred MB or a GB straight up?
    – Cramer
    Sep 20, 2013 at 5:13
  • Sure, just multiply by another 1024.
    – Ryan
    Sep 20, 2013 at 5:14
  • Okay, I've edited it a bit. Better? If you still think it is completely valueless, tell me so I can just delete it.
    – Ryan
    Sep 20, 2013 at 16:50
  • @Puppy long time ago, but are there any other changes you'd like to see?
    – Ryan
    Jun 7, 2018 at 1:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.