1

I am issuing a SQL query using CodeIgniter which can return upwards of 80,000 rows as results. Each row has three columns all of which are integers, and I am getting a PHP error: Fatal error: Allowed memory size of 134217728 bytes exhausted

It seems that I am trying to use more than 128MB to retrieve the results from the MySQL server. I am using $query->result_array() to retrieve the results. There is obviously a serious overhead in terms of space in the results I am getting. Say I retrieve 100,000 rows, with 3 integers. So 100,000*((3*4 + 10)= 2.1MB. (The 10 is the number of bytes used for column id etc).

Am I doing something wrong?

----------------------SOLVED-----------------------

Solved by modifying CodeIgniter code: link.

Execution is now faster and the script is only taking ~3MB of memory instead of upwards of 128MB.

5
  • 4
    retrieve chunk, process; retrieve next chunk ...
    – user557846
    Jan 10, 2012 at 23:08
  • take the suggestion of Dagon do not take all data at once. Jan 10, 2012 at 23:11
  • @Dagon can you show us some example on how can "retrieve chunk, process; retrieve next chunk" be done on a large query?
    – fujisan
    Feb 21, 2014 at 9:19
  • 2
    Your solution link is dead. Please explain how you solved this issue Jun 16, 2015 at 20:46
  • Possible duplicate of CodeIgniter Active Record: Load One Row at a Time
    – Aziz Saleh
    Apr 3, 2016 at 1:19

5 Answers 5

2

I'm not sure exactly how CI returns results, but you definitely don't want that entire data set in a single array. Why not loop through each row as needed?

<?php

$result = $this->db->query('SELECT ...');
while($row = $result->next_row())
{
  // do something with that single row
}
2
  • 2
    In my experience, this causes CodeIgniter 2.x to load the complete result set into memory anyway.
    – Nate
    Jul 23, 2015 at 22:25
  • I think @Nate is right. $result will contain all the results from the query Mar 31, 2021 at 9:15
1

Except you really need to retrieve all records - e.g. for export or batch like purposes - I strongly propose to always query chunks.

MySQL's SELECT statement has this option:

[LIMIT {[offset,] row_count | row_count OFFSET offset}]

For example

Limit 10, 200

provides 200 records beginning with record 10.

1
  • Good point. But also when doing exports, you'll want to do this in batches/chunks. For CI queries there's limit(), e.g. $this->db->limit(0,10) Jun 3, 2022 at 10:10
1

If you are displaying the results to the page you will most likely need to use pagination. If you are performing actions on the data then you will need to do it in pieces using LIMITS and OFFSETS. Otherwise you will need to increase your memory limit, which I don't suggest being larger than 128MB.

0

Even though the question is answered, 3MB is still a lot. Do you really need to store it all in memory? A normal PHP footprint runs around 1MB or less given that it was programmed with memory in mind. The fact that you might even reach past 8MB is very disturbing, and in a real production environment with thousands of page hits, your server will crash and burn. I highly recommend you revise your code.

1
  • 1
    This is what opcode and other caches are for. 3MB is nothing for an app of reasonable complexity. And RAM is relatively cheap these days. If you're getting thousands of page hits (in a time frame where 3MB is problematic), hopefully you're monetizing your user base to pay for it...
    – landons
    Jan 11, 2012 at 1:26
0

I ran into the same issue with CI. Unfortunately, modifying the DB backend was not an option for me. So my solution was to just use mysqli_ functions in the model and loop through each record on its own:

$result = mysqli_query($sql);
while ($row = mysqli_fetch_assoc($result)){
    // process batch
}

And work with the records this way, I didn't have to do sql pagination nor store all the info on 1 array. This also works with mysql_ functions as well (depending on the DB driver you are using). The drawbacks here:

  • You are not using CI DB engine, so should you decide to use a different driver, you will have to manually update these queries.
  • Won't work with multiple DBs unless you somehow store the DB identifier.

There might be other drawbacks, but for me it works and doesn't consume that much data.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.