2

I'm working an a script with product filters. Everything works fine but it's too slow. The reason is too much query's. Query5 selects about 200 entrys. Query12 "search" in 130.000 entry's.

Is it possible to make a mysql join or something faster of this with less query's?

    <?php
                $query5 = mysql_query("SELECT id,fabrikant,naam FROM producten_new WHERE icecat_cat='".secure_in($obj->icecat)."'".$selecteren.""); 
                while($row5 = mysql_fetch_assoc($query5)){ 

                    foreach($_GET as $key=>$value){ 
                            if($key != "p" AND $key != "id" AND $key != "merk" AND $key != "submit" AND $value != "0"){ 

                                $query12 = mysql_query("SELECT COUNT(id) as aantal FROM producten_specs WHERE name='".secure_in(str_replace("_", " ", $key))."' AND value='".secure_in($value)."' AND product_id='".secure_in($row5['id'])."'"); 
                                $data12 = mysql_fetch_assoc($query12); 

                                if($data12['aantal']){ 
                                    $product_present = TRUE; 
                                } else { 
                                    $product_present = FALSE; 
                                    break;
                                } 

                            } 
                    }
                    if($product_present){
                        echo $row5['id']."<br />";
                        $product_present = FALSE;
                    }
               }
    ?>

Thanks.

EDIT: Ok, fresh start. I hope its better (faster) this way:

        //Select products from category and brand
        $query5 = mysql_query("SELECT id FROM producten_new WHERE icecat_cat='".secure_in($obj->icecat)."'".$selecteren."");
        while($row5 = mysql_fetch_assoc($query5)){
            $query5_ids .= $row5['id'].", ";
        }
        //Remove last ','
        $query5_ids = substr($query5_ids, 0, -2);

        //Get all $_GET's
        foreach($_GET as $key=>$value){
            if($key != "p" AND $key != "id" AND $key != "merk" AND $key != "submit" AND $value != "0"){
                $specs_gets .= $key.",".$value.";";
                $count_gets++;
                $selection .= "name='".secure_in(str_replace("_", " ", $key))."' AND value='".secure_in($value)."' OR ";
            }
        }
        //Remove last ';'
        $specs_gets = substr($specs_gets, 0, -1);


        //$selection = " AND ".substr($selection, 0, -4);
        $selection = substr($selection, 0, -4);


        //THE query..
        //$query12 = mysql_query("SELECT product_id FROM producten_specs WHERE product_id IN ('".secure_in($query5_ids)."') ".$selection." ORDER BY product_id ASC");
        $query12 = mysql_query("SELECT product_id FROM producten_specs WHERE ".$selection." ORDER BY product_id ASC");
        while($row12 = mysql_fetch_assoc($query12)){

            if($product_hold == $row12['product_id']){
                $product_hold_count++;
            } else {
                $product_hold_count = FALSE;
                $product_hold = $row12['product_id'];
                $product_hold_count = 1;
            }

            if($product_hold_count == $count_gets){

                $query12_ids_ok .= $row12['product_id'].", ";
                $count_products++;
            }

        }
        //Remove last ','
        $query12_ids_ok = substr($query12_ids_ok, 0, -2);

EDIT2 Link to the website: snip

EDIT3 Some more information:

Example of the URL:

index.php
?p=categorie
&id=5
&brand=Asus
&Display_inch=0
&Resolution=0
&Buldin_camera=0
&Chipset=0
&Processor-speed=2400+MHz
&Processorfamilie=0
&Hard+disk-interface=0
&Total+capacitie=500+GB
*snip (because its very long)*
&submit=Submit+Filter

The database:

product_new (contains 3500 entrys) - Contains general product information

id, supplier, price, icecat_cat, name, etc.

1, Asus, 500, Notebook, K72F

product_specs (contains 130.000 entrys) - Contians product specifications

id, product_id, cat, name, value

1, 1, Processor, Processor-speed, 2400 Mhz

2, 1, Harddisk, Total-capacitie, 500 GB

EDIT4 I'm getting closer, see the code in my first edit, only product_id IN ('".secure_in($query5_ids)."') must be added into query12. Then I get the results I want and a lot faster! Anyone?

8
  • 1
    200 rows in producten_new 10 GET variables = 2K queries! you need to restructure !!!
    – RobertPitt
    Mar 17, 2011 at 12:55
  • Indexes? How often does the for loop get executed? Mar 17, 2011 at 12:55
  • It is a little bit complex, perheps separate things. It is slowing down 'cause you have and foreach inside a while statement. A few rows will work fine, but if the data inscrease, the time also increase. Mar 17, 2011 at 13:04
  • A bit hard to understand what you're doing cos your tables seem to be in Dutch (guessing!) But the first thing you could do is getting a bunch of items at once by using IN() and ORDER BY FIELD when you send in your second query (assuming you don't have an identifying field in your second query)
    – JohnP
    Mar 17, 2011 at 13:06
  • 1
    @JohnP: It is Dutch, and yes, he should consider using English as lingua franca (variable, table, class & function names, comments, ...).
    – wimvds
    Mar 17, 2011 at 15:13

4 Answers 4

1

Because you are dutch (can see it in your programming) I will also post my answer in dutch.


Ik heb 1 puntje al als snelheid verbetering: omdat je $product_niet_aanwezig op true zet doet hij de rest van de foreach wel sneller maar hij doet het niet te min. Je kan ook een break statement erin zetten. Dan "breakt" die uit de foreach loop.


I have one point of improvement: Because you set $product_niet_aanwezig to true it will continue doing the rest of the foreach loop alot quicker but it will still do it. You could also use a break statement to "break" out of the foreach loop.

1
  • I'm using the break now, thanks! But not really a preformance tweak.
    – Roy
    Mar 17, 2011 at 17:04
1
  • You should first consider verifying indexes on all the keys used in the 'where' conditions. (name, value, product_id in table producten_specs for instance)

  • query12 can be called only once : your loop should build one big conditions, then only 1 query is sent. AS you're working with "count", you should be able to build something like

COUNT(id) FROM producten_specs WHERE [the condition built in your loop] GROUP BY product_id (this may not be totally accurate though)

  • I don't know your db shema, but trying to build a view with a query performing as much as possible. A view with a join using your product_id key, and listing only the columns you need, would be easier to interrogate

My 2 cents :)

3
  • I've edited my start post and added database information. I hope its clearer now.
    – Roy
    Mar 18, 2011 at 12:06
  • I'm not totally into your problematic. Though I can recommend to work with small steps : - continue to run your query12 with a result from the results of query5 - try to show the generated SQL code, and run que the select query in a DB editor (phpmyadmin, mysql workbench, etc.) As soon as you get a working query, you can try to "upgrade" the way you generate it. With your new code, does your query5 work ?
    – Rodolphe
    Mar 18, 2011 at 16:12
  • Query5 works fine. I've no idea how to get the results. For example how to select 1 and 2 (from the example in start post at product_specs, see edit3) by product_id from query5 and name and value from a $_GET?
    – Roy
    Mar 18, 2011 at 16:25
1

Your code

I can't see what's wrong with you code since i'm Italian and I do not understand the real logic behind that script. Despite this it seems to me that you are running a loop that will execute either thousands of queries, or just one (because of the if condition), in both way I'd suggest the using of PDO statements that somehow cache the queries and make them run faster the second time.

PDO Fever

Since the second query is executed inside a loop, I'd suggest you to use PDO which will store the statement and every time the statement is executed it is faster.

I'm quoting the Manual here:

By using a prepared statement the application avoids repeating the analyze/compile/optimize cycle. This means that prepared statements use fewer resources and thus run faster

I want to make you notice that PDO prevents also from SQL Injection:

[...] the developer can be sure that no SQL injection will occur [...]

Indexes

You have to set indexes in the database side so that to interrogate specific tables will take less time. Basically SQL indexes are similar to books indexes, they provide a fast way to locate informations.

3
  • Thanks for the reply but I've no expirence with PDO. I'll follow some tutorials about it. I hope it can do what I want.
    – Roy
    Mar 18, 2011 at 10:58
  • @Roy, PDO is the future. I did tried mysql, mysqli and PDO and it's a revolution. Trust me.
    – Shoe
    Mar 18, 2011 at 13:11
  • I've read some tutorials and PDO is not so diffucult but with the same query (query12) the script remains slow...? The script have to change totaly. I've no idea how.. I've tried (see edit2 in my start post).
    – Roy
    Mar 18, 2011 at 13:57
0

Got it!! Thanks for cooperate!

    //Producten selecteren uit categorie en merk 
    $query5 = mysql_query("SELECT id FROM producten_new WHERE icecat_cat='".secure_in($obj->icecat)."'".$selecteren.""); 
    while($row5 = mysql_fetch_assoc($query5)){ 
        $query5_ids .= $row5['id'].", "; 
    } 
    //Laatste , weg halen 
    $query5_ids = substr($query5_ids, 0, -2); 

    //Gets op een rijte zetten 
    foreach($_GET as $key=>$value){ 
        if($key != "p" AND $key != "id" AND $key != "merk" AND $key != "submit" AND $value != "0"){ 
            $specs_gets .= $key.",".$value.";"; 
            $aantal_gets++; 
            $uiteindelijk_selecteren .= "name='".secure_in(str_replace("_", " ", $key))."' AND value='".secure_in($value)."' OR "; 
        } 
    } 
    //Laatste ; weg halen 
    $specs_gets = substr($specs_gets, 0, -1); 


    //$uiteindelijk_selecteren = " AND ".substr($uiteindelijk_selecteren, 0, -4); 
    $uiteindelijk_selecteren = substr($uiteindelijk_selecteren, 0, -4); 


    //Overeenkomend met de get's selecteren 
    $query12 = mysql_query("SELECT product_id FROM producten_specs WHERE product_id IN (".secure_in($query5_ids).") AND ".$uiteindelijk_selecteren." ORDER BY product_id ASC");
    while($row12 = mysql_fetch_assoc($query12)){ 

        if($product_onthouden == $row12['product_id']){ 
            $product_onthouden_aantal++; 
        } else { 
            $product_onthouden_aantal = FALSE; 
            $product_onthouden = $row12['product_id']; 
            $product_onthouden_aantal = 1; 
        } 

        if($product_onthouden_aantal == $aantal_gets){ 

            $query12_ids_ok .= $row12['product_id'].", "; 
            $aantal_producten++; 
        } 

    } 
    //Laatste , weg halen 
    $query12_ids_ok = substr($query12_ids_ok, 0, -2);
1
  • Glad it worked ! You should then consider to "accept" the solution that brought you the most inputs. (low acceptance user tend to receive less answers as time goes by)
    – Rodolphe
    Mar 21, 2011 at 8:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.