0

I have a database with two tables. "speechesLCMcoded" includes 400K lines of coded text, and "concreteness" includes 80k words with scores.

I wrote a script that looks into the table with parsed text (speechesLCMcoded), I check for each word in another table after removing the tags (concreteness table) and I add up the resulting scores.

I am a beginner in PHP and my code is not optimized at all. I do not mind if my script runs for the entire day, but I cannot have it run for a week. How would you advise me to optimize my script?

My scripts performs everything I need. It is just way too slow.

<?php
//Include functions
        include "functions.php";
        ini_set('max_execution_time', 900000);
        echo 'Time Limit = ' . ini_get('max_execution_time');

//Conecting the database
        if (!$conn) {
         die('Not connected : ' . mysql_error());}

// make LCM the current db
        mysql_select_db('senate');
        $data = mysql_query("SELECT `key`, `tagged` FROM speechesLCMcoded") or die(mysql_error());

// puts the "data" info into the $info array 
        while($info = mysql_fetch_array( $data) ){
        $key=$info['key'];
        $tagged=$info['tagged'];
        unset($weight);
        unset($count);
        $weight=0;
        $count=0;

// Print out the contents of the entry 
        Print "<b>Key:</b> ".$info['key'] .  " <br>";

// Explodes the sentence
        $speech = explode(" ", $tagged);

// Loop every word  
        foreach($speech as $word) {

//Print each word       
        //Print "<b>Key:</b> ".$word .  " <br>";

//Check if string contains our tag

if(!preg_match('/({V}|{J}|{N}|{RB})/', $word, $matches)) {} else{

//Removes our tags
        $word = str_replace("{V}", "", $word);
        $word = str_replace("{RB}", "", $word);
        $word = str_replace("{J}", "", $word);
        $word = str_replace("{N}", "", $word);
        $word = str_replace("{/V}", "", $word);
        $word = str_replace("{/RB}", "", $word);
        $word = str_replace("{/J}", "", $word);
        $word = str_replace("{/N}", "", $word);

        //print $word .  " <br>";

        //Check for the score
        $checksql = "SELECT word, score FROM concreteness WHERE word = '$word'"; 
        $query = mysql_query("$checksql"); 
        $check_count = mysql_num_rows($query);          
            if($check_count > 0 ){ 
            $data2 = mysql_fetch_assoc($query);
            $weight=$weight+$data2['score'];
            $count=$count +1;
        //  echo $weight;
        //  print "<br>";
        //  echo $count;
        //  print "<br>";
            } else {  
        //  echo"The word was NOT found.<br>";            
 }   }      
        }

        $sql = "UPDATE speechesLCMcoded SET weight='$weight', count='$count' WHERE `key`='$key';" ;
        $retval = mysql_query( $sql, $conn );
        if(! $retval )
        {die('Could not update data: ' . mysql_error());}
        echo "Updated data successfully\n";   

}?> 
6
  • When you say "speechesLCMcoded" includes 400K lines of coded text, do you mean the table contains 400k rows of varying sized sentences? If so, how many words are in the sentences? May 8, 2014 at 18:34
  • Yes, the table contains 400K rows. Each row actually contains pretty long texts. It can go from 50 to 2000 words. May 8, 2014 at 18:36
  • You've got a preg_match and 2 SQL queries inside an inner loop, one of which is an update. These are relatively expensive operations. You need to rethink your querying strategy (and possibly your database design). And if you're iterating over 400,000 rows then anything that's expensive in terms of time is only going to be exacerbated by being inside an inner loop.
    – GordonM
    May 8, 2014 at 18:36
  • Get your concreteness table as well in some PHP var and do all your calculations in PHP. In the end/or after some interval do an update on all rows. Your network latency would be reduced to a great extent May 8, 2014 at 18:38
  • @user1029296 Will a whole word actually be a tag, like here: I {v} Turtles or more like I li{v}ke Turtles? May 8, 2014 at 18:41

1 Answer 1

1

For every row from speechesLCMcoded (400K rows), you exec str_replace and sql query.

You can remove tags into first sql query use replace function (http://dev.mysql.com/doc/refman/5.0/en/replace.html). You do not need exec str_replace x 8 for every row.

It's first step.

To second step you can use one query only with use join to get all data from two tables.

9
  • Thanks for the answer. How would you join the tables? They do not share parameters as one has rows on text and the other one is a lexicon. May 8, 2014 at 19:01
  • SELECT c.word, c.score, s.key, s.tagged FROM concreteness c INNER JOIN speechesLCMcoded s ON LOCATE(c.word, REPLACE(s.tagged,.....)) > 0 May 8, 2014 at 19:10
  • I am not sure I understand. Sorry, I am not very advanced. Can you explain to me what it does and where I should put it? May 8, 2014 at 19:11
  • How I understand your script, for every row from speechesLCMcoded, you get tagged. tagged field you explode to words, and every word you search to second table. In my query I join two tables and use locate function. Locate function search word to tagged. You receive all rows, if word exist to tagged. May 8, 2014 at 19:17
  • I have question. Why you store all tags in one field? You can store all tags of speechesLCMcoded to separate table and use condition table (key, tag_id). May 8, 2014 at 19:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.