0

I have just started to learn how to use regular expressions to extract data from websites. The first goal of mine is to extract the title of a website. Here is what my code is like:

<?php 
    $data = file_get_contents('http://bctia.org');
    $regex = '/<title>(.+?)<\/title>/';
    preg_match($regex,$data,$match);
    var_dump($match); 
?>

The result of var_dump is empty:

array(0) { }

At first I thought, "maybe bctia.org does not have a title"? However, this is not the case, as I have checked the source of bctia.org, and it does have content between <title> and </title>.

Then I thought, maybe my code does not work? However, this is not the case either, as I have substituted bctia.org with other websites, say, bing.com, or apple.com, and they both returned correct results. For example, with apple.com I get the correct result

array(2) { [0]=> string(20) "" [1]=> string(5) "Apple" }

So I have to come to the conclusion that bctia.org is a very special website that prevents me from extracting its title...

I am wondering if that is actually the case? Or maybe my code has some problems that I have not identified?

Thank you in advance!

8
  • In this specific case, lack of /s modifier.
    – mario
    Jun 12, 2013 at 0:16
  • 3
    I'll leave this here: stackoverflow.com/questions/1732348/…
    – styfle
    Jun 12, 2013 at 0:18
  • Hi @mario, thanks for the suggestion. Unfortunately adding /s to the expression still does not work for bctia.org..
    – Bing
    Jun 12, 2013 at 0:34
  • @Bing I just tested your code with someone's example below, and it could be the fact that <title> might be written in uppercase <TITLE>, as was the case for me. My site has title in uppercase, which is why it didn't work in the first place. I asked myself the question and came up with that. Another possible reason could be that they're running some fancy JS. Look into other possible factors. Another reason is redirection, which is another possible factor, as this was something happening on my site. Jun 12, 2013 at 0:48
  • @Bing To add, I tried your code the way it is, and replaced <title></title> with uppercase letters, and it worked fine for me. Try and incorporate a function in there (strtoupper) that will take your lowercase title and render it in uppercase. I.e. $regex = strtoupper($regex); Jun 12, 2013 at 0:50

3 Answers 3

3

This specific website's server-side code assumes that the client sends a User-Agent header, and apparently, your PHP installation is not configured to send one. So a 500 Internal Server Error is returned, causing file_get_contents to return false.

Source Error:
Line 66: //LOAD: Compatibility Mode
Line 67: //<meta http-equiv="X-UA-Compatible" content="IE=7,IE=9" />
Line 68: string BrowserOS = Request.ServerVariables["HTTP_USER_AGENT"].ToString();
Line 69: HtmlMeta compMode = new HtmlMeta();
Line 70: compMode.Content = "IE=7,IE=9";


Source File: c:\inetpub\wwwroot\BCTIA\Website\bctia\layouts\Main Layout.aspx.cs   
Line: 68

Stack Trace:
[NullReferenceException: Object reference not set to an instance of an object.]
   Layouts.Main_Layout.Page_Load(Object sender, EventArgs e) in c:\inetpub\wwwroot\BCTIA\Website\bctia\layouts\Main Layout.aspx.cs:68
   System.Web.Util.CalliHelper.EventArgFunctionCaller(IntPtr fp, Object o, Object t, EventArgs e) +24
   System.Web.UI.Control.LoadRecursive() +70
   System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) +3063

To work around this issue, you can just set a user-agent string before making the request:

ini_set('user_agent', 'Mozilla/5.0 (compatible; Examplebot/0.1; +http://www.example.com/bot.html)');
1
  • @Bing True, yet you'll have to re-configure your regex in regards to the <title> problem. I used his workaround and it did find the lang as you suggested earlier, but not the <title>. In order for it to work, you'll have to use $regex = '/<title>(.+?)<\/title>/s';. It won't matter if they're on seperate lines. That worked, I just tried it. $regex = '/<title>([^<]+)<\/title>/'; will also work. Jun 12, 2013 at 5:06
0

Do not use regular expression .. !!

Instead Use xpath take a look at: xpath

regular expression will not work well.

0

Parsing html code with regex isn't a good way because you may be surprised by his permissive structure.

The reason why your pattern don't work is that the dot don't match newlines.

If you want that the dot matches newlines use the s modifier at the end of the pattern, or don't use the dot:

$regex = '/<title>(.+?)<\/title>/s';

or

$regex = '/<title>([^<]+)<\/title>/';

[^<] is a character class that contains all characters but <, as you can see with that you don't need to use a lazy quantifier: + instead of +?

2
  • Hi Casimir, thank you for your answer. However, unfortunately I was not able to get the correct result of extract bctia.org using either of the pattern you suggested :< Could we explore other possibilities together?
    – Bing
    Jun 12, 2013 at 0:33
  • @Bing: the PleaseStand answer solve your first problem, after, using one of the two patten will solve your second. Jun 12, 2013 at 4:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.