O'Reilly

Parse Web Pages with PHP Simple HTML DOM Parser

By on  
Gervinho to Arsenal

For those of you who have had the pleasure of following me on Twitter (...), you probably know that I'm a complete soccer (football) fanatic.  I even started a separate Twitter account to voice my footy musings.  If you follow football yourself, you'll know that we've just started the international transfer window and there are a billion rumors about a billion players going to a billion clubs.  It's enough to drive you mad but I simply HAVE TO KNOW who will be in the Arsenal and Liverpool first teams next season.

The problem I run into, besides all of the rubbish reports making waved, is that I don't have time to check every website on the hour.  Twitter is a big help, but there's nothing better during this time than an official report from each club's website.  To keep an eye on those reports, I'm using the power of PHP Simple HTML DOM Parser to write a tiny PHP script that shoots me an email whenever a specific page is updated.

PHP Simple HTML DOM Parser

PHP Simple HTML DOM Parser is a dream utility for developers that work with both PHP and the DOM because developers can easily find DOM elements using PHP. Here are a few sample uses of PHP Simple HTML DOM Parser:

// Include the library
include('simple_html_dom.php');
 
// Retrieve the DOM from a given URL
$html = file_get_html('http://davidwalsh.name/');

// Find all "A" tags and print their HREFs
foreach($html->find('a') as $e) 
    echo $e->href . '<br>';

// Retrieve all images and print their SRCs
foreach($html->find('img') as $e)
    echo $e->src . '<br>';

// Find all images, print their text with the "<>" included
foreach($html->find('img') as $e)
    echo $e->outertext . '<br>';

// Find the DIV tag with an id of "myId"
foreach($html->find('div#myId') as $e)
    echo $e->innertext . '<br>';

// Find all SPAN tags that have a class of "myClass"
foreach($html->find('span.myClass') as $e)
    echo $e->outertext . '<br>';

// Find all TD tags with "align=center"
foreach($html->find('td[align=center]') as $e)
    echo $e->innertext . '<br>';
    
// Extract all text from a given cell
echo $html->find('td[align="center"]', 1)->plaintext.'<br><hr>';

Like I said earlier, this library is a dream for finding elements, just as the early JavaScript frameworks and selector engines have become. Armed with the ability to pick content from DOM nodes with PHP, it's time to analyze websites for changes.

The Script

The following script checks two websites for changes:

// Pull in PHP Simple HTML DOM Parser
include("simplehtmldom/simple_html_dom.php");

// Settings on top
$sitesToCheck = array(
					// id is the page ID for selector
					array("url" => "http://www.arsenal.com/first-team/players", "selector" => "#squad"),
					array("url" => "http://www.liverpoolfc.tv/news", "selector" => "ul[style='height:400px;']")
				);
$savePath = "cachedPages/";
$emailContent = "";

// For every page to check...
foreach($sitesToCheck as $site) {
	$url = $site["url"];
	
	// Calculate the cachedPage name, set oldContent = "";
	$fileName = md5($url);
	$oldContent = "";
	
	// Get the URL's current page content
	$html = file_get_html($url);
	
	// Find content by querying with a selector, just like a selector engine!
	foreach($html->find($site["selector"]) as $element) {
		$currentContent = $element->plaintext;;
	}
	
	// If a cached file exists
	if(file_exists($savePath.$fileName)) {
		// Retrieve the old content
		$oldContent = file_get_contents($savePath.$fileName);
	}
	
	// If different, notify!
	if($oldContent && $currentContent != $oldContent) {
		// Here's where we can do a whoooooooooooooole lotta stuff
		// We could tweet to an address
		// We can send a simple email
		// We can text ourselves
		
		// Build simple email content
		$emailContent = "David, the following page has changed!\n\n".$url."\n\n";
	}
	
	// Save new content
	file_put_contents($savePath.$fileName,$currentContent);
}

// Send the email if there's content!
if($emailContent) {
	// Sendmail!
	mail("[email protected]","Sites Have Changed!",$emailContent,"From: [email protected]","\r\n");
	// Debug
	echo $emailContent;
}

The code and comments are self-explanatory.  I've set the script up such that I get one "digest" alert if many of the pages change.  The script is the hard part -- to enact the script, I've set up a CRON job to run the script every 20 minutes.

This solution isn't specific to just spying on footy -- you could use this type of script on any number of sites.  This script, however, is a bit simplistic in all cases.  If you wanted to spy on a website that had extremely dynamic code (i.e. a timestamp was in the code), you would want to create a regular expressions that would isolate the content to just the block you're looking for. Since each website is constructed differently, I'll leave it up to you to create page-specific isolators. Have fun spying on websites though...and be sure to let me know if you hear a good, reliable footy rumor!

Track.js Error Reporting

Upcoming Events

Recent Features

  • Convert XML to JSON with JavaScript

    If you follow me on Twitter, you know that I've been working on a super top secret mobile application using Appcelerator Titanium.  The experience has been great:  using JavaScript to create easy to write, easy to test, native mobile apps has been fun.  My...

  • 7 Essential JavaScript Functions

    I remember the early days of JavaScript where you needed a simple function for just about everything because the browser vendors implemented features differently, and not just edge features, basic features, like addEventListener and attachEvent.  Times have changed but there are still a few functions each developer should...

Incredible Demos

  • Scroll IFRAMEs on iOS

    For the longest time, developers were frustrated by elements with overflow not being scrollable within the page of iOS Safari.  For my blog it was particularly frustrating because I display my demos in sandboxed IFRAMEs on top of the article itself, so as to not affect my site's...

  • Using jQuery and MooTools Together

    There's yet another reason to master more than one JavaScript library: you can use some of them together! Since MooTools is prototype-based and jQuery is not, jQuery and MooTools may be used together on the same page. The XHTML and JavaScript jQuery is namespaced so the...

Discussion

  1. Kecs

    Dear David,

    It’s cool, but i think you could do it a better way!
    If the target website has an advertise system, every time when you download it, you will get a different source code becase the advert image or string is different…

    For this reason, i would like to offer the Simple Html Dom or other server side object, which could reach the real content of webpages by IDs! If you know what is the ID of content div then you can exclude the ads, and other unwanted elements for example last forum posts :)
    And you could compare the real contents of webpages :)

    Daniel

    • Outstanding suggestion Kecs! I’ll be making a major update to this website shortly.

      Cheers!

  2. noop

    You’ve got two ending semi-colons in your example:

    $currentContent = $element->plaintext;;

    Thanks for the script. Very useful. :)

  3. flies

    Keep in mind that it’s very easy to cause a memory leak in this script. I’ve had this problem a few months back. It’s memory grews rapidly if you want to reuse your $html / $element variable. Let’s take for example $html variable.

    If you use it like that:


    foreach($file in $files)
    {
    $html = str_get_html($file);
    // do something bla bla bla
    }

    you will get a huge memory leak very fast.

    You must always keep in mind to clear your variable (just like that):


    foreach($file in $files)
    {
    $html = str_get_html($file);
    // do something bla bla bla

    $html->clear();
    }

  4. Lenny

    I use this library extensively. I’m thinking of porting it to C#. The brother to this in C# uses Xpath.

  5. Good share david you following facebook and twitter

  6. GER – VIN – HO … la la la lala

  7. Greg

    I had no idea about this. Can’t wait to give it a try.

    Thanks David!

  8. David: this is awesome! I had no idea this PHP library existed; it’ll make my current project (scraping microformats out of Lanyrd) much easier. Thanks for the easy-to-follow code!

  9. Being a JavaScript guy, I figured you’d go for something like phpQuery for parsing the HTML.

    http://code.google.com/p/phpquery/

    • Wow, that looks interesting. I’ll have a look at that too!

  10. This will be very nice next time I need to write a spider.

  11. subtain

    can any one tell me that how to parse the select option value….please write code for this

  12. michel j

    Hi, great website and examples!

    I`ve been puzzling for over a week to somehow get data from a Google website into my online database table (mySQL). Here`s the thing:

    The following website contains a table with historical stock data of AEGON:
    http://www.google.com/finance/historical?q=AMS:AGN&start=0&num=160

    I want to somehow download this table into a MYSQL table. It must get the same headers, so that would be: Date, Open, High, Low, Close, Volume. The PHP code should import all 160 lines of data.

    I tried numerous php codes from the web, but always something was wrong.

    any tips you have for me ;-)
    THANKS!
    Regards,
    Michel

    • Awais

      Hi Michel,

      Recently i was tried the same, did you succeed? i want to save the Google result links into my database.

  13. Daniel

    The htmlDOM Parser don’t work in some sites like for example livingsocial[dot]com or Facebook

    I’ts there any solution to this – I’m new to this but I think is something to do with the browser name (there is no browser name when fetching the website, so the website block the simple htmlDOM parser)

    Sorry for my English
    Daniel

  14. @Daniel,
    Please try BlaineSch suggestion http://code.google.com/p/phpquery/ since that can emulate the javascript as well.

  15. saijin

    On:

    $url = 'http://www.google.com/search?hl=en&q=php&btnG=Search';

    // Create DOM from URL
    $html = file_get_html($url);

    // Match all 'A' tags that have the class attribute equal with 'l'
    foreach($html->find('a[class=l]') as $key => $info)
    {
    echo ($key 1).'. '.$info->plaintext."\n";
    }

    Can someone give me a hint on how can I output the data on each table cell.
    Example:

    Data 1
    Data 2
    Data 3
    Data 4

    Thanks in advance.

  16. Thank you for sharing! I’ve been looking for something that functions like innertext. It’s not documented on any of the sites that I’ve visited including the homesite.

  17. Thank you for sharing! I’ve been looking for something that features like innertext. It’s not recorded on any of the websites that I’ve frequented such as the homesite.

  18. Darius

    Awesome library, I made parsing demo http://dari.us.lt/demo/parser/ from page http://www.skelbikas.lt/ , really easy, thanks :)

    • Omer Rosenbaum

      Would you be able to share the script?

  19. Jones

    Just though you might like to know this page ( http://davidwalsh.name/php-notifications ) displays poorly on my system using Firefox. I can not see the full width of the page until I open the window very wide, then the page shrinks in width to about 25% of the browser window. Pretty strange.

  20. Rejitha

    hi David,

    Have been struggling with a problem related to simplehtmldom. hope atleast i will get an help from you.

    I am using PHPword and simplehtmldom to convert html to docx format.

    I am using fckeditor html output as the input for converting to docx. A single line break is reflected as multiple lines in the docx converted file.

    How do i fix this ? The enter key used in fckeditor is adding a p tag in the html output. Is there any number of line breaks defined for a para tag. how do i change it or where do i change it.

    Please help

  21. I have tried accessing some news sites but I cannot access the information using the code.

  22. Great and useful tutorial, thanks very much!

  23. I tried to use for a wordpress plugin but not sure why it always return a empty object.

  24. erata

    hi. thanks for tutorial. i have taken mistakes they say
    ————————————————————————————————-..
    Warning: file_get_contents(http://davidwalsh.name/): in C:\xampp\htdocs\simple_html_dom\simple_html_dom.php on line 75

    Fatal error: Call to a member function find() on a non-object in C:\xampp\htdocs\simple_html_dom\index.php on line 9
    —————————————————————————————————–
    firstly i used simple html dom 1.5 version and then 1.1 but nothing changed.
    i don’t find where is the problem. what should i do?

    thanks for tutorial and offer.

  25. kalloo

    Thank you David , it save my time.Great work.Thank you very much…

  26. To exclude ads we can use regex something like

    preg_replace(‘/<script(.*)/s’,”,$variable);

  27. The WORKING version of my script is as follows, but I am trying to modify it to only run the loops on submit with PHP:

    find('.article') as $element) {
            if($element->class == 'header2') {
                 $ret['header2'] = $element->value;
            }
        }
    
        $ret['Glossary Term:'] = '' . $html->find('h2[class="header2"]', 0)->innertext . '';
        $ret['Definition:'] = '' . $html->find('.article p', 0)->innertext . '';
    
        return $ret;
    }
    
    
    $links = array (
    ‘http://www.ucla.edu/campus-life/la-lifestyle’,
    ‘http://www.ucla.edu/students/prospective-students’,
    ‘http://www.ucla.edu/’,
    );
    
    ?>
    
    
    
    
    
    
    
    
    
    	body { margin: 0 auto; width:960px; padding: 20px; border: 1px solid #ccc; }
    	h2 { margin:0; pading:0; }
    	p { margin-top:0; padding-top:0; }
    
    
    
    	$v) {
            echo ''.$k.''.$v.'';
        }
    }  
    
    
  28. azi

    https cannot load by simple html dom why.? can we fix this.?

  29. Noel Whitemore

    Thank you David – a very helpful article.

    azi – you can use other methods for retrieving HTML content over HTTPS, such as file_get_contents() or cURL.

  30. Omer Rosenbaum

    Any idea how do I add a search form to the script above when scraping a DIV from a webpage?

  31. The page i am trying to crawl has 4 html tables in it and I want the data in the 3rd table only, any idea on how to get this done?

  32. Hi,

    I want to know that is there is only one method find ? I want to grab one elements value and insert it somewhere else in the dom as we usually do in jQuery. How can we do in this library ?

  33. steve

    why using a library when there are so much powerful tools in php like dom->xPath?? Can’t understand

  34. sunil

    Call to a member function find() on a non-object in C:\xampp\htdocs\scrapping\simplehtmldom\simple_html_dom.php on line 1113
    error in inner loop

Wrap your code in <pre class="{language}"></pre> tags, link to a GitHub gist, JSFiddle fiddle, or CodePen pen to embed!