DOMDocument and UTF-8 Problem

By  on  

A few weeks back I shared how I used PHP DOMDocument to reliably update all image URLs from standard HTTP to HTTPS.  DOMDocument made a difficult problem seem incredibly easy ... but with one side-effect that it took me a while to spot:  UTF-8 characters were being mutated into another set of characters.  I was seeing a bunch of odd characters like "ãç³" and"»ã®é" all over each blog post.

I knew the problem was happening during the DOMDocument parsing and that I need to find a fix quickly.  The solution was just a tiny bit of code:

// Create a DOMDocument instance 
$doc = new DOMDocument();

// The fix: mb_convert_encoding conversion
$doc->loadHTML(mb_convert_encoding($content, 'HTML-ENTITIES', 'UTF-8'));

After setting the character set with mb_convert_encoding, the odd characters vanished and the desired characters were back in place.  Phew!

Recent Features

  • By
    JavaScript Promise API

    While synchronous code is easier to follow and debug, async is generally better for performance and flexibility. Why "hold up the show" when you can trigger numerous requests at once and then handle them when each is ready?  Promises are becoming a big part of the JavaScript world...

  • By
    Convert XML to JSON with JavaScript

    If you follow me on Twitter, you know that I've been working on a super top secret mobile application using Appcelerator Titanium.  The experience has been great:  using JavaScript to create easy to write, easy to test, native mobile apps has been fun.  My...

Incredible Demos

  • By
    MooTools Typewriter Effect Plugin

    Last week, I read an article in which the author created a typewriter effect using the jQuery JavaScript framework. I was impressed with the idea and execution of the code so I decided to port the effect to MooTools. After about an hour of coding...

  • By
    CSS Selection Styling

    The goal of CSS is to allow styling of content and structure within a web page.  We all know that, right?  As CSS revisions arrive, we're provided more opportunity to control.  One of the little known styling option available within the browser is text selection styling.

Discussion

  1. Markus

    instead of converting the input string from your encoding into UTF8 you can also tell the DOMDocument with the 2nd arg in which encoding your string is.

    http://php.net/manual/en/domdocument.construct.php

    This should save you some cpu-cycles and reduce memory consumption.

  2. Whelping

    Actually I was already doing

    $html = new DOMDocument( null, 'UTF-8' );

    and still getting weird characters like  – adding your fix did the trick for me, thanks David!

    • Whelping

      It turns out that specifying the encoding argument when you instantiate a DOMDocument doesn’t encode the contents, just sets the document’s header – see this comment http://php.net/manual/en/domdocument.construct.php#78027. It probably wasn’t necessary to specify UTF-8 anyway, as that’s the default. So something like David’s fix is needed to change any unwanted encodings in the content.

  3. Raja Amer Khan

    Thanks for the tutorial. It really saved time.

  4. Dario

    Thank you very much you life saver

  5. Sam

    Thank you very for this :D
    Caught this bug too, even after instantiating it properly

  6. Rafael

    Awesome, character encoding has always been a pain the a.

  7. Thank you very much David.
    This helped me a lot. because I had bad characters in wordpress the_content.
    This method removed special characters pasted from word document.
    :D

  8. Thanks a lot, David.
    I was about to pull my hair out :D

Wrap your code in <pre class="{language}"></pre> tags, link to a GitHub gist, JSFiddle fiddle, or CodePen pen to embed!