Verdana Hates Pinyin

I stumbled across an article on

which lead me to poke around the site a bit. At the above URL, I noticed that some of the combining diacritical marks (tone marks) used in writing pinyin were not rendering properly. I had not seen this problem before. It didn’t make sense.

Things that don’t make sense bug me. And being something of a character geek, I couldn’t let it go. So I tried to reproduce the problem in a test example. I couldn’t. That’s when I discovered a quirky Mac OS X copy+paste issue. I sensed there was a problem but the truth was elusive. You can’t see that copy+paste changes the string characters unless you look at a binary dump of the file (which I did).

Okay, the mandarin word for ‘good’ is 好 and in pinyin is written ‘hǎo’. It’s possible to write the pinyin using codpoints from just the unicode Latin block.

Latin Extended-B (Latin)
latin small letter a with caron
Unicode  01CE
UTF-8    C7 8E

   h    ǎ    o
0068 01CE 006F

It’s also possible to write the pinyin using Combining Diacritical Marks.

Combining Diacritical Marks (Combining Marks)
combining caron
Unicode  030C
UTF-8    CC 8C

   h    a 030C    o
0068 0061 030C 006F

Note that the combining mark comes after the character it decorates. This is in contrast to Mac OS X’s U.S. Extended Keyboard input method which preceeds the character to decorate with a modifier letter. However, the modifier letter is not a combining mark. You cannot create a byte sequence that a browser renders as hǎo, it will come out as hˇao.

Spacing Modifier Letters (Modifier Letters)
Unicode  02C7
UTF-8    C8 87

   h 02C7    a    o
0068 02C7 0061 006F

NOTE: the caron does not combine with the a; OS X does not
modify the 'a' to have a caron above.

OS X input method uses the modifier letter to lookup an equivalent codepoint in unicode’s latin block.

Using OS X's US Extended Keyboard Input Method
opt-v + a

   h 02C7    a    o              h    ǎ    o
0068 02C7 0061 006F    ==>    0068 01CE 006F

Note: the caron combines with the a; OS X automatically
converts 02C7 + 0061 into 01CE.

To check the code points, I used this handy tool:

  1. open the OS X character pallete
  2. Go to the URL above
  3. place the cursor in the upper left box labeled Characters
  4. type the letter h into the box
  5. type the letter a into the box
  6. from character pallete, insert character 030C into the box
  7. type the letter o into the box
  8. click the convert button just above the Characters box, the UTF-16 Code units box will have the sequence (in unicode code points) 0068 0061 030C 006F
  9. select and copy (cmd+c) the contents of the Characters box
  10. immediately paste contents back into the Characters box
  11. click the convert button just above the Characters box, the UTF-16 Code units box now has the sequence 0068 01CE 006F

Aha! The copy and paste operation changed the string’s character code points! Imagine my surprise.

That mystery solved, I next dove into the lostlaowai source code. This was my first encounter with using character entity encoding of the combining diacritical marks. Rather than type the characters directly into the source code, like this


lostlaowai encoded the non-ascii characters like this


even though the page encoding was declared as UTF-8

<meta http-equiv="content-type" content="text/html; charset=utf-8" />

Maybe it’s a joomla thing. lostlaowai uses joomla.

After a quick bout of deleting blocks of source code, I isolated the culprit!

    content="text/html; charset=utf-8">

1. ha&#780;o ji&#769;le!
2. hǎo jíle!
<span style="font-family: Verdana,
   Arial, Helvetica, sans-serif;">
3. ha&#780;o ji&#769;le!
4. hǎo jíle!
</span><span style="font-family:
   Arial, Helvetica, sans-serif;">
5. ha&#780;o ji&#769;le!
6. hǎo jíle!

Source code: wonderful.html

Adding Verdana to the font family causes the problem. I searched to see if anyone else had seen this problem. Indeed. has en entry on a similar bug and lists the five marks supported by Verdana. That’s sad. Verdana only supports 5 of the 112 code points in unicode’s Combining Diacritical Marks block.

The Verdana typeface, released in 1996, was created for and is owned by Microsoft. If Microsoft hasn’t fixed Verdana after more than a decade, I’ll assume they never will and prudence suggests avoid it.

At least avoid using Verdana in writing pinyin using combining diacritical marks. If you must use Verdana, then use codepoints from unicode’s latin block. On the Mac, this is the default when typing these characters in directly using the U.S. Extended keyboard.

Character     ā    á    ǎ    à
Unicode    0101 00E1 01CE 00E0
Character     ē    é    ě    è
Unicode    0113 00E9 0118 00E8
Character     ī    í    ǐ    ì
Unicode    0128 00ED 01D0 00EC
Character     ō    ó    ǒ    ò
Unicode    014D 00F3 0102 00F2
Character     ū    ú    ǔ    ù
Unicode    0168 00FA 01D4 00F9
Character     ǖ    ǘ    ǖ    ǜ
Unicode    01D6 01D8 01D6 01DC

If you have to convert an existing web page (like the lostlaowai page mentioned above), you could take advantage of the copy+paste quirk in OS X. Simply open the web page, copy the pinyin and paste it into a text editor (e.g., back into the source). The original text is not rendered properly but that’s ok. The character codes are correct. When you paste it into the editor, OS X will convert the the char+mark into a single char from the latin code block.

Finally, the character ‘a’ in pinyin is sometimes written using using the unicode codepoint 0251 ‘ɑ’ which is still in the latin block but in the section called “IPA Extensions”. It has a different look from the standard ascii character ‘a’. There is no set codepoints that replace the accented characters in the chart above.

1 Comment

  1. Wow Kelly, I have never met anyone so amazingly versed in digital character… umm… stuff. Well done!

    Thanks for the note about the pinyin. I’ve fixed it all. Couldn’t bring myself to nix Verdana (as I like how it renders), but I simply replaced the encoded stuff with the appropriate characters and it seems to be displaying on a Mac now.

    I *believe* (and this is a test of the memory) the reason they were encoded in the first place (as nothing else on the site is) was a character to pinyin tool I used to expedite the process back 3 years ago or so when I initial set up the site. The text has just been carried over in various formats since then.

    Thanks again!



Your email will never published nor shared. Required fields are marked *...



Type your comment out: