To Documents

UTF-8 Example

• Problem: Encode the following characters in binary  and hex UTF-8:
  

  Greek letters: αβγ
  

  Use this ASCII Table and this Hex Digits Table. Also use the table in the Encoding Section of the Wikipedia Article on UTF-8.
  
  Ans: Follow these steps:
  
  1. Translate the characters in
     

     Greek letters: αβγ
     

     into binary ASCII.
     The Unicode codes are the same as the ASCII codes for characters in the Basic Latin character set. Check them out here.
     

     Char:      G        r        e        e        k 
     Hex:     4   7    7   2    6   5    6   5    6   B 
     Binary:01000111 01110010 01100101 01100101 01101101
     
     Char:    space      L        e        t        t     
     Hex:     2   0    4   C    6   5    7   4    7   4 
     Binary:00000010 01001100 01100101 01110100 01111000
      
     Char:      e        r        s        :      space
     Hex:     6   5    7   2    7   3    3   A    2   0
     Binary:01100101 01110010 01110011 00111010 00100000
     

  2. However, the characters α, β, and γ are not in the Latin alphabet; they have Unicode codes between hex 0080 and 7FFF:
     

     α → Hex: 03B1 → Binary: 00000011 10110001
     β → Hex: 03B2 → Binary: 00000011 10110010
     γ → Hex: 03B3 → Binary: 00000011 10110011
     

     Verify these Unicode codes on this webpage. Now use Row 2 of the encoding scheme shown in the Encoding Section of the Wikipedia Article on UTF-8. This row applies to the translation of Unicode codes that are between 0080 hex (00000000 1000000 binary) and 07FF hex (00000111 11111111).
     

     Number of bytes	Bits for code	Minimum code	Maximum code	UTF-8 Byte 1	UTF-8 Byte 2
     2	11	0080	07FF	110xxxxx	10xxxxxx

     
     The blue bits are the 6 rightmost bits of the Unicode code; the red bits are the next 5 bits, counting from the right.  The silver (light gray) bits are the special bits that are inserted to designate a two byte UTF-8 code.  Hence we have the following translations for α, β, and γ:
     

     Character	Hex Unicode	Binary Unicode	Binary UTF-8	Hex UTF-8
     α	03B1	0000001110110001	1100111010110001	CEB1
     β	03B2	0000001110110010	1100111010110010	CEB2
     γ	03B3	0000001110110011	1100111010110011	CEB3

• We can verify the Hex UTF-8 representation by looking at the hex dump of the characters
  

  Greek letters: αβγ
  

  Hex dump:
  

  00000 47 72 65 65 6B 20 6C 65 74 74 65 72 73 3A 20 CE 
         G  r  e  e  k     L  e  t  t  e  r  s  :     . 
  00010 B1 CE B2 CE B3 0D 0A 
         .  .  .  .  . \r \n
  

• To obtain a hex dump on a Windows PC, copy these letters into a NotePad text file named greek-letters.txt:
  

  Greek letters: αβγ
  

  Then open a Command Prompt Window, go to the folder that contains greek-letters.txt, open PowerShell, then enter this line in PowerShell:
  

  > Format-Hex greek-letters.txt
           00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F
  
  00000000 47 72 65 65 6B 20 6C 65 74 74 65 72 73 3A 20 CE
  00000010 B1 CE B2 CE B3

• To create a hex dump on a Mac, copy the characters
  

  Greek letters: αβγ
  

  into a file named greek-letters.txt using the TextEdit editor.  Under Plain Text Encoding, select Unicode (UTF-8). Then enter the test string in a Terminal Window:
  

  Greek letters: αβγ
  

  Finally, exit the editor, open a Terminal Window, navigate to the folder that contains greek-letters.txt, and obtain the hex dump by entering the following at the prompt:
  

  > od -x greek-letters.txt
  

  Actually, od means octal (base 8) dump, but the -x flag obtains the hex dump instead.