软件一般采用三种方式来决定文本的字符集和编码:
检测文件头标识,提示用户选择,根据一定的规则猜测
最标准的途径是检测文本最开头的几个字节,开头字节Charset/encoding,如下表:
EF BB BF UTF-8
FE FF UTF-16/UCS-2, little endian
FF FE UTF-16/UCS-2, big endian
FF FE 00 00 UTF-32/UCS-4, little endian.
00 00 FE FF UTF-32/UCS-4, big-endian.