public class CharsetToolkit extends Object
Utility class to guess the encoding of a given byte array. The guess is unfortunately not 100% sure. Especially for 8-bit charsets. It's not possible to know which 8-bit charset is used. Except through statistical analysis. We will then infer that the charset encountered is the same as the default standard charset.
On the other hand, unicode files encoded in UTF-16 (low or big endian) or UTF-8 files with a Byte Order Marker are easy to find. For UTF-8 files with no BOM, if the buffer is wide enough, it's easy to guess.
Tested against a complicated UTF-8 file, Sun's implementation does not render bad UTF-8 constructs as expected by the specification. But with a buffer wide enough, the method guessEncoding() did behave correctly and recognized the UTF-8 charset.
A byte buffer of 4KB or 8KB is sufficient to be able to guess the encoding.
Usage:
// guess the encoding Charset guessedCharset = com.glaforge.i18n.io.CharsetToolkit.guessEncoding(file, 4096); // create a reader with the charset we've just discovered FileInputStream fis = new FileInputStream(file); InputStreamReader isr = new InputStreamReader(fis, guessedCharset); BufferedReader br = new BufferedReader(isr);Date: 18 juil. 2002
| Constructor and Description |
|---|
CharsetToolkit(byte[] buffer)
Constructor of the
com.glaforge.i18n.io.CharsetToolkit utility class. |
CharsetToolkit(byte[] buffer,
Charset defaultCharset)
Constructor of the
com.glaforge.i18n.io.CharsetToolkit utility class. |
| Modifier and Type | Method and Description |
|---|---|
static Charset[] |
getAvailableCharsets()
Retrieves all the available
Charsets on the platform, among which the default charset. |
Charset |
getDefaultCharset()
Retrieves the default Charset
|
static Charset |
getDefaultSystemCharset()
Retrieve the default charset of the system.
|
boolean |
getEnforce8Bit()
Gets the enforce8Bit flag, in case we do not want to ever get a US-ASCII encoding.
|
Charset |
guessEncoding()
Guess the encoding of the provided buffer.
|
static Charset |
guessEncoding(File f,
int bufferLength) |
static Charset |
guessEncoding(File f,
int bufferLength,
Charset defaultCharset) |
static Charset |
guessEncoding(org.apache.commons.vfs2.FileObject file,
int bufferLength) |
static String |
guessEncodingName(File file) |
static String |
guessEncodingName(org.apache.commons.vfs2.FileObject file) |
void |
setDefaultCharset(Charset defaultCharset)
Defines the default
Charset used in case the buffer represents an 8-bit Charset. |
void |
setEnforce8Bit(boolean enforce)
If US-ASCII is recognized, enforce to return the default encoding, rather than US-ASCII.
|
public CharsetToolkit(byte[] buffer)
com.glaforge.i18n.io.CharsetToolkit utility class.buffer - the byte buffer of which we want to know the encoding.public CharsetToolkit(byte[] buffer,
Charset defaultCharset)
com.glaforge.i18n.io.CharsetToolkit utility class.buffer - the byte buffer of which we want to know the encoding.defaultCharset - the default Charset to use in case an 8-bit charset is recognized.public void setDefaultCharset(Charset defaultCharset)
Charset used in case the buffer represents an 8-bit Charset.defaultCharset - the default Charset to be returned by guessEncoding() if an 8-bit
Charset is encountered.public static Charset guessEncoding(org.apache.commons.vfs2.FileObject file, int bufferLength) throws FileNotFoundException, IOException
FileNotFoundExceptionIOExceptionpublic static String guessEncodingName(org.apache.commons.vfs2.FileObject file) throws FileNotFoundException, IOException
FileNotFoundExceptionIOExceptionpublic static String guessEncodingName(File file) throws FileNotFoundException, IOException
FileNotFoundExceptionIOExceptionpublic void setEnforce8Bit(boolean enforce)
charset rather than US-ASCII.enforce - a boolean specifying the use or not of US-ASCII.public boolean getEnforce8Bit()
public Charset getDefaultCharset()
public Charset guessEncoding()
Guess the encoding of the provided buffer.
If Byte Order Markers are encountered at the beginning of the buffer, we immidiately return the charset implied by this BOM. Otherwise, the file would not be a human readable text file.If there is no BOM, this method tries to discern whether the file is UTF-8 or not. If it is not UTF-8, we assume the encoding is the default system encoding (of course, it might be any 8-bit charset, but usually, an 8-bit charset is the default one).
It is possible to discern UTF-8 thanks to the pattern of characters with a multi-byte sequence.
UCS-4 range (hex.) UTF-8 octet sequence (binary) 0000 0000-0000 007F 0xxxxxxx 0000 0080-0000 07FF 110xxxxx 10xxxxxx 0000 0800-0000 FFFF 1110xxxx 10xxxxxx 10xxxxxx 0001 0000-001F FFFF 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx 0020 0000-03FF FFFF 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 0400 0000-7FFF FFFF 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
With UTF-8, 0xFE and 0xFF never appear.
public static Charset guessEncoding(File f, int bufferLength) throws FileNotFoundException, IOException
FileNotFoundExceptionIOExceptionpublic static Charset guessEncoding(File f, int bufferLength, Charset defaultCharset) throws FileNotFoundException, IOException
FileNotFoundExceptionIOExceptionpublic static Charset getDefaultSystemCharset()
Charset.public static Charset[] getAvailableCharsets()
Charsets on the platform, among which the default charset.Charsets.