Encoding to support English, Western European Multi Lingual

[memrinal]

Hi All,

We have a requirement, wherein we need to process the data for several countries through the same set of jobs belonging to the same project.

the data may be with one of the following encodings
1. English
2. Western Euraopean ( Which along with English is suppported by ISO-88591)
3. Multi Lingual Data - For countris like Japan, China etc.

The final output files have to be UTF-8

The same set of jobs in the same project has to be used across all the countries.

We can have wrapper jobs to convert the encoding of input files from one encoding to another, but for the main jobs in the project there should be one encoding.

If any one has faced such a situation, please let me know the approach that was taken.

Please let me know what encoding i should use and also, with which encoding i should read the data for different encodings, in the wrappers.

Thanks in Advance.