how to remove "shift in/out characters"

[yimwai]

1st,I use PxCFF stage to read an EBCDIC file "F" and load data to table "T", then i read the same file again and use difference stage to compare the data from "T" and "F". ( only compare on column with Chinese characters.)
I find that result is "all different" which is supposed to be "all the same".
I guess that it is caused by shift in/out characters "0E 0F",right?
If right, why DS just discard them when reading the EBCDIC file?then when DS discard the characters?only when writing into the database?can I do the remove action and make the result "all the same"?

[ArndW]

The Shift-In and Shift-Out characters will be the same only if you ensure that you define the two stages (CFF and database) with the same NLS settings, otherwise you won't be able to compare the two streams.

[ray.wurlod]

Shift-In and Shift-Out are sometimes used to indicate a change of mapping (particularly in Japanese character sets). Beware that these might be important characters before arbitrarily discarding them. Once they're inside a job, they should be able to be discarded with a simple Convert() function.