Handling Line Terminator's

[naren6876]

I have a input file which is fixed width of 100.But, sometimes i am receiving it with Line Terminator and sometimes without any Line terminator.How can i handle this?.

I would be grateful If anyone could shed some light on this?.

Thanks in advance.

[ogmios]

First if it's possible to get the source fixed, fix it.

For the rest there are numerous options:

1) pre-process the files outside of DataStage, call a script before running the job that puts the line terminators right.
2) Use something as Sequential file -> transformer -> ... in which you read lines of 101 bytes long and you substring the first 100... then you do regular processing
3) In the sequential file stage switch of line termination handling, add a column for the line terminator. Then in the columns tab of seqential file stage go to the right (the options you need are normally not visible). There you will find an action to be performed on missing columns. Don't know the correct option to choose out of my head. Disadvantage of this is that it will leave warnings in your job log.

Ogmios

[kcbland]

Should the file be fixed width with end of line terminators or no end of line terminators just 100 characters per "record"? There's a big difference. I don't possibly see how you can fix a file under either circumstance.

[ogmios]

Should the file be fixed width with end of line terminators or no end of line terminators just 100 characters per "record"? There's a big difference. I don't possibly see how you can fix a file under either circumstance.

I assumed he means 100 characters per line fixed width as input but e.g. that because of different sources (and ftp transmissions) in one set of he will either get or not get line terminators. So essentially he just was to have the first 100 characters per line.

Ogmios

[kcbland]

Are there supposed to be line terminators or not?

The first 100 chars of each line means that each line has a terminator, because you are saying LINE.

If there are just 100 char blocks of data, then there is no concept of a LINE, thus a line terminator will mess things up.

Either you have line terminators or not. You can't arbitrarily remove a line terminator (LF) because that is assuming that it is an extraneous character and removing will not shift data. I'm not willing to take a position on doing that.

Per the original post:

sometimes i am receiving it with Line Terminator and sometimes without any Line terminator

No line terminator to me means that two "lines" run together continuously as 100 character blocks.

[naren6876]

Actually iam expecting a file with Line Terminator either fixed width or variable width.So,How can i put a Line terminator if I dont receive it in the file?

Should the file be fixed width with end of line terminators or no end of line terminators just 100 characters per "record"? There's a big difference. I don't possibly see how you can fix a file under either circumstance.

[ogmios]

Actually iam expecting a file with Line Terminator either fixed width or variable width.So,How can i put a Line terminator if I dont receive it in the file?

No go for that... you have different source formats so use diferent DataStage jobs.

[kcbland]

Actually iam expecting a file with Line Terminator either fixed width or variable width.So,How can i put a Line terminator if I dont receive it in the file?

Then 2 lines run together as 1 line? How do you know where the first derived line is separate from the second if they're variable width?

[ratikmishra1]

Dos this file has a header or trailer record that has the record count in the file? If yes, you may think of writing a shell script to check the total bytes of the file with the computed total bytes for size 100 and for size 101. And see which one matches.

Then call the fix_100 or fix_101 job to do the processing.

[naren6876]

If it is variable width , Under the Format tab of sequential file If I specify the delimiter as 013 and Line Termination as UNIX style, then I can see all the LINE's.

Actually iam expecting a file with Line Terminator either fixed width or variable width.So,How can i put a Line terminator if I dont receive it in the file?

Then 2 lines run together as 1 line? How do you know where the first derived line is separate from the second if they're variable width?

[ray.wurlod]

Open the Columns grid, scroll right and find where you can set the "column contains line terminators" rule. This should allow you to handle any intermediate line terminator characters in the data.

[naren6876]

Hi ray,
It doesn't solve my problem.

Is there any way to find out a file has Line Termination of UNIX style or DOS style or None.?

I would be grateful if anyone could shed some light on this.
Thanks in advance.

Open the Columns grid, scroll right and find where you can set the "column contains line terminators" rule. This should allow you to handle any intermediate line terminator characters in the data. ...

[kcbland]

I think Ray neglected to read the entire discussion, otherwise he hopefully wouldn't have given a bad answer.

[ray.wurlod]

Ultimately the answer to the original problem is to force the supplier of the file to be consistent in the style of delivery. Other than that, test the fnal byte (prior to any end-of-file indicator) of the file to determine whether it is a line termination character, if it isn't then you have an unterminated record structure. Based on this outcome, start either a job that handles terminated records or a job that handles unterminated records. You can not do both in the one job, since line termination is a stage property that can not be parameterized.

It was the apostrophe that made me write a bad answer!