Folder Stage to filter Record

Post questions here relative to DataStage Server Edition for such areas as Server job design, DS Basic, Routines, Job Sequences, etc.

Moderators: chulett, rschirm, roy

Ocean
Participant
Posts: 18
Joined: Tue Jul 18, 2006 1:51 am

Post by Ocean »

chulett wrote:You can't really stop 'reading' at the first record delimiter. What I supposed the suggestion is would be to attempt to pull off the first 'record' in the Derivation after the entire file has been read in. May work... but I don't think that's the right choice of character to look for. Is this a UNIX file or a DOS file?

Try with the FIELD function, telling it the field delimiter is a CHAR(254) and that you want the first field. If that doesn't work, try again with CHAR(10) or CHAR(13).

And no, the derivation field in the Folder stage is not active. All objects of that nature use the same GUI 'widget' but not every stage actually uses all of the components.
It is DOS file. Is there any different in this case? Field function works but data has to load into transformer first to use Field funtion. Right?
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

Yes, DOS record terminators are different from UNIX ones. UNIX uses a single LF character while DODS uses a combination of CR/LF, so I'd suggest you look for the a CR - a CHAR(13) - in your derivation.

Yes, the data has to load into the Transformer first to use the Field or any other functions on the data, but that can't be avoided if you use this Folder mechanism that's been discussed. The entire file will be read in as a single record and then from that physical record you'll hack off the first logical record.

Me, personally, I wouldn't take this approach as it would fail on 'large' file sizes, whatever large means here. If available, I'd do a "head -1" in the Filter of the Sequential File stage to just get the first record. Either that or the rather brain-dead method of reading the file normally and then using a constraint of @INROWNUM=1 in the Transformer to ensure only the first record is passed through. You'd be amazed how quickly it will happily read in and pitch records 2-n. :wink:

Edited to add: Ack, forgot about the 'and get the filename' part until I reread everything after posting. You'll need to stick with the Folder stage for that. Either that or perhaps a scripted 'pre process' that goes through each file and builds something specific for this processing, two record files which just have the filename as the first record and the first record as the second record, if that makes sense.
-craig

"You can never have too many knives" -- Logan Nine Fingers
Post Reply