how to check whether source file is correct or not

praveenk · Post by **praveenk** » Fri Feb 18, 2011 6:15 am

hi,

i have a src file which i get every day for example src_20110218 (src_yyyymmdd) and again next day will be src_20110219..
i need to check whether iam picking the right file every day or not before reading the records in the file.

can anyone help me on this

Thankx a lot in advance

jwiles · Post by **jwiles** » Fri Feb 18, 2011 6:56 am

Is the file consistently named as in your example? src_YYYYMMDD

If so, you could try this: As your sequential file stage Source option, use the File Pattern instead of File and specify the following as your file pattern:

ls /pathname/src_*|tail -n 1

File Pattern allows you to specify a shell command which returns a list of files to process. The command above should return the latest file based on the example filenames you provided.

Regards,

chulett · Post by **chulett** » Fri Feb 18, 2011 7:36 am

Define for us in words what "the right file" means here... what are your rules to validate the "rightness" of it?

praveenk · Post by **praveenk** » Sat Feb 19, 2011 12:13 am

chulett: right file exactly means the latest file..
as i get the file which is in the format of 'Src_yyyymmdd' every day for example src_20110219 and again next day filename will be src_20110220, and i need to run the datastage job every day which reads the source file and do some transofmation on that....so, before i read the file i need to check whether it is the latest file or not....hope u got my requirement...

jwiles: i'll try your code and let you know...thaks for your answer

kkalyanrao@gmail.com · Sat Feb 19, 2011 12:49 am

Consider moving the source file to some other directory (archiving it) once it is processed by your job.

chulett · Post by **chulett** » Sat Feb 19, 2011 8:13 am

Saying the "latest" file doesn't really narrow things down, I was asking how strict you need to be in ensuring you have just recieved the "right" file. Sure, one simple way to make detection easier is to move processed files to another area, an archive area for example, once your done so you don't have to worry about processing it again accidently.

However, what if the next day they send you the same file again? Is it ok to reprocess it or do you need to be smart enough to recognize the problem? What if they skip a day? Do you need to recognize there is a problem if you get these two files a day apart?

src_20110218
src_20110220

Those are the kind of things that I was asking you to think about and lay out for us.

chulett · Post by **chulett** » Sat Feb 19, 2011 8:14 am

ps. I moved this to the General forum as it really is a general topic and not specific to any of the job types.

greggknight · Post by **greggknight** » Sun Feb 20, 2011 12:20 am

I agree with kkalyanrao,
I recieve trigger files everyday from an external process when they have completed their endday processing which contains an endday number that I use in a where clause to select my data.

I use a wait for file and after the file arives and I have read its contents I move the file to an archive directory. If there is a file in the landing directory it is the one I want to process.

Your just asking for issues by leaving already processed files laying around.