Setting File size

gomez · Post by **gomez** » Fri Mar 09, 2007 7:28 pm

Hi:
I am facing a strange issue.
I have a routine looking for a file (source.txt)to arrive. This routine is equivalent of a "WAIT FOR FILE" activity but created to log informational message when it times out.
Once a file is found, the control is passed on to another simple routine which copies (DOS Copy command) the file onto a destination folder.

We received a 207 KB file at the source today which went to the target with a size of 204 KB. Apparently there were 153 records in the source file but only 150 made it to the target. Another attempt in processing the same file sent 149 records to the target

I tried to reproduce the issue so we could address the problem. When I repeat the process in our test environment with the same file, the entire file is picked up and written to the target. Hence I am not able to arrive at why this happens. I checked the settings in DataStage Adminstrator for both test and Production environments. They seem to match

Any ideas/suggestions on this?

Is this a DataStage Issue at all or Network or file system? !!!

Thanks,
Gomathy

kumar_s · Post by **kumar_s** » Fri Mar 09, 2007 8:58 pm

hi Gomathy,

"We received a 207 KB file at the source today which went to the target with a size of 204 KB." Is it after any transformation rules, or just the COPY you are referring about?
Make sure you are waiting for the file to read till the last record is appeard. There is a chance where you might have copied the partial data and compared with fully available file.

ray.wurlod · Post by **ray.wurlod** » Fri Mar 09, 2007 9:16 pm

Get them to send two files - the data file then another, possibly empty file.
Wait for the second file, then process the first. Given that they don't send the second file till after they've sent the first, you can be confident that all of the first file has been received when the second file arrives (network contention excepted).

gomez · Post by **gomez** » Thu Mar 15, 2007 3:04 pm

Thanks Kumar and Ray for the suggestions
I guess the possible reason could be what kumar_s suggested. The file was copied on to the destination before being completely written.
I am doing a copy file command from source to target. There are no transformations involved at this step

Ray, I happen to know that the source system generates a statistics file on every file write to indicate the number of records written to the file. I am looking at working on the job to poll for the statistics file than the source file.

But during this investigation, I also thought I would do a DOS move than a copy command. I tested this step - Just to make sure move cannot happen when a file is being written. I tested by opening the source text file in our development environment and tried running the job that moves the file. The file was infact moved. There was no lock on the file though it was opened for editing

Any idea if the write operation's lock applies to .txt files or not?

Thanks again.

DSguru2B · Post by **DSguru2B** » Thu Mar 15, 2007 3:12 pm

Pay attention to what Ray said. You always have to send the trigger file which is usally an empty file after your main file transfer is complete. Hence get them to send two files and wait for the trigger file (second file).

gomez · Post by **gomez** » Thu Mar 15, 2007 5:31 pm

Thanks DSGuru2B
This process involves many applications with different user groups. Hence changing their process would involve considerable effort.

I was looking at leveraging what is currently available from them before going ahead asking them to change their process

Thanks

chulett · Post by **chulett** » Thu Mar 15, 2007 5:39 pm

Didn't you say they also send a 'stats' file? Couldn't you use that as you've mentioned earlier, poll for that file to trigger the processing of the main data file? Assuming they send it last, of course.

The other 'work arounds' I've seen people use:

1) When the file shows up, snag the file size. Continue polling until the file size hasn't changed for an agreed upon period of time.

2) Utilize a utility like 'fuser', something that would let you know the file is 'in use' which should mean 'still be transferred'. Continue checking until the command lets you know the file is no longer 'in use'.

Food for thought. Neither is fool-proof of course, but they're better than nothing.

gomez · Post by **gomez** » Tue Mar 20, 2007 3:04 pm

Thanks. I pointed the polling to Stats file and it is running fine now

Thanks again