DSXchange

Hi,

we have a file with variable number of date values(each time the number of rows changes) which define the number of times the loop would run.
so, wanted to know is there an alternate way to without having to go through looping method. I am not able to think any alternate way.

The reason why I ask for the alternate way is in each loop 100's of billions of records are read and filtered based on the date values and written to different files based on the date values which is taking very long hours to finish. each time entire file needs to be read so I was looking for an alternate way where we can avoid reading the file in a loop and filter the records with just one iteration of records.

Load the dates into a lookup stage, including a filename or file number as part of the table entry, and pass your data against that. A script running in an external target stage could handle writing records to the correct file.

Regards,

well, based on each value the records pertaining to that would be moved to one single file. so having it in the lookup would not help as we cant define multiple file names in the same sequential file stage.

As noted in my first reply, you could use an external target stage which could utilize a script or simple program to handle the multiple files.

I believe that the Folder stage from the Server canvas can also accomplish this. Refer to the Server Job Developer Guide for more detail.

Regards,