Hi,
we have a file with variable number of date values(each time the number of rows changes) which define the number of times the loop would run.
so, wanted to know is there an alternate way to without having to go through looping method. I am not able to think any alternate way.
The reason why I ask for the alternate way is in each loop 100's of billions of records are read and filtered based on the date values and written to different files based on the date values which is taking very long hours to finish. each time entire file needs to be read so I was looking for an alternate way where we can avoid reading the file in a loop and filter the records with just one iteration of records.
Avoid looping
Moderators: chulett, rschirm, roy
Load the dates into a lookup stage, including a filename or file number as part of the table entry, and pass your data against that. A script running in an external target stage could handle writing records to the correct file.
Regards,
Regards,
- james wiles
All generalizations are false, including this one - Mark Twain.
All generalizations are false, including this one - Mark Twain.
-
- Participant
- Posts: 150
- Joined: Tue Mar 13, 2007 1:17 am
As noted in my first reply, you could use an external target stage which could utilize a script or simple program to handle the multiple files.
I believe that the Folder stage from the Server canvas can also accomplish this. Refer to the Server Job Developer Guide for more detail.
Regards,
I believe that the Folder stage from the Server canvas can also accomplish this. Refer to the Server Job Developer Guide for more detail.
Regards,
- james wiles
All generalizations are false, including this one - Mark Twain.
All generalizations are false, including this one - Mark Twain.