DSXchange

Hello,

I have a job designed for reading data using sequential stage where I do some transformations and load to new file.

I have around 2000 files to follow the same process, they all have the same layout and they are dated depending on their arrival.

Is there any processto automate this in a parallel job? Please suggest

Thank you

do you have to read them sequentially, like the one with earliest date should be processed first or the order doesn't matter?

The stage supports a wildcard pattern so you could do them all at once. If they need to be loaded individually (one at a time) build a looping Sequence job.

@Priyadarshikunal
Order does not matter

@Chulett
ok, I will use wild card like filename*.txt so that it pulls all the files.
When loading it does not need to be loaded one at a time, new files can be created all at a time.
I will use looping if they need to be loaded one at a time in sequence, if that is the case shall the use the same parallel job in the sequence job and execute it at a time?

Thank you

Yes, for a loop you can use 'the same job' as long as it doesn't use a wildcard but instead passes the filename in as a job parameter each time.

Are you expecting your input files to come in ad-hoc while your job is executing? If not, you could concatenate your files using operating system commands prior to processing using a command stage in the sequence.