Reading multiple xml files using External Source stage

skadiam · Post by **skadiam** » Mon Feb 20, 2012 12:09 pm

Hi, I tried reading multiple xml files using External Source stage.. I mean I used the ls command to list out the files in the dir. But the issue is there are around 50000 files in the dir and I am getting the error like too many return values. I shall send the exact error by tomorrow. Seems like the list is large and not able to capture this list. When I tried say some 25000 files its working fine..Is there a way to read all the 50000 files at the same time.

eostic · Post by **eostic** » Mon Feb 20, 2012 1:59 pm

Please start a new topic with this new and independent question....

chulett · Post by **chulett** » Tue Feb 21, 2012 7:57 am

So let it be written, so let it be done.

A quick google for "argument list too long" (which is what I assume your issue is) should help with alternatives to the ls you are doing.

skadiam · Post by **skadiam** » Wed Feb 22, 2012 4:58 am

Hi Thanks a lot for your advise, I got the solution for this. I used the below command in the External source stage and it worked fine..

find ./ -maxdepth 1 -name "*.txt"

Thanks again..