Dynamic output files

dsconultant · Post by **dsconultant** » Sun Dec 06, 2009 12:20 am

Hi,
I have a data file which has output like following

abc20091011
abc20091011
abc20091111
abc20091111
xyz20091011
xyz20090909

I need separate files for abc20091011, abc200911, xyz20090909 and so on. Problem is the output is dynamic. It could have any number of records. Right now I have 4 unique data but it will change to maybe 20 in my next run. So for each unique record I need separate file. How can I handle this?

chulett · Post by **chulett** » Sun Dec 06, 2009 9:04 am

I'm not sure if there are any stage in PX that would allow output to dynamic files, perhaps the External Target stage? Server allegedly does - the Folder stage but to be honest I've never used it in that fashion. Or you could always go the 'build op' route and put together something in C++ to do that for you.

Another possibility is building a single file and then 'splitting' the file post job, using something like csplit perhaps.

JoshGeorge · Post by **JoshGeorge** » Sun Dec 06, 2009 7:36 pm

Link to - Datastage parallel (C++) routine to create files dynamically for every record in single job.

http://it.toolbox.com/blogs/dw-soa/data ... ally-21095

dsconultant · Post by **dsconultant** » Tue Dec 15, 2009 9:09 pm

Thanks, I will try the routine.

vinnz · Post by **vinnz** » Wed Dec 16, 2009 6:11 pm

You could try awk as well to post-process your file as suggested above

awk '{ print >> $1 ; close($1)}' yourfilename

keshav0307 · Post by **keshav0307** » Wed Dec 16, 2009 8:07 pm

if you want to do in datastage then , in a transformer stage do a mapping like

'echo ':INPUT_COLUMN:' >> ':INPUT_COLUMN:'.txt'

and the in after job subroutine, just execute the target file.