Page 1 of 1
Urgent question: How to dynamic set file names parameter
Posted: Wed Jun 21, 2006 7:48 pm
by xmw2000
Dears,
sorry to bother! I have one urgent question, hope you can give some suggestions! Thanks in advance.
From the logic, the job is very simple. It just analysis one input file, then according to the file content, output the related results to different sequential files and these sequential file name is dynamic.
Take an example, if input file contains "American", the it will put all the American data into "American.txt" file, while if contains "flower", the output file name should be "flower.txt".
Of course, we can draw all stages in one panel of DS designer, but it's very, very tedious because we will have 600 more output files. So is there one way to dynamic update file name parameter?
Posted: Wed Jun 21, 2006 8:00 pm
by kcbland
Welcome to our forum.
Your requirement is not one that DataStage is meant to solve. Of course, you could use job parameters for the file names, but because you don't know the file names ahead of time you can't use job parameters.
What I would suggest is that you add a column to the output file, say in the first position, that is a partitioning value. Then, after processing all data into single output file, you can use a simple ksh or perl script to break it into the indeterminate number of output files. That is really easy to do in ksh. If the data volume is large, go for a perl, java, or c program. If the data volume is small, ksh is probably fine.
Posted: Wed Jun 21, 2006 8:30 pm
by xmw2000
It's a great idea. I will try to use it!
Appreciate for your help!
Posted: Thu Jun 22, 2006 7:20 am
by sb_akarmarkar
Try with folder stage and aggregator stage...
Thanks,
Anupam
Posted: Thu Jun 22, 2006 8:11 am
by kumar_s
sb_akarmarkar wrote:Try with folder stage and aggregator stage...
Thanks,
Anupam
Do you mean writing the record id as file name for each set of group.
Posted: Thu Jun 22, 2006 8:20 am
by kcbland
For some reason, I was thinking the Folder stage was Windoze only. I checked the 7.5 documentation and it looks like it might work. I'm pretty sure that it's one-row to one-file, unless you pack all rows for a file together. The only way to do this is to SORT by the filename column I suggested, then using stage variables in a Transformer concatenate all rows into a single row, pass to an Aggregator on filename break, use Aggregator LAST derivation on the data to insure all rows come from Transformer, then on to the Folder stage.
Posted: Thu Jun 22, 2006 8:43 am
by parag.s.27
If you know the exact column from where this data is comming then write a routine to read data from file...use help..
then in a sequence use routine execution stage to extact the data and pass it through a user variable stage...and now use this variable in file names
Posted: Thu Jun 22, 2006 8:49 am
by parag.s.27
that is
Code: Select all
USERVARIABLE stage----->Routine Activity Stage---->then for ur final job
give file path here for file name use
and the stage name use: -
<Routine Stage Name>.$ReturnValue
every time this value should be fed to the job as a parameter
Posted: Thu Jun 22, 2006 1:44 pm
by ray.wurlod
Etiquette Note
We don't do "urgent" here. This is an all-volunteer site; no-one gets paid for posting. We post as and when we can. Some of us are asleep when your need is urgent, if indeed it really is. If you want urgent response, sign up with your support provider and learn the true cost of "urgent".
Posted: Fri Jun 23, 2006 4:23 am
by sb_akarmarkar
kumar_s wrote:Do you mean writing the record id as file name for each set of group.
Yes... Kenneth has given solution ... and it works fine ...
Just in concatenation put CHAR(13) and CHAR(10) for new line...
Thanks,
Anupam