Runtime sequential file creation

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
RAJEEV KATTA
Participant
Posts: 103
Joined: Wed Jul 06, 2005 12:29 am

Runtime sequential file creation

Post by RAJEEV KATTA »

I have a scenario where in the source is a file with multiple columns.
One of the column name is cost center & the values in it are dynamic.if there are 10 cost centers then target should create 10 files with cost center=1 to first file and cost center=2 to second file.Depending on no of cost centers in input being generated dynamically no of files with respective cost center data should be created in target.What stage do I need to use for the above solution.
Minhajuddin
Participant
Posts: 467
Joined: Tue Mar 20, 2007 6:36 am
Location: Chennai
Contact:

Post by Minhajuddin »

Well, there is no "stage" in Datastage which does this automatically. You may have create a Custom stage for this purpose. Or you can do it by creating a Parallel routine, which will do this for you.
Minhajuddin

<a href="http://feeds.feedburner.com/~r/MyExperi ... ~6/2"><img src="http://feeds.feedburner.com/MyExperienc ... lrow.3.gif" alt="My experiences with this DLROW" border="0"></a>
JoshGeorge
Participant
Posts: 612
Joined: Thu May 03, 2007 4:59 am
Location: Melbourne

Post by JoshGeorge »

Try a parallel routine. See if this POST helps.
Joshy George
<a href="http://www.linkedin.com/in/joshygeorge1" ><img src="http://www.linkedin.com/img/webpromo/bt ... _80x15.gif" width="80" height="15" border="0"></a>
wahi80
Participant
Posts: 214
Joined: Thu Feb 07, 2008 4:37 pm

Re: Runtime sequential file creation

Post by wahi80 »

RAJEEV KATTA wrote:I have a scenario where in the source is a file with multiple columns.
One of the column name is cost center & the values in it are dynamic.if there are 10 cost centers then target should create 10 files with cost center=1 to first file and cost center=2 to second file.Depending on no of cost centers in input being generated dynamically no of files with respective cost center data should be created in target.What stage do I need to use for the above solution.
Step 1: Get Distinct no: of cost centers store in a file say DistinctCost.txt. eg: Lets say a value 10 is stored
Step 2: In a sequencer do the following steps:
Use exec command stage to read the value stored in DistinctCost.txt (cat pathaname/DistinctCost.txt)
Step 3:Start Loop use the command output value of exec command stage in the max value of the loop so that the loop is executed only 10 times
Step 4: Have a job which will read your input file (the one that contains the cost center column and other data) the job should look some what like this SFile --> Transformer -->TFile

Define a parameter (#Counetrval#) in the job this parameter should use the counter value of the loop eg: in first looping 1 will be that value passed to (#Counetrval#) ..in second round 2 will be passed
In the transformer constarint give a constraint where cost center value= Counetrval
In TFile define the name of file as xx#Counetrval#, this will ensure filename as xx1 infirst looping xx2 in second looping and so on

Regards
Wah
Post Reply