pass column value as parameter
Moderators: chulett, rschirm, roy
-
- Premium Member
- Posts: 296
- Joined: Sun Nov 16, 2008 7:41 pm
pass column value as parameter
i have job like
dataset 1 -- > transformer ---> sequential file.
let say dataset 1 has data like
a,1,2
b,1,2
c,1,2
Now i wish to create three files named a,b,c and a will contain record a,1,2
b will contain b,1,2 and c will contain c,1,2.
Some way i have to pass col1 values as paramter to sequenntial file name.
Is this possible. Depending on the value of col1 in dataset, a new sequential file need to be created. If the file is existing, it will append or if file doesnt exist it should be created. Please reply.
dataset 1 -- > transformer ---> sequential file.
let say dataset 1 has data like
a,1,2
b,1,2
c,1,2
Now i wish to create three files named a,b,c and a will contain record a,1,2
b will contain b,1,2 and c will contain c,1,2.
Some way i have to pass col1 values as paramter to sequenntial file name.
Is this possible. Depending on the value of col1 in dataset, a new sequential file need to be created. If the file is existing, it will append or if file doesnt exist it should be created. Please reply.
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Hard coding a link for every col1 value possibility is one way. The links that don't process any rows will still create files - you can clean these up with an after-job subroutine.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
-
- Premium Member
- Posts: 296
- Joined: Sun Nov 16, 2008 7:41 pm
I really do not know all the possible values for col1. It can be 10,20 even 50. Thats why i wish to do it dynamically. So we do not have a way to pass the col1 values as a parameter to file name.ray.wurlod wrote:Hard coding a link for every col1 value possibility is one way. The links that don't process any rows will still create files - you can clean these up with an after-job subroutine.
-
- Participant
- Posts: 3593
- Joined: Thu Jan 23, 2003 5:25 pm
- Location: Australia, Melbourne
- Contact:
It looks like a funny way to share data - who creates dozens of tiny sequential files? What's the purpose? I think this can be done quite easily via XML files just by configuring the XML output to create a different file for each key field but it's not the type of thing you would want to do with sequential files due to the file maintenance overhead.
Maybe a Parallel job that reads the Dataset and uses a BASIC Transformer to call a Server Routine that writes data out to a file.
Maybe a Parallel job that reads the Dataset and uses a BASIC Transformer to call a Server Routine that writes data out to a file.
Certus Solutions
Blog: Tooling Around in the InfoSphere
Twitter: @vmcburney
LinkedIn:Vincent McBurney LinkedIn
Blog: Tooling Around in the InfoSphere
Twitter: @vmcburney
LinkedIn:Vincent McBurney LinkedIn
-
- Participant
- Posts: 246
- Joined: Mon Jun 30, 2008 3:22 am
- Location: New York
- Contact:
Re: pass column value as parameter
You Can write all your column1 values to a file as comma delimited and create a sequence with loop and pass this value to you counter and to your file name parameter.reachmexyz wrote:i have job like
dataset 1 -- > transformer ---> sequential file.
let say dataset 1 has data like
a,1,2
b,1,2
c,1,2
Now i wish to create three files named a,b,c and a will contain record a,1,2
b will contain b,1,2 and c will contain c,1,2.
Some way i have to pass col1 values as paramter to sequenntial file name.
Is this possible. Depending on the value of col1 in dataset, a new sequential file need to be created. If the file is existing, it will append or if file doesnt exist it should be created. Please reply.
Arun
-
- Participant
- Posts: 3337
- Joined: Mon Jan 17, 2005 4:49 am
- Location: United Kingdom
You can write the output file and then split them in an after-job.
Or you can use an external stage with command something like
But the second will have issues if your job fails in the middle.
Or you can use an external stage with command something like
Code: Select all
awk '{print > $1}' yourFileName
-
- Participant
- Posts: 13
- Joined: Fri Jul 15, 2005 12:01 am
Re: pass column value as parameter
I think you can achieve this by splitting the job into two jobs,
In first job get distinct of Column1 values.
in second job use this values to crate file and filter the data on Column1.
in sequencer job call JOb1 then use a loop activity and call JOB2.
if distinct list of COLUMN1 in JOB1 is 1,2......n
then you need to run JOB2 that many time.. using loop activity.
JOB1-----Loop Activity------------Job2
In first job get distinct of Column1 values.
in second job use this values to crate file and filter the data on Column1.
in sequencer job call JOb1 then use a loop activity and call JOB2.
if distinct list of COLUMN1 in JOB1 is 1,2......n
then you need to run JOB2 that many time.. using loop activity.
JOB1-----Loop Activity------------Job2