pass column value as parameter

[reachmexyz]

i have job like

dataset 1 -- > transformer ---> sequential file.
let say dataset 1 has data like
a,1,2
b,1,2
c,1,2
Now i wish to create three files named a,b,c and a will contain record a,1,2
b will contain b,1,2 and c will contain c,1,2.
Some way i have to pass col1 values as paramter to sequenntial file name.
Is this possible. Depending on the value of col1 in dataset, a new sequential file need to be created. If the file is existing, it will append or if file doesnt exist it should be created. Please reply.

[ray.wurlod]

Hard coding a link for every col1 value possibility is one way. The links that don't process any rows will still create files - you can clean these up with an after-job subroutine.

[reachmexyz]

Hard coding a link for every col1 value possibility is one way. The links that don't process any rows will still create files - you can clean these up with an after-job subroutine.

I really do not know all the possible values for col1. It can be 10,20 even 50. Thats why i wish to do it dynamically. So we do not have a way to pass the col1 values as a parameter to file name.

[chulett]

Server job. Folder stage. Well, except perhaps for the 'append' bit.

[vmcburney]

It looks like a funny way to share data - who creates dozens of tiny sequential files? What's the purpose? I think this can be done quite easily via XML files just by configuring the XML output to create a different file for each key field but it's not the type of thing you would want to do with sequential files due to the file maintenance overhead.

Maybe a Parallel job that reads the Dataset and uses a BASIC Transformer to call a Server Routine that writes data out to a file.

[chulett]

Yup, I've noted in other threads that (oddly enough) the XML Output stage can do this as well.

[arunkumarmm]

i have job like

dataset 1 -- > transformer ---> sequential file.
let say dataset 1 has data like
a,1,2
b,1,2
c,1,2
Now i wish to create three files named a,b,c and a will contain record a,1,2
b will contain b,1,2 and c will contain c,1,2.
Some way i have to pass col1 values as paramter to sequenntial file name.
Is this possible. Depending on the value of col1 in dataset, a new sequential file need to be created. If the file is existing, it will append or if file doesnt exist it should be created. Please reply.

You Can write all your column1 values to a file as comma delimited and create a sequence with loop and pass this value to you counter and to your file name parameter.

[Sainath.Srinivasan]

You can write the output file and then split them in an after-job.

Or you can use an external stage with command something like

Code: Select all

awk '{print > $1}' yourFileName

But the second will have issues if your job fails in the middle.

[satyanarayana]

I think you can achieve this by splitting the job into two jobs,

In first job get distinct of Column1 values.
in second job use this values to crate file and filter the data on Column1.
in sequencer job call JOb1 then use a loop activity and call JOB2.
if distinct list of COLUMN1 in JOB1 is 1,2......n
then you need to run JOB2 that many time.. using loop activity.


JOB1-----Loop Activity------------Job2