Pivot stage upstream of Column Import stage

rajeshknl · Post by **rajeshknl** » Wed Jul 23, 2008 3:44 pm

Hi,
I have to read a delimited file(,) with varying number of columns. Also turn the columns into multiple rows.

EX: Input rec_time,Acc_time,Para_ID,Tower_ID
Ouptput rec_time
Acc_time
Para_ID
Tower_ID

I can use a schema file as the no of columns vary but how do i connect the pivot stage without knowing the no/names of columns.

Is there any alternative solution. Any help is appreciated.

ray.wurlod · Post by **ray.wurlod** » Wed Jul 23, 2008 5:14 pm

Welcome aboard.

To use a Pivot stage you must name at least the pivot key column and the pivoting columns. Therefore a "pure" RCP solution is not viable.

You need to pre-parse your data. That is, you can not rely on a schema file in this case.

rajeshknl · Post by **rajeshknl** » Wed Jul 23, 2008 5:47 pm

Thanks

Is there a way to count the no of times the delimiter repeats and pass that as a parameter. i am just guessing wildly.

ray.wurlod · Post by **ray.wurlod** » Wed Jul 23, 2008 7:43 pm

Please explain that requirement more carefully. Parameters can only be passed before the job runs, so whatever it is that you are proposing would require some kind of pre-processing of the source data.