Need best partitioning method for hierarchy mgmt
Posted: Mon Jan 30, 2006 3:06 pm
Hi,
I have a data file with data like this
1,a
1-10,t
1-11,u
1-12,v
1-10-20,x
1-10-25,y
1-11-26,z
I need to convert this data as
1,a,1-10,t,1-10-20,x
1,a,1-10,t,1-10-25,y
1,a,1-11,u,1-11-26,z
1,a,1-12,v,,,
For this, I created a job like Sequential -> Sort -> Transformer -> Sequential.
In the Transformer stage, I used stage variables and storing the incoming data in different variables based on the # of occurances of hyphen(-) and writing only the final level data to output file with the stage variables.
My logic will work fine in a server job. But since it is a parallel job, I am not getting the desired output. If I change the partition method to 'Entire', then I am getting the proper output, but the results are duplicated due to the more # of nodes.
The other way we are thinking is using the data file as lookup as well and forming the hierarchy. It will work fine, but little complex.
Is there any way to get the result using the first method without changing the # of nodes?
Thanks in advance.
I have a data file with data like this
1,a
1-10,t
1-11,u
1-12,v
1-10-20,x
1-10-25,y
1-11-26,z
I need to convert this data as
1,a,1-10,t,1-10-20,x
1,a,1-10,t,1-10-25,y
1,a,1-11,u,1-11-26,z
1,a,1-12,v,,,
For this, I created a job like Sequential -> Sort -> Transformer -> Sequential.
In the Transformer stage, I used stage variables and storing the incoming data in different variables based on the # of occurances of hyphen(-) and writing only the final level data to output file with the stage variables.
My logic will work fine in a server job. But since it is a parallel job, I am not getting the desired output. If I change the partition method to 'Entire', then I am getting the proper output, but the results are duplicated due to the more # of nodes.
The other way we are thinking is using the data file as lookup as well and forming the hierarchy. It will work fine, but little complex.
Is there any way to get the result using the first method without changing the # of nodes?
Thanks in advance.