Server Design To Parallel Design...Need verification...

[kaps]

I have a job in Server which I am converting to Parallel. Please let me know if this is the optimal way to do this. If not please advise me on the best way to do it.

Basically, reading from the sequential file and doing looup to a DB2 table two times(based on a condition) and then another DB2 table lookup and then split them to two files based on a condition.

Code: Select all

Server Job Design:

               DB2	 DB2
	       ||	  |
	       ||  	  |
	       ||  	  |
	       ||	  |
Seq File-----Xfmr1------xfmr2-----file1
                          |
			  |
			  |
		        file2

Parallel Job Design:

From Seq File1 To Filter Stage1 (To split two types of records)
From FilterStage1 To Join Stage1
From FilterStage1 To Join Stage2

From DB2API stage To Copy Stage (For Lookup)
From Copy Stage To Filter Stage2 (To split the records) To JoinStage1 To Funnel To Filter Stage4
From Copy Stage To Filter Stage3 (To split the records) To JoinStage2 To Funnel


Filter Stage4 To Xfmr To Seq File
Filter Stage4 To Seq File

[ray.wurlod]

If the reference data sets are small Lookup stages will be preferable to Join stages. If using Join stages you need to ensure that their inputs are correctly sorted. Have you given thought to the appropriate partitioning algorithm?

You might consider using a single Transformer stage (with constraint expressions) rather than a Filter stage and a Copy stage.

[vmcburney]

You may be forced to use the Lookup stages if you want to do a time based lookup - the Lookup stage lets you lookup on a key field and filter between two timestamps. May be necessary for time based dimension lookups.