Writing 64 Target files

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
Gokul
Participant
Posts: 74
Joined: Wed Feb 23, 2005 10:58 pm
Location: Mumbai

Writing 64 Target files

Post by Gokul »

We have a requirement to write the data into 64 target files. The data is written into one of the 64 files based on value of a column. The target file is an ebcidic file.

Same is the design for 6 of the jobs. These leads to bottleneck with most of the execution time involved in write operations.

These hampers performance. Is there any environment variable which can be set to improve the write performance.
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

What O/S? Talk to your SysAdmins, have them monitor the I/O on the system. Perhaps you can move your target to faster drives or they can 'adjust' some filesystem journalling options, etc.
-craig

"You can never have too many knives" -- Logan Nine Fingers
Sainath.Srinivasan
Participant
Posts: 3337
Joined: Mon Jan 17, 2005 4:49 am
Location: United Kingdom

Post by Sainath.Srinivasan »

Why don't you write in a single file and then split based on some value / flag ?
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

Are all the files on the same file system? Do you have the flexibility to spread the files over multiple disks?
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Post Reply