DSXchange

Thank you guys, I'll look into it.

Hmmm. Did you put the path name /usr/bin in front of the 'cd'? cd is a shell builtin, not an executable file or script in /usr/bin. Check in /usr/bin, is there actually a file there called 'cd'? Try running the command you pasted above from the Unix command line and see what happens - it certainly w...

I have a job sequence that kicks off a large number of independent jobs. There are no links between the job activities, because there are no dependencies. What I want is some mechanism to limit the load placed on the system. Ideally I'd like functionality such as the Unix at/batch/cron command that ...

Pavan, Take a look in the sdk/KeyMgt folder of Server Routines, there are some routines there that can help with surrogate key allocation. As for SCD's, depends on whether you want Type 1,2, or 3. If you search this forum you'll find the subject is fairly well covered. In short, DataStage (Server) d...

It's a bit ugly, but...
You could put the SEQ stage inside a Shared Container and create a parameter to the stored container that you reference in the SEQ file name.

When you drag the container into your job, set the parameter to anything you like.

Try a filter command of:

(cd /path/path; xargs mkdir -p)

Try searching this forum for @USER (use "exact match"). There are four global variables that can be set in a routine and accessed elsewhere in the job. I'm unsure as to whether they can be used in a filename like a parameter.

My original post was incorrectly worded. Where I used Array Size I should have been referring to Transaction Size. Apologies for the confusion. I actually set both Array Size and Transaction Size to 255, although I see no reason why you couldn't set Array Size larger if you found performance gains. ...

How wide are these rows? That is slow. What about mb/sec? Sorry, I misled you earlier with the environment. The DS server (our dev box) is actually a virtual machine running Linux over vmware. It makes benchmarking a bit more thought provoking. The critical speed is not the transformation, but the ...

Craig,

By "Reject Event", I mean the type of log message in Director. My jobs that have rows going down the reject link raise a single 'Reject' message in the log, but no 'Warning' messages.

I don't use the "abort after" option, so cannot say whether this affects results.

Just benchmarked vmcburney and Ray's suggestion of writing to SEQ stage instead of OCI Load with Manual Load option. I re-ran all options and found that today is a bit faster than yesterday, but that's benchmarking for you: * OraBulk - 210 rows/sec * SEQ File - 220 rows/sec * OCI Load (Manual) - 210...

I was hoping for some spirited discussion, and I reckon I got it... Firstly aartlett: We havent found maintainability to be a problem. The splitting of inserts/updates is all templated, so there is no development overhead. The only issues I've found are: * If a table structure changes you have to re...

We have found with 7.5.1A just checking the Reject Row will only produce a Reject event I haven't experienced this problem with 7.5.1A. All of our jobs that process files to tables perform validation in a transformer and send failures down a link with the Reject condition checked and no constraint....

Nic,

Do me a favour and create an Annotation in your Job that describes what you are doing, and repeat it in the stage notes for the SEQ file stage. The solution is really a mis-use of the tool, so this might help the next guy who looks at it.

Great tips, thanks. These techniques are a bit distant from my current methodology to easily benchmark them, but I can make some pretty accurate guesses based on the figures I already have. Externally Organised Tables (EOTs) Using EOTs involves transforming to a flat file, and then using what is eff...