DSXchange

If time is not an issue and then you could simply use a server job and hash file. 1. Read data 2. Lookup up key in hash file and pull back surrogate key, if there. Obviously will be empty to start 3. If not in lookup assign key and write Key field and surrogate key to hash file. 4. Output data. Just...

I ran the test again but with only 2 cols (4k) and the load is much quicker, although I still ended up with 2 2Gb files.

The file I need to load has record length of 44k (varchar).

Does a dataset treat a varchar as a char, because in most cases my actual record lengths are no where near 44k.

I'm loading 4 million rows from oracle into a dataset. I find that the size of the dataset files tends to be 10s of Gigs but when I load the same data into a loojup fileset of simple a fileset its Megs. The load to the dataset aborts due to a space issue, but I estimate that it would require 80gb fo...

Take you a look at the transaction handling tab. Set the rows per transaction to 0 to do a single commit.

A couple of things spring to mind. 1. Add the @INROWNUM value to each record as it passes through the transformer and then sort on this in the sequential file. I suspect you'll need to execute the stage sequentially though. 2. Is the data in the input file sorted in anyway. If so you could just re-a...

Sounds like you got yourself a couple of useful scripts there.

The only thing I would be wary of when letting the jobs create their own directories is ensuring that the job has not mis-spelt the directory names, but easy to bug fix.

If you're not concerned about processing all the data or the dataset is relatively small, this could be achieved by using an after job routine to check the link count on the reject row.

Can you specify the calls you're making in this post together with any arguments.

Data issue in the end. Thanks guys.

you may need to elaborate a little !!

I'm using a lookup fileset with about 10 columns and a single key. When the job runs the lookups appear to be successful with the key column coming through, but all remaining columns are mapped as null.

Any ideas?

Is this related to

viewtopic.php?p=299560#299560

If so you should try and avoid multiple threads on the same problem

Could you clarify whether the source is a db or sequential file

You should be able to achieve the same in a PX job by using a basic transformer.

If you don't see a basic transformer you'll need to add it to your palette using the customise option

As the guys have said the patch is called dsclient_xpsp2. One thing to note is that after installation of this patch (and even prior), the 7.x client is a little unstable and frequently closes down itself when you work within Job Sequences.

Just remeber to save frequently