DSXchange

For some business requirements I need to JOIN 2 big files. The left link is about 30,000,000 records and the right link is 10,000,000 records. The performance is not good enough even though I split the input file into files according to 1 primary key column.(e.g. Key1 records go to File1, File2, the...

i could see "log report message after each job run"

maybe tick this checkbox will generate the report.

if so, I have another question. how could I output the logs from director to a flat file? So it could be review after long time.

STATUS REPORT FOR JOB: JOB1111111 Generated: 2016-01-22 10:05:21 Job start time=2016-01-22 10:05:11 Job end time=2016-01-22 10:05:21 Job elapsed time=00:00:10 Job status=3 (Aborted) Stage: stage11111111111111, 0 rows input ..... ..... Dear friends, I would like to ask how could I generate this "...

at last the job complete successfully.

It takes more than 2 hr to run the job, normally only 10 min is required.

Hi all, I am having a weird problem. when I am running a job, it "stops" in the middle, the job is running, but looks like it is hung. This is the log from the previous run (which completed successfully), the log is normal. sctrHMIPCOProdC36.GCC_Prod_Filter,0: Import complete; 1170290 reco...

Got it.

Thanks

Hi

I cannot find this "Unstructured Data stage" in my DataStage client. What's the problem?

Hi, As I know, when we use join stage to join 2 input links, even though we don't set the 'perform sort' in the input link, DS will help to sort. (Is this correct?) my question is, can we decide to use which function to sort? e.g. use the memory or hard disk to sort. The default option of Join stage...

Is it a good habit to use RCP when doing project?

it will save a lot of effort to maintenance the columns.

Or will you use RCP to develop a new project?

I think.
fixed-length rows != fixed-width format file

haha, I didn't notice this point actually when i read the doc.

And yes I used it in delimit file and the performance is better.

Read from multiple nodes Number Of readers per node Some days ago I just learnt we can set "Number Of readers per node" to improve the performance to read a cma file, that means there are some readers reading in the same time in one node. And then I realize this one "Read from multipl...

wierd!

After import the dsx of the routine again, the problem dispear!

Dear, I develop a routine to call a java program ================routine============= Cmd='sh /xxxx/abm/common/script/1.sh' Call DSExecute("UNIX", Cmd, Output, ExitCode) Ans=Output ================routine============= ================script============== #!/usr/bin/ksh cd /xxxx/abm/java /u...

Hi, I meet a problem that the sequential job process too long time. there are many Parallel jobs to run in order, as the input files' size are very big, that cause the temporary files so big that the job require long time to read/write. In such condition, what should I do to tune the job so that the...