DSXchange

Hi,

I am using standardize and match frequency stages.

i can able to run standardize job successfull same as below

input1=> standardize1 => stanout1
input2=> standardize2 => stanout2

then i have used stand outs for match frequncy same as below

stanout1 => match_frequency4 => matchfreq1
stanout2 => match_frequency5 => matchfreq2

if it is for less records like input1 = 10000, input2 = 60000 - both jobs are working fine and data was also good.

if it is more data like input1 = 45000, input2 = 1389749 - standarze job is working fine, but iam getting an error in matchfrequency job

please find the below error messages and advise me what to do -

APT_CombinedOperatorController(1),0: Fatal Error: Tsort merger aborting: Scratch space full [sort/merger.C:876]
APT_CombinedOperatorController(1),0: Fatal Error: Pipe read failed: short read [sort/merger.C:1734]
buffer(1),0: Fatal Error: waitForWriteSignal(): Premature EOF on node dedwbt01 No such file or directory [iomgr/iocomm.C:1636]
main_program: Step execution finished with status = FAILED. [sc/sc_api.C:242]


it is soo important to me to use moredata, please reply back to me with your valuable suggetions.

Thanks,

Disable operator combination so you can discover which stage is throwing the error. You are running out of scratch space, and there is at least one short record in your input data. This can be discerned directly from the error messages.

ray.wurlod wrote:Disable operator combination so you can discover which stage is throwing the error. You are running out of scratch space, and there is at least one short record in your input data. This can be disce ...

--

How to clean this scratch space, what does mean a short recor in input data - what to do with that ??

Rupesh

Hi rupesh,

you dont have to free scratch space..

all the temp files neded the jobs are created in scratch space and then the processed data are written to dataset folders in dataset files.then the files in scratch folder are cleared by the sysem once the job is done.

So you need to increase the size of your scratch folder and dataset folder.

suggestion: after your standardization job drop the unwanted column like unhandled pattern, etc which is not used by the matching.

you can reduce the size some how...

regards
vairamuthu