Remove Duplicates Stage

[vick]

I m trying to remove duplicates while loading into an Oracle table.

I get this error when I use the remove duplicates stage ---- " Fatal Error: Tsort merger aborting: Scratch space full"

I checked the directory but there is space available. I have 12000 records to be loaded into the table.

Code: Select all

SeqStage---------->Remove_Duplicate_Stage----->Tfm----->OracleTable

I tried using the Sort Stage as well it gives me the same error.


Pl suggest

TIA

[Raghava]

I m trying to remove duplicates while loading into an Oracle table.

I get this error when I use the remove duplicates stage ---- " Fatal Error: Tsort merger aborting: Scratch space full"

I checked the directory but there is space available. I have 12000 records to be loaded into the table.

Code: Select all

SeqStage---------->Remove_Duplicate_Stage----->Tfm----->OracleTable

I tried using the Sort Stage as well it gives me the same error.


Pl suggest

TIA

Hi ,
The Problem is With Scratch space ,MAy i know abt ur configration settings
SCratch Disk and REsource scratch Disk Space.????
What is the size of Data u are loading to the Target?

Regards
Rag

[DSguru2B]

Its the /tmp space that the tsort uses to store intermediate temp files. Send in sorted data and uncheck the option of "Perform Sort".

[vick]

I m trying to load 12000 records.

tmp dir

Code: Select all

df -k /tmp
Filesystem    1024-blocks      Free %Used    Iused %Iused Mounted on
/dev/hd3           524288    507000    4%      689     1% /tmp

Scratch Dir
Filesystem    1024-blocks      Free %Used    Iused %Iused Mounted on
/dev/fslv03        131072    130656    1%   4     1% /opt/IBM/IIS/scratch0
/dev/fslv04        131072    130724    1%   4     1% /opt/IBM/IIS/scratch1

I unchecked the PERFORM SORT option. I m using "AUTO" partition option in sort stage.

On the keys I specified to for Sorting if I use the "Sort" option I get
Fatal Error: Tsort merger aborting: Scratch space full
but if I use the "Dont sort Previously Sorted" option then about 2000 rows passes thru Sort stage and 1980 is inserted into Oracle Table. In this case the job still says "Aborted".
[/b]

[DSguru2B]

Run your job then simultaneously keep running df -k /tmp to keep monitoring when it runs out. I know unix sort uses this directory by default. I am not sure what tsort uses.

[vick]

It did'nt help.

If I use the "Dont sort Previously Sorted" option then about 2000 rows out of 12000 passes thru.

Are there other options to remove duplicates?

[DSguru2B]

Yes they are. You can do it at the unix level using uniq. But get your code fixed. Its not a woo-haa error that cannot be fixed. Do an exact search on "Tsort merger aborting" for more information.

[vick]

Thanks DSGuru2B!!!

The network team set more space and it worked. It had space to read only 5900 records and I had 12000 records.

Sorry if I had a dumb post.

Thanks once again for all responses.

[DSguru2B]

Nothings dumb. We learn through our mistakes.