LinkCollector Issue

vinaymanchinila · Post by **vinaymanchinila** » Tue Jun 21, 2005 10:46 am

hi,
I am using a LinkCollector to read from 2 sources with identical structure and write it to a flat file which has output to a transformer. If it extracts 20 records from each link, the total of 40 records are loaded into the flat file but only 20 are returned to the end transformer!

Link1 (20 records)
} LinkCollector(40)--->FlatFile (40)-->Transformer(20)
Link2 (20 records)

Any way to work around this issue.

Thanks,

vinaymanchinila · Post by **vinaymanchinila** » Tue Jun 21, 2005 10:53 am

Or can I just use the "Union ALL" in just one ODBC stage instead of two extracts as my columns are the same but the join conditions are different for both the queries?

The join conditions are too many though, so would it effect the performance?.
Thanks,

Sainath.Srinivasan · Post by **Sainath.Srinivasan** » Tue Jun 21, 2005 12:58 pm

Did you check whether you have 20 records into the transformer or out of the transformer following some logic embedded in the transformer?

vinaymanchinila · Post by **vinaymanchinila** » Tue Jun 21, 2005 1:04 pm

I am writing intoa flat file from the transformer with no constraints at all. Also checked for the file names to be sure.
Thanks,

pnchowdary · Post by **pnchowdary** » Tue Jun 21, 2005 1:20 pm

Hi Vinay,

1) Try to do view data for the link between the flat file and the transformer. It is a possibility that you are not able to read all the records from the flat file, for some reason. If you dont have any constraints in the transformer, then you should see all the 40 records in its output.

2) Alternatively, you can put the following system variables (@INROWNUM and @OUTROWNUM) in the tranformer and observe what those values are. This might give you a clue about what's happening.

3) Try to put break points before and after the transformer and run the job in the debug mode and observe how many rows are being processed.

I hope this helps.

vinaymanchinila · Post by **vinaymanchinila** » Tue Jun 21, 2005 1:23 pm

Say If I use this file seperately in the same job or some new job it gives me the 40 records!
It just doesnt seem to populate the other side of the transformer if it also acts as a source file.

baabi_26 · Post by **baabi_26** » Tue Jun 21, 2005 7:53 pm

Vinay,

Can you delete the merge file (created by previous run) before running the job and let me know what message you get. I suspect transformer might be reading incomplete data (even before the file is loaded) which, ofcourse, should not happen. Just want to make sure that is not the case.

- Nav

talk2shaanc · Post by **talk2shaanc** » Tue Jun 21, 2005 10:28 pm

can i know whatz on the other side of the transformer ? I mean which passive stage you are using after the transformer.

If its any of the databases you have in datastage, then check the director if your 20 "missing" rows are not rejected.

Second thing, I would like to know(jus for my understanding); why do you need a sequential file in b/w link collector and transformer.
You can have ;

Code: Select all

link collector1
              |---------------transformer-----------next stage
link collector2

vinaymanchinila · Post by **vinaymanchinila** » Wed Jun 22, 2005 7:03 am

I am using a flat file the other side.
I need a flat file because the Link Collector does not support active stages.

vinaymanchinila · Post by **vinaymanchinila** » Wed Jun 22, 2005 7:05 am

I guess the transfomer is reading before the Link Collector stage is done. As of now I had to split it into 2 jobs one which extracts the data and when I read the file from other job it gives me the 40 records !

Sainath.Srinivasan · Post by **Sainath.Srinivasan** » Wed Jun 22, 2005 7:08 am

Did you check by directing the link from link collector into a seq file rather than passing via transformer?

Also you can rewrite the job where the first 'creates' and the second 'appends' to the file.

vinaymanchinila · Post by **vinaymanchinila** » Wed Jun 22, 2005 7:17 am

Sometimes the first extract starts early and sometimes the second so I cant exactly go by the order of which one creates and which one apends.
Will try to point the Lcolector the flat file and see.
Thanks,

talk2shaanc · Post by **talk2shaanc** » Thu Jun 23, 2005 1:33 am

vinaymanchinila wrote:I am using a flat file the other side.
I need a flat file because the Link Collector does not support active stages.

Still you dont need an intermediate flat file.

Just go to Job Properties --->Performance --->click on "Enable Row Buffer" ---> "Inter Process"

or you can also use Inter-process stage, in place of sequential file.

vinaymanchinila · Post by **vinaymanchinila** » Thu Jun 23, 2005 7:25 am

Thanks wil go ahead with it!

vinaymanchinila · Post by **vinaymanchinila** » Thu Jun 23, 2005 3:12 pm

Hey,
Used the IPC Stage and applied changes in the peroformance tab. Still I have no idea why it is still not giving me the whole output!