two streams into one??

Luk · Post by **Luk** » Thu Mar 24, 2005 2:09 am

Hi!!

I have a question. How in DataStage server 7.5 can I make one data stream from few others streams??

For example: in one stream I have 100 rows and in another one 100 rows. Both streams have the same columns but different data. Both streams processing is parralel.
How can I make one stream with the same columns with (in this example) 200 rows??

Regards

ArndW · Post by **ArndW** » Thu Mar 24, 2005 3:29 am

Luk,

this is what the MERGE stage does, it will merge your two data streams into one.

garthmac · Post by **garthmac** » Thu Mar 24, 2005 3:30 am

Hi Luk, you can use the link collector stage. This allows up to 64 inputs, and then outputs as one single stream of data.

Luk · Post by **Luk** » Thu Mar 24, 2005 3:52 am

this is what the MERGE stage does, it will merge your two data streams into one.

MERGE is making this with files NOT streams
am I wrong??

Luk · Post by **Luk** » Thu Mar 24, 2005 3:54 am

you can use the link collector stage. This allows up to 64 inputs, and then outputs as one single stream of data.

Link collector is working fine. Thanks.
I don't know why I did't see that :D

Luk · Post by **Luk** » Thu Mar 24, 2005 3:58 am

one more quick question!
what does derivation field in collector?? if you could give me some exemple it would be perfect

garthmac · Post by **garthmac** » Thu Mar 24, 2005 4:02 am

Hi Luk, It's possible to aggregate data in the derivation part of the output of the link collector.

roy · Post by **roy** » Thu Mar 24, 2005 4:14 am

Hi,
How about using a filter command that concatenates both files and gives you a single output link? (should be good if the files are not big, better then merge in any case).

using link collector will probably pose issues in this case.

IHTH,

Luk · Post by **Luk** » Thu Mar 24, 2005 4:18 am

Hi again!

I don't want merging two FILES but two data streams!!! Link collector does not support active inputs (for example from transformer stage)!!!
It works only when I'm merging data from files or database!

Is it posible to merge two streams without using files as buffers ??

garthmac · Post by **garthmac** » Thu Mar 24, 2005 4:48 am

It is actually possible to merge two streams without using files as buffers. I always use files to source and collect my data, but a quick read of the documentation tells me you can use inter process row buffering. You can find this option in the performance tab of the properties page to enable active to active links. Try this, and let us know how you get on.

Luk · Post by **Luk** » Thu Mar 24, 2005 5:07 am

thanks garthmac !!
It sounds reasonably! I try that and let you know.

chulett · Post by **chulett** » Thu Mar 24, 2005 8:01 am

ArndW wrote:Luk,

this is what the MERGE stage does, it will merge your two data streams into one.

Just for the record, this is a PX answer. The confusion comes from having two stages, performing two different (yet somewhat similiar) functions, with the same name - MERGE.

The Merge in PX is similar to the Link Collector in Server, while the Merge in Server actually 'merges' two flat files by treating them like tables (behind the scenes) and allowing them to be 'joined' to create a single input source.

garthmac · Post by **garthmac** » Thu Mar 24, 2005 8:14 am

Craig,

I have noticed that many people have experienced problems with the link collector stage, when enabling inter process row buffering (ds_ipcopen() - Error in open). Have you experienced issues with this method? My workaround was to not use any row buffering, instead I write data to multiple seq files as source, and output data to a seq file.

Luk, I wouldn't be surprised if you encounter this too!