active-to-active link problem

sendmk · Post by **sendmk** » Thu May 04, 2006 11:33 pm

i have two transformers directly connected in the job.

if i use a sequential file or inter-process stage between them the job runs.

instead if i enable in-process or inter-process rowbuffer it does not work.

what does it mean.

can anyone throw some light.

ray.wurlod · Post by **ray.wurlod** » Fri May 05, 2006 2:04 am

What exact release please? You ought to be able to apply buffering to an active-to-active link, but there were some issues in the early 7.x versions.

sendmk · Post by **sendmk** » Fri May 05, 2006 3:13 am

hi ray, it is 7.5.1.A

without using a sequential file or inter-process stage between them, the job just hangs

kumar_s · Post by **kumar_s** » Fri May 05, 2006 3:41 am

A sequential stage between two transformers?
Do you mean a reject link to the second transformer?
What exactly is your transformations in the transformer?

sendmk · Post by **sendmk** » Fri May 05, 2006 3:54 am

nothing to do with transformations we are just using couple of hash files along with transformer for look up.

Code: Select all

seq------transformer-------hash------transformer
                  |                     |
                  |                     |
               hash                  |
                  |                     |
odbc----transformer-----transformer-----seq

this is the structure.

kumar_s · Post by **kumar_s** » Fri May 05, 2006 4:08 am

sendmk wrote:nothing to do with transformations we are just using couple of hash files along with transformer for look up.
Code: Select all
seq------transformer-------hash------transformer
                  |                     |
                  |                     |
               hash                  |
                  |                     |
odbc----transformer-----transformer-----seq

this is the structure.

So which part as the issue, is it the two transformer coupled next to next?

sendmk · Post by **sendmk** » Fri May 05, 2006 4:18 am

yes

i have two transformers directly connected in the job.

if i use a sequential file or inter-process stage between them the job runs.

instead if i enable in-process or inter-process rowbuffer it does not work.

balajisr · Post by **balajisr** » Fri May 05, 2006 5:13 am

Code: Select all

seq1------transformer2-------hash2------transformer4
                  |             |
                  |             |
               hash1         |
                  |             |
odbc----transformer1-----transformer3-----seq2

this is the structure.

I am posting this for sendmk.

This is the actual job design. There was a problem in drawing the job structure in the previous post and did not come out correctly. Sorry.

Hash1 acts as a reference link to transformer2 and hash2 acts as a reference link to the transformer3. (Could not draw it accurately in the diagram)

When we run the job it runs forever.

When i remove only hash2 from the job it runs correctly. If i remove only hash1 from the job it runs correctly.

Is it due to deadlock between the processes of the job?

chulett · Post by **chulett** » Fri May 05, 2006 6:37 am

balajisr wrote:(Could not draw it accurately in the diagram)

Then post a screen print of your job so it is accurate.

There are many places offering free image hosting, upload a .bmp or .jpeg there and use the Img tags to include it in your post. Then maybe we'll be able to understand the structure and flow of your job.

sendmk · Post by **sendmk** » Fri May 05, 2006 10:07 am

this is all the information regarding the job

Code: Select all

                 seq2
                   |
                   |
seq1------transformer2-------hash2------transformer4
                  |             |
                  |             |
                 hash1          |
                  |             |
odbc----transformer1-----transformer3-----seq2
                  |
                  |
                seq3

Hash1 acts as a reference link to transformer2 and hash2 acts as a reference link to the transformer3.

this is the structure.

ArndW · Post by **ArndW** » Fri May 05, 2006 10:13 am

(There is no need to quote whole posts!!!)

The problem is that you have a circular dependancy in your job.

DataStage is looking at your job and a transformer2 is going to wait for the portion of the job that writes the Hash1 to complete before continuing. This is fine until at transformer3 that process is going to wait for the one starting at seq1 to complete. This is a deadlock situation.

chulett · Post by **chulett** » Fri May 05, 2006 10:19 am

Was curious to see the actual job canvas, as circular dependancies are normally caught by the compiler.

ArndW · Post by **ArndW** » Fri May 05, 2006 10:48 am

I thought so as well, but haven't had time to test this circular one to see if the compiler misses it. But it seems to be a logical explanation of the behaviour.

sendmk · Post by **sendmk** » Sun May 07, 2006 12:11 am

this is all the information regarding the job

Code: Select all

                 seq2
                   |
                   |
seq1------transformer2-------hash2------transformer4------odbc2
                  |             |
                  |             |
                 hash1          |
                  |             |
odbc1----transformer1-----transformer3-----seq2
                  |
                  |
                seq3

Hash1 acts as a reference link to transformer2 and hash2 acts as a reference link to the transformer3.

this is the full structure, to satisfy ur curiosity.

ray.wurlod · Post by **ray.wurlod** » Sun May 07, 2006 2:06 am

Actually, it's not complete, as we can't tell which way data flow with respect to the Hashed File stage hash1.

The circularity to which Arnd refers is in the Hashed File stage hash2 - it can not open its outputs until its input is closed.

Is there any reason this can't be done as two jobs - one to populate "hash2" and the other to use it?