Output link opens, even if the job aborts

[DS_SUPPORT]

I am having an issue, even if the job aborts, the output link from a Hashed File Stage opens, and processing certain (100) rows. I dont want to process, if the Job aborts, because i am doing some business caluclations in that link.

My job design , looks like

Code: Select all

DRS 1-------->IPC----------> TRANSFORMER1 ------> Seperate Process (No Issues in this path)
                               |
                               |
                               |
                               |
           Hashed File2        |I
                ^              |N
               I|              |P
               N|              |U
               P|              |T
               U|              |
     Output    T|     OUTPUT   |
DRS2<-------Transformer2<------V
                |         Hashed File Stage1
                |
               I| 
               N| 
               P| 
               U| 
               T|
                V
           Hashed File3

In the main stream , i am reading the rows from DRS, and processing. In Transformer1 , if some constraint is true, i will store the values to a Hashed File1. And a output link from that hashed file is there, which inturn updates two more preprocessed hashed files (2 and 3) and send the output to backend (DRS2).

The output from the Hashed File should fire, only if all the rows are processed, but in my testing process, i aborted the job in the transformer1 by using utilityabort, Still the output link opens from the hashed file stage1 and processed 100 rows, I confirmed by checking the outputs in Hash file 2 and 3, but it didnt do any changes in the backend (DRS2).

I dont want this to happen, because the other two hashed files will be used in other jobs, with the wrong results.I am expecting the output link should open only if the rows are processed sucessfully.

Is there any way, to stop this happening, if the job aborts, the output link should not open. Is there any settings , I need to do for acheving this?

[chulett]

I am having an issue, even if the job aborts, the output link from a Hashed File Stage opens, and processing certain (100) rows. I dont want to process, if the Job aborts, because i am doing some business caluclations in that link.

The output from the Hashed File should fire, only if all the rows are processed

Your expectations need to be adjusted, as this is what we call The Way It Works. All passive stages in each 'segment' are opened when the segment starts and individual records flow from beginning to end as each are processed.

If you'd like to write to the hashed file only after the job processes all records correctly / completely then you need to introduce a process break. In this case I'd suggest a Sequential File stage. Write to the sequential file rather than the hashed file and then read from the sequential file to populate Hashed File Stage1.

[DS_SUPPORT]

I need to run this job parallely (Allow Multiple Instance). If i use sequential file, i may get error , because sequential file cannot be accessed by multiple processes.

And the other thing is, i have to update the flags in the Hashed file, based upon the keys. In Sequential file, i cannot achieve this.

Is there any other way to achive this?

[chulett]

Neither of these concerns are problems. Include the InvocationID in the name of the sequential file, just like you've probably done with the hashed files. And you write to the sequential file exactly the same information that you want to write to the hashed file. The end result will be the same regardless of the source.

[DS_SUPPORT]

But , if i introduce Sequential file, my overall job finishing time will increase, because this job will be processing millions of rows for every run. That is the concern.

[chulett]

No, it won't, not to speak of. Never mind that something of this nature is the only way to solve your "issue" and a sequential file 'staging table' is cheap and fast... unless you've got a crap disk subsystem, I suppose.

Try it.

[DS_SUPPORT]

I have introduced a Sequential File between Transformer1 and Hashed File Stage 1 , but the output is same.

Still the output link is firing. It is firing from the sequential file stage itself.

[chulett]

Of course it is "still firing". As I said, that is How It Works and nothing will change that fact. Adding the process break means the output is NOT going into your hashed file which you seemed to want to protect, they go into your flat file first and then only get written to the hashed file if that part of the job completes successfully. All or nothing.

Isn't that exactly what you wanted? If it isn't then we'll need to start over.

[DS_SUPPORT]

No, Still the Hashed File gets updated, after using the sequential file also.

This workaround works for me,

I have written a routine, which takes jobname as input and gives the status of that job as output. In the Transformer2 , i called that routine in a stage variable and getting the status of the job , for the 1st row alone.

And used a constraint , to load the hashed file , if the status is 0 only.


Actually , First i have implemented this logic , in the transformer2 itself by using DSGetJobStatus(DSJ.ME, DSJ.Jobstatus), But by this method, i get the status as 0, and contibue loading the hashed file.

But after writing this logic , to a routine. I am getting desired output.


Still waiting, for the inputs, for any other methods?

[ray.wurlod]

What a surprise! When you ask a running job for its status, it returns "Running" (which is what 0 means). You need to re-think your logic, probably to split it into two separate pieces, the first of which does any processing, the second of which only executes if the first does not abort.

The only alternative to that is something like the Tardis stage. Because what you are "requiring" is the ability to go back in time to a point before the first row was processed, after some condition occurs in a subsequent row that causes the job to abort.

Caveat (so I don't get sued/pursued) - there is no such thing as a Tardis stage.

[DS_SUPPORT]

Thanks for all the inputs. Yes, Finally decided to split the job.

[chulett]

No, Still the Hashed File gets updated, after using the sequential file also.

You seem to be missing the point here, that or your description of the job design doesn't match what you are actually building. What exactly are you saying here? Of course the hashed file gets updated, but only if the job does not abort. If the job aborts while writing to the sequential file, then no changes are made to the hashed file. Assuming you've done what I think you've done, and assuming that is actually what you are trying to accomplish.

I would be interested in seeing your actual job design, a screen capture of your canvas which you can upload to any of the free image hosting sites out in the wild then link into your post. Or you can stick with your 'workaround'. [shrug]