When does DS issue RDBMS commits

[rleishman]

Apols for an easily tested question, but I don't have an instance of DS at the moment to test it on.

If you have a job with multiple pipelines, does an OCI stage perform its final commit immediately after the last row / after-SQL is processed, or does it wait until the end of the job.

Code: Select all

OCI ----> TX -----> OCI -----> TX ----> OCI
      A        B          C         D

Does it go: Process all txns through A & B, Commit link B, Process all txns through C and D, Commit link D

Or is it: Process all txns through A & B, Process all txns through C and D, Commit links B and D.

I suspect it is the first one, I just want to make sure.

[chulett]

It performs a 'final' commit when the stage closes. So, with the design you've shown, your first answer is correct.

[ray.wurlod]

I seriously doubt you can have a "passive" stage with an input and an output in a parallel job. This would be a blocking operation - which would interfere with pipeline parallelism. These are banned in the parallel job environment - the output link would be treated (and painted) as a reject-handling link.