Might not be that common, but I'm just wondering if it is possible to have different parallel threads in same stage? E.g.,
This might be helpful during data preparation stage.
Thanks
Parallel threads in same stage
Moderators: chulett, rschirm, roy
Parallel threads in same stage
Hi,
Might not be that common, but I'm just wondering if it is possible to have different parallel threads in same stage? E.g.,
This might be helpful during data preparation stage.
Thanks
Might not be that common, but I'm just wondering if it is possible to have different parallel threads in same stage? E.g.,
This might be helpful during data preparation stage.
Thanks
Yes, you can have multiple, distinctly separate job flows in a single canvas (as shown in your image), however there are considerations:
- You can't read / write to the same targets at the same time. For example in your image you show one flow writing to a sequential file and another flow reading from a sequential file. They had better be different files or it isn't going to work.
- You can't guarantee any "timing" between the distinct threads. In other words, don't count on one thread getting to a particular stage, or finishing, before the other thread.
- The number of processes active at one time will increase, which could cause resource issues, depending on the number of stages and threads are on the canvas.
- You can't easily run just one of the threads on the canvas. This doesn't matter for small workloads, but for very large workloads it can slow down a recovery effort (you only need to re-run one thread, but three are on the canvas).
- You can't read / write to the same targets at the same time. For example in your image you show one flow writing to a sequential file and another flow reading from a sequential file. They had better be different files or it isn't going to work.
- You can't guarantee any "timing" between the distinct threads. In other words, don't count on one thread getting to a particular stage, or finishing, before the other thread.
- The number of processes active at one time will increase, which could cause resource issues, depending on the number of stages and threads are on the canvas.
- You can't easily run just one of the threads on the canvas. This doesn't matter for small workloads, but for very large workloads it can slow down a recovery effort (you only need to re-run one thread, but three are on the canvas).