Parallelism in Datastge
Moderators: chulett, rschirm, roy
Parallelism in Datastge
This question is about architecture of datastage. If i am correct, Datastge creates that much no of partition as specified in configuration file. So if i have with one node configuration file , can datastge achieve partition parallelism? Because I found that every partition algorithim partitions the input data as per no of nodes specified in configuration file.
Goutam Sahoo
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
... unless, of course, you design in partition parallelism, in a similar fashion as you would in server jobs. But please don't go down this path! Stop thinking like a server job designer and wrap your mind around how the Orchestrate engine works - it will look after data partitioning for you.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Ray is correct. You want to let the framework do the work for you...and technically speaking, as it relates to parallelism, you get both. Operators (stages for the purpose of this discussion) run in their own process [not always, and this can be tweaked, but lets keep it simple]...so even with a single node config, you will have several processes running, performing pipeline partitioning. Once you start adding nodes, you will get partitioning also... of course, here is where you make decisions on what "kind" of partitioning (hash on a value, round-robin, etc.), but the "degree" of partitioning is dictated by the number of nodes (and of course, again, there are vast ways to tweak this). The key is that the Job you see in the Designer remains largely the same, whether you have 1 or 4 or 8 "nodes." It's too complex a subject for a paragraph here and there, but this is the general idea.
Ernie
Ernie
Ernie Ostic
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>