Hey All,
Hopefully an easy question. We have 2 sequential files (same structure) loading to one table. We were using link collector stage but our job has failed a few times after running for a period of time. The files have different amounts of data (1st has 50 million rows and 2nd has 70 million). Would this cause a timeout error using the link collector stage, round-robin algorithm? Is the simplest solution to break it up into 2 jobs and process each separately? Thanks in advance!
Data Set for Link collector links need to be the same size?
Moderators: chulett, rschirm, roy
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
The Link Collector stage is notorious for this behaviour. It takes Round Robin to mean "wait", rather than "skip if not ready". It does not process the "end of data" token gracefully.
You could cat the files together in a Filter command then, within the job if you want to, use Link Partitioner and Link Collector stages in concert to cause parallel processing. But try it without these stages first - I think that the speed of the Sequential File stage will be more than adequate.
You could cat the files together in a Filter command then, within the job if you want to, use Link Partitioner and Link Collector stages in concert to cause parallel processing. But try it without these stages first - I think that the speed of the Sequential File stage will be more than adequate.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.