Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.
Moderators: chulett, rschirm, roy
-
poorna_76
- Charter Member
- Posts: 190
- Joined: Thu Jul 08, 2004 10:42 am
Post
by poorna_76 »
Hi All,
We are trying to store large volumes of data (> 9 Million records) in either a DataSet or in a Lookup File Set for doing a lookup.
Is there any Size limitations with DataSets & Lookup File Sets?
Are there any known issues appending data to DataSets?
Thanks in Advance.
-
ray.wurlod
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
-
Contact:
Post
by ray.wurlod »
20TB x #partitions (approximately).
You're much more likely to run out of disk space than to hit this limit.
9 million records poses no problem at all, even if they're very wide records.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
-
poorna_76
- Charter Member
- Posts: 190
- Joined: Thu Jul 08, 2004 10:42 am
Post
by poorna_76 »
ray.wurlod wrote:20TB x #partitions (approximately).
You're much more likely to run out of disk space than to hit this limit.
9 million records poses no problem at all, even if they're very wide records.
Thanks Ray.
-
poorna_76
- Charter Member
- Posts: 190
- Joined: Thu Jul 08, 2004 10:42 am
Post
by poorna_76 »
ray.wurlod wrote:20TB x #partitions (approximately).
You're much more likely to run out of disk space than to hit this limit.
9 million records poses no problem at all, even if they're very wide records.
Is it 20TB or 20GB?
20TB seems to be very huge.
Thanks in Advance.
-
DSguru2B
- Charter Member
- Posts: 6854
- Joined: Wed Feb 09, 2005 3:44 pm
- Location: Houston, TX
Post
by DSguru2B »
Yup, in other words, just load 'em datasets without worrying about size limit. Your disk might burn out
Creativity is allowing yourself to make mistakes. Art is knowing which ones to keep.
-
ray.wurlod
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
-
Contact:
Post
by ray.wurlod »
20 Terabytes. Per partition. (I think there's a limit of 1 million partitions, too, but not totally certain on that one.)
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.