Hi,
I wanted to know if a dataset is created in Job1 with 20 columns.
In Job2 we read from this dataset and the metadata is defined only for 10 columns, the remaining 10 columns are never defined in the dataset.
The job executes successfully, but I wanted to know will the data integrity be maintained.Currently data looks fine, but I'm not sure of large data loads
Anyone faced similar issues?
What if I have a Job3 and I define only 5 columns out of 20 in metadata?
Regards
Ankur
Metadata and Dataset
Moderators: chulett, rschirm, roy
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Is Runtime Column Propagation enabled? What (precisely) do you mean by "data integrity" here?
In no stage type (except Sequential File stage*) do you need to read all columns from source.
* Even in Sequential File stage there is a column property "drop on import" available. But you still have to read every byte in the file to get to the next.
In no stage type (except Sequential File stage*) do you need to read all columns from source.
* Even in Sequential File stage there is a column property "drop on import" available. But you still have to read every byte in the file to get to the next.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
-
- Participant
- Posts: 53
- Joined: Fri Apr 18, 2008 8:13 am
- Location: United States
Metadata and Dataset
Hi,
As Nagarjuna rightly said, unless you make any changes (Metadata Change) to your dataset in the first job, you can always go ahead and use it for the specified number of columns you want in the next successive jobs.
As Nagarjuna rightly said, unless you make any changes (Metadata Change) to your dataset in the first job, you can always go ahead and use it for the specified number of columns you want in the next successive jobs.
Regards
Ajay
Ajay
-
- Premium Member
- Posts: 278
- Joined: Wed Oct 03, 2007 8:45 am