Hi,
Is there anyway to modify the record schema, which was created by datasets? For eg. When i created a job, some of the columns are defined as not null columns but few columns had null values so those columns were created null allowed columns in the record schema. And i need to overwrite the record schema definition because those columns should not allow null values.
Pls anyone help me how to change the record schema for the dataset.
Thanks
Saravanan
Record schema of DataSets
Moderators: chulett, rschirm, roy
Record schema of DataSets
In OSH or thru the PX GUI? If the record schema has been saved to a file, you can open it in DS Manager and manipulate it there.
- BP
- BP
-
ray.wurlod
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Although I can't answer the question I'd like to pose one. Why do you want to change the definition of something that exists, potentially introducing a mismatch between the thing and its description? 
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Hi Ray,
Thanks for your response.
Let me explain the scenario to change the record schema of a dataset. For eg. there are 3 key columns in the input so those columns are defined as not nullable column but when i ran the job the input has null value for one of the key columns so the record schema was changed by a job to allow null value for that key column and subsequently the input data was fixed that key columns have the proper value and that situation i need to change the record schema for the existing data set but i have no idea how to change it and i do not want to create a new dataset.
Hope this will clarify your question.
Regards
Saravanan
Thanks for your response.
Let me explain the scenario to change the record schema of a dataset. For eg. there are 3 key columns in the input so those columns are defined as not nullable column but when i ran the job the input has null value for one of the key columns so the record schema was changed by a job to allow null value for that key column and subsequently the input data was fixed that key columns have the proper value and that situation i need to change the record schema for the existing data set but i have no idea how to change it and i do not want to create a new dataset.
Hope this will clarify your question.
Regards
Saravanan
Okay, let me rephrase what you said to ensure I understand it:
1. You have an input stream (sequential file?) that apparently provided null values for one of the three intended key fields. This is being given to a dataset.
However, you are also saying...
2. You do not want to create a new dataset.
Therefore, from what I understand, you are populating an existing dataset with data that all of suddenly would have null values for non-nullable columns.
I would assume that you have considered and turned down the possibility of providing a default value in place of this null value. You also have considered and turned down the possibility that the data provided is wrong.
Now I am going to bring up a philosophy beaten into my head by the great Ken Bland (hey, if he worship Kimball, let me worship him!
) - Datasets are temporary. If you are carrying over data from week to week, you run the risk of adding more time to your processing in case of failure/corruption of that dataset -- extra hours you may not be ready to afford during a production run. Holding data within a dataset also prevent it from being clearly audited by third party tools for data validity. As wonderful as DataStage is, it is best to have an independent tool do the CYA audit.
I do hope your DataStage administrator is also aware of this intent in using datasets as a permanent storage -- the more data used by a dataset, the less space there is for the scratch space -- especially if both are shared on the same mount points. Your administrator could be running scripts that automatically clean up old files. If your program does not use a file within the time provided, your precious datasets could disappear.
My recommendation is to rebuild the datasets. It is far easier, and safer in the end to do so. After what I said, if you REALLY want to tweak it, just say so, and we will give you the red button to push.
-T.J.
1. You have an input stream (sequential file?) that apparently provided null values for one of the three intended key fields. This is being given to a dataset.
However, you are also saying...
2. You do not want to create a new dataset.
Therefore, from what I understand, you are populating an existing dataset with data that all of suddenly would have null values for non-nullable columns.
I would assume that you have considered and turned down the possibility of providing a default value in place of this null value. You also have considered and turned down the possibility that the data provided is wrong.
Now I am going to bring up a philosophy beaten into my head by the great Ken Bland (hey, if he worship Kimball, let me worship him!
) - Datasets are temporary. If you are carrying over data from week to week, you run the risk of adding more time to your processing in case of failure/corruption of that dataset -- extra hours you may not be ready to afford during a production run. Holding data within a dataset also prevent it from being clearly audited by third party tools for data validity. As wonderful as DataStage is, it is best to have an independent tool do the CYA audit.I do hope your DataStage administrator is also aware of this intent in using datasets as a permanent storage -- the more data used by a dataset, the less space there is for the scratch space -- especially if both are shared on the same mount points. Your administrator could be running scripts that automatically clean up old files. If your program does not use a file within the time provided, your precious datasets could disappear.
My recommendation is to rebuild the datasets. It is far easier, and safer in the end to do so. After what I said, if you REALLY want to tweak it, just say so, and we will give you the red button to push.
-T.J.
Developer of DataStage Parallel Engine (Orchestrate).
-
ray.wurlod
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
The best way to learn this stuff is to play with it.
Why not contact your local Ascential office, tell them your desires, and enrol on the DataStage Essentials (PX) class? They may even be nice enough to allow you extra "play time" with the product!
Why not contact your local Ascential office, tell them your desires, and enrol on the DataStage Essentials (PX) class? They may even be nice enough to allow you extra "play time" with the product!
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.