I am using Column Propagation because I have 19 output Sequential files, all defined exactly the same way. I would like to maintain a single schema file and use it with column propagation. Works great except DS cannot generate a reliable schema from a table definition. This is extremely aggravating.
Here is an example of two fields from the schema file that appear to me to be identically defined;
AD_Tool:nullable string[max=5] {null_field='',prefix=2};
Alloc_Unit_ID:nullable string[max=5] {null_field='',,prefix=2};
In fact, I have exported the table definition and manually updated the definition of the second field to actually BE identical to the first (except for the name).
Upon import, the result is as above. The second field - Alloc_Unit_ID has an extra comma that prevents the schema file from being usable.
I'm forced to build a sed script to remove the extra commas.
Anybody know why this is the case?
How I can generate a schema file from a Table Definition that works?
Schema File formats
Moderators: chulett, rschirm, roy
You can make schema files from tables in any database.
When you goto the option to Import the table definitions -> Orchestrate Schema file, there you have to option to connect any where.
Instead if you have the ER diagram get the ddl, generate those tables in Sandbox and import the metadata. There you are!
IHTH.
When you goto the option to Import the table definitions -> Orchestrate Schema file, there you have to option to connect any where.
Instead if you have the ER diagram get the ddl, generate those tables in Sandbox and import the metadata. There you are!
IHTH.
Creativity is allowing yourself to make mistakes. Art is knowing which ones to keep.
Thank you Mr. Guru. You might take a peek at my questions as they don't appear to correlate to your reply. I apologize if I didn't state my problem succintly.
I have a schema file, generated from a table definition, that is incorrect. DataStage is failing when attempting to use it with column propagation when writing to many sequential files. It is failing because there are extra commas in the schema file. I'd like to automate this process and eliminate the manual process. This means I would like to eliminate the shell script I have written to correct the schema file manually in order to make it usable by DataStage.
Has anyone experienced this problem?
Why are there extra commas and how can I fix it?
I have a schema file, generated from a table definition, that is incorrect. DataStage is failing when attempting to use it with column propagation when writing to many sequential files. It is failing because there are extra commas in the schema file. I'd like to automate this process and eliminate the manual process. This means I would like to eliminate the shell script I have written to correct the schema file manually in order to make it usable by DataStage.
Has anyone experienced this problem?
Why are there extra commas and how can I fix it?
I dont get this. If the table definition is incorrect, then its a problem with the way the table was created, not with the schema importer.trobinson wrote: I have a schema file, generated from a table definition, that is incorrect. DataStage is failing when attempting to use it with column propagation when writing to many sequential files.
I think the table you are getting your metadata from, is not created appropriately. Try using those table as target and switch off RCP.
Creativity is allowing yourself to make mistakes. Art is knowing which ones to keep.
Ah, the light bulb goes off. When I say Table Definition I was referring to a DataStage object called a Table Definition. These things do not need to be spawned from a dbms object, they can be native to DataStage. This particular one comes from the column tab from a Copy stage. It results from saving the columns of a link. It has nothing to do with any dbms ddl.
This is straight Datastage folks no smoke and no mirrors. I save the columns to a Table Definition and attempt to generate a schema file from that saved Table Definition. The purpose in doing so is to maintain the Table Definition in a single place and use it in many places via a schema file.
And now...the rest of the story;
I have a job that needs to write 19 different sequential files that ALL have the same format but contain different data. Think of buckets. I have 19 buckets. Using Switch and filter stages, I route data to the proper buckets. Let's say I then want to add a field to all the files. Do I add this 19 times? Heck no! I define the Sequential stages with the exact same schema file which I maintain as a Table Definition. Theoretically, this would enable me to add a field to this table definition, generate a schema file, save it to the OS and boom! the new field shows up in all 19 files because I am also using column propagation. Holy Master Metadata Batman! Gotta love it. If it worked, that is, which it don't. I have to constantly massage the dang schema file. Hence this post. What time is it in Australia?
This is straight Datastage folks no smoke and no mirrors. I save the columns to a Table Definition and attempt to generate a schema file from that saved Table Definition. The purpose in doing so is to maintain the Table Definition in a single place and use it in many places via a schema file.
And now...the rest of the story;
I have a job that needs to write 19 different sequential files that ALL have the same format but contain different data. Think of buckets. I have 19 buckets. Using Switch and filter stages, I route data to the proper buckets. Let's say I then want to add a field to all the files. Do I add this 19 times? Heck no! I define the Sequential stages with the exact same schema file which I maintain as a Table Definition. Theoretically, this would enable me to add a field to this table definition, generate a schema file, save it to the OS and boom! the new field shows up in all 19 files because I am also using column propagation. Holy Master Metadata Batman! Gotta love it. If it worked, that is, which it don't. I have to constantly massage the dang schema file. Hence this post. What time is it in Australia?
I first need to understand how is this occuring in order to see where the problem is.
As said by you, the schema file results from the columns tab in the copy stage, did you try to view what is being generated.
There can be two possiblities where the problem is occuring.
1. The schema file is not generated properly.
2. The importer is not operating as it should be.
As said by you, the schema file results from the columns tab in the copy stage, did you try to view what is being generated.
There can be two possiblities where the problem is occuring.
1. The schema file is not generated properly.
2. The importer is not operating as it should be.
Last edited by DSguru2B on Thu May 11, 2006 11:13 am, edited 1 time in total.
Creativity is allowing yourself to make mistakes. Art is knowing which ones to keep.
I first need to understand how is this occuring in order to see where the problem is.
As said by you, the schema file results from the columns tab in the copy stage, did you try to view what is being generated.
There can be two possiblities where the problem is occuring.
1. The schema file is not generated properly.
2. The importer is not operating as it should be.
And yea I am not in Australia.
My existance is in USA.
As said by you, the schema file results from the columns tab in the copy stage, did you try to view what is being generated.
There can be two possiblities where the problem is occuring.
1. The schema file is not generated properly.
2. The importer is not operating as it should be.
And yea I am not in Australia.
My existance is in USA.
Last edited by DSguru2B on Thu May 11, 2006 11:13 am, edited 1 time in total.
Creativity is allowing yourself to make mistakes. Art is knowing which ones to keep.
Hi,
I'm following everything up to where you 'generate a schema file' from the table definition. I have 7.5, how do you do this?
I'm following everything up to where you 'generate a schema file' from the table definition. I have 7.5, how do you do this?
Carter[/quote].... Theoretically, this would enable me to add a field to this table definition, generate a schema file, save it to the OS and boom! the new field shows up in all 19 files because I am also using column propagation.
You mean what is step 5? That's easy. Copy and paste the results of step 3 and paste it into a UNIX file. Yes, it would be oh so sweet to do a save as and FTP it to UNIX but baby steps, baby steps. I'm concerned with generating it correctly. Actually saving it as a schema file is a bell and whistle until I get the first problem resolved.