I was wondering whether I can somehow use a schema to define the input from a data set.
I was planning to use data sets to pass the data between my jobs, as the documentation suggests "using data sets wisely can be key to good performance in a set of linked jobs".
However I can't find in the doco how to actually do it.
Job 1 looks like this:
SeqFile --> ColumnImport -> Transformer --> Dataset
\
\--> SeqFileOut
ColumnImport and SeqFileOut are using the same schema file. RCP is enabled for all ouput links. This job works fine. I can't verify exaclty what is stored in a dataset, but the contents of the SeqFileOut has all the expected columns and values.
Job 2 should be able to read in the dataset created by Job2. Something like this:
Dataset --> <Some processing> --> DBTable
I'm not sure how I can retrieve the data from the Dataset created in Job1 using the same schema. Can I instruct dataset to use schema at all?
I suspect that it might not be, as I can't find any option to set in the Dataset stage.
Are there any alternative ways to read the dataset using a schema file?
Dataset and schemas
Moderators: chulett, rschirm, roy
Re: Dataset and schemas
Oops! The link to SeqFileOut should start in the Transformer stage.evee1 wrote: SeqFile --> ColumnImport -> Transformer --> Dataset
\
\--> SeqFileOut
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
You could use a fileset instead of a dataset if you wish, then you can specify a schema file. If your schema won't be changing over time, you can store the metadata in a table definition and load the column definitions from that in Job 2.
View the contents of the dataset using either the dataset management tool GUI or orchadmin from a command shell.
Regards,
View the contents of the dataset using either the dataset management tool GUI or orchadmin from a command shell.
Regards,
- james wiles
All generalizations are false, including this one - Mark Twain.
All generalizations are false, including this one - Mark Twain.