RCP and adding new columns to flow
Moderators: chulett, rschirm, roy
RCP and adding new columns to flow
Hello!
I am trying out Runtime Column Propagation (RCP) using a simple sequential file (reading the metadata from a Schema file), a Transformer and another sequential file to write the data to.
What do i need to do:
Add a date column to the flow.
What do i have:
Just three columns, lets suppose i have this file:
col1,col2,col3
100,200,300
i read it using RCP and a schema file then process it and i want this as output:
date,col1,col2,col3
20090303,100,200,300
Is there any way to do that using RCP? because i have like 40 jobs where the only thing i have to do is this (well that and write from text to dataset) and i would love to save the development time.
Currently, adding a single column in the transformer stage writing the date only returns:
20090303
and none of my original schema read columns.
Any help? maybe another way to do it?
Thanks!
I am trying out Runtime Column Propagation (RCP) using a simple sequential file (reading the metadata from a Schema file), a Transformer and another sequential file to write the data to.
What do i need to do:
Add a date column to the flow.
What do i have:
Just three columns, lets suppose i have this file:
col1,col2,col3
100,200,300
i read it using RCP and a schema file then process it and i want this as output:
date,col1,col2,col3
20090303,100,200,300
Is there any way to do that using RCP? because i have like 40 jobs where the only thing i have to do is this (well that and write from text to dataset) and i would love to save the development time.
Currently, adding a single column in the transformer stage writing the date only returns:
20090303
and none of my original schema read columns.
Any help? maybe another way to do it?
Thanks!
--
Juan José Sisti
Juan José Sisti
-
- Participant
- Posts: 3337
- Joined: Mon Jan 17, 2005 4:49 am
- Location: United Kingdom
I have verified that RCP is enabled.
I am trying the column generator option, but how do i do so it uses certain function while generating the data?
the only option i see is
Cycle (cycles thru several options)
Alphabet (chooses one letter each time from a chosen string)
Thanks in advance!
I am trying the column generator option, but how do i do so it uses certain function while generating the data?
the only option i see is
Cycle (cycles thru several options)
Alphabet (chooses one letter each time from a chosen string)
Thanks in advance!
--
Juan José Sisti
Juan José Sisti
What is the 'certain function' you want to apply? From your example, I think we (at least I was) assumed you were using a literal each time. If it's more complex than that, this stage won't suit as it's intended to produced test data really, providing the same value over and over or working around a list over and over. It can't employ transform functions if you are wanting to do something fancier with your date you want to add.
Mark Winter
<i>Nothing appeases a troubled mind more than <b>good</b> music</i>
<i>Nothing appeases a troubled mind more than <b>good</b> music</i>
-
- Participant
- Posts: 3337
- Joined: Mon Jan 17, 2005 4:49 am
- Location: United Kingdom
I think the only solution is to process the file as a whole huge record and then add another column (having the current date) at the transformer level, this should be done in a first job, the second job should use RCP and i will already have [date]<separator>[original data], hence i will be able to use only one job.
And since the first job only reads the data as one huge record instead of delimited, i dont need to define any special metadata for any of the files.
that way with three jobs (the third being a dataset to oracle inserter) i am able to process all the files, only using schema definitions.
If anyone thinks of a better solution i am still open to your help!
And since the first job only reads the data as one huge record instead of delimited, i dont need to define any special metadata for any of the files.
that way with three jobs (the third being a dataset to oracle inserter) i am able to process all the files, only using schema definitions.
If anyone thinks of a better solution i am still open to your help!
--
Juan José Sisti
Juan José Sisti
-
- Participant
- Posts: 3337
- Joined: Mon Jan 17, 2005 4:49 am
- Location: United Kingdom
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
-
- Participant
- Posts: 3
- Joined: Tue Jan 11, 2011 10:25 am
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Welcome aboard. How do you expect juancho to know what U (one of our other posters) did?
Please strive for a professional standard of written English on DSXchange. It is a web site, not a mobile telephone, so there is no need for SMS-style abbreviations. Many of our posters (and lurkers) do not have English as their first language, so even good English is enough of a challenge for them.
Please strive for a professional standard of written English on DSXchange. It is a web site, not a mobile telephone, so there is no need for SMS-style abbreviations. Many of our posters (and lurkers) do not have English as their first language, so even good English is enough of a challenge for them.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.