DataBase loading
Moderators: chulett, rschirm, roy
-
- Participant
- Posts: 248
- Joined: Fri Jul 30, 2010 9:04 am
DataBase loading
A file is sent to the server every week and the records from the file should be loaded into a table.
Suppose first week there are 100 records second week there may be more than that around 120 to 140.
before loading into the table we need to find out only the new ones and load them I mean append to the table.
So my idea is to do a lookup with the table(loaded earlier) and the file and the rows which are rejected are loaded in the table.
Is this a good idea,please suggest me
Suppose first week there are 100 records second week there may be more than that around 120 to 140.
before loading into the table we need to find out only the new ones and load them I mean append to the table.
So my idea is to do a lookup with the table(loaded earlier) and the file and the rows which are rejected are loaded in the table.
Is this a good idea,please suggest me
Regards,
Kenny
Kenny
-
- Participant
- Posts: 248
- Joined: Fri Jul 30, 2010 9:04 am
-
- Participant
- Posts: 248
- Joined: Fri Jul 30, 2010 9:04 am
-
- Participant
- Posts: 248
- Joined: Fri Jul 30, 2010 9:04 am
The job design is
ftp stage------>transforrmer(stream to lookup)---------->lookup(and the reference isTable)-------->(reject from lookup)Table
The ftp stage is used to read the data from an other server and file comes every week to that server and after reading the file and transform according to the requirements I had to do a lookup with the table and load the rows which are not loaded last week.
just like the rows which does not match with the last weeks data are appended.
so I used only a reject fromt he look up which gives only the records which are not matched.
this error occurs when compiling the job.
the options I used in the properties of lookup are reject and reject.
ftp stage------>transforrmer(stream to lookup)---------->lookup(and the reference isTable)-------->(reject from lookup)Table
The ftp stage is used to read the data from an other server and file comes every week to that server and after reading the file and transform according to the requirements I had to do a lookup with the table and load the rows which are not loaded last week.
just like the rows which does not match with the last weeks data are appended.
so I used only a reject fromt he look up which gives only the records which are not matched.
this error occurs when compiling the job.
the options I used in the properties of lookup are reject and reject.
Regards,
Kenny
Kenny
-
- Participant
- Posts: 342
- Joined: Tue Nov 04, 2008 10:38 am
- Location: Chennai, India
-
- Participant
- Posts: 248
- Joined: Fri Jul 30, 2010 9:04 am
When I use the output link and reject link that works but my job is to get only the rejected records so I used only the reject link bot the output link.
so I got this error.
Is there a stage other then I look up stage to get only the rejected records as
I need only the rejected records.
so I got this error.
Is there a stage other then I look up stage to get only the rejected records as
I need only the rejected records.
Regards,
Kenny
Kenny
-
- Participant
- Posts: 248
- Joined: Fri Jul 30, 2010 9:04 am
-
- Participant
- Posts: 342
- Joined: Tue Nov 04, 2008 10:38 am
- Location: Chennai, India
-
- Participant
- Posts: 248
- Joined: Fri Jul 30, 2010 9:04 am
Lookup requires a stream output, but does not require a reject link. If you only have a reject link and no stream output the compiler will complain. Send the lookup success rows to a copy with no output or a seq file stage and write to /dev/null or some such thing.
Mike Hester
mhester@petra-ps.com
mhester@petra-ps.com