DataBase loading

kennyapril · Post by **kennyapril** » Thu Oct 28, 2010 2:15 pm

A file is sent to the server every week and the records from the file should be loaded into a table.

Suppose first week there are 100 records second week there may be more than that around 120 to 140.

before loading into the table we need to find out only the new ones and load them I mean append to the table.

So my idea is to do a lookup with the table(loaded earlier) and the file and the rows which are rejected are loaded in the table.

Is this a good idea,please suggest me

chulett · Post by **chulett** » Thu Oct 28, 2010 3:36 pm

Yup, good idea.

kennyapril · Post by **kennyapril** » Fri Oct 29, 2010 12:24 pm

thanks,

When I do a look up there should be a stream and a reject.
Can I link only a reject from the lookup to the table with out the stream link?

chulett · Post by **chulett** » Fri Oct 29, 2010 12:27 pm

As far as I recall, yes. Have you tried it yet?

kennyapril · Post by **kennyapril** » Mon Nov 01, 2010 12:54 pm

I tried to use only a reject as said in my message but I see an error for that which says
"An UnIdentified error has occured"

please suggest!!

chulett · Post by **chulett** » Mon Nov 01, 2010 5:31 pm

Can you post the details of your job design, please? And you get this error when, exactly? Runtime?

kennyapril · Post by **kennyapril** » Tue Nov 02, 2010 9:25 am

The job design is

ftp stage------>transforrmer(stream to lookup)---------->lookup(and the reference isTable)-------->(reject from lookup)Table

The ftp stage is used to read the data from an other server and file comes every week to that server and after reading the file and transform according to the requirements I had to do a lookup with the table and load the rows which are not loaded last week.
just like the rows which does not match with the last weeks data are appended.

so I used only a reject fromt he look up which gives only the records which are not matched.

this error occurs when compiling the job.
the options I used in the properties of lookup are reject and reject.

vinothkumar · Post by **vinothkumar** » Tue Nov 02, 2010 9:34 am

Can you create a simple job having a look-up with reject link and otuput link and check whether you can compile there. If you can't compile there, then it will be a patch issue. We also had same issue when we use reject link from LookUp. After patch fix, it went off.

kennyapril · Post by **kennyapril** » Tue Nov 02, 2010 9:42 am

When I use the output link and reject link that works but my job is to get only the rejected records so I used only the reject link bot the output link.

so I got this error.

Is there a stage other then I look up stage to get only the rejected records as
I need only the rejected records.

thompsonp · Post by **thompsonp** » Tue Nov 02, 2010 10:55 am

Instead of using the reject link, why not set the lookup to 'continue' when no match is found.
Then filter the rows that out the rows that matched and only process the new ones.

kennyapril · Post by **kennyapril** » Tue Nov 02, 2010 12:08 pm

thanks,

I see only two conditions in the lookup
one is condition not met and other one lookup failure
do I need to use continue for these two?
or do I need to assign a new condition, please suggest how can I do that?

vinothkumar · Post by **vinothkumar** » Tue Nov 02, 2010 12:24 pm

For LookUp failure keep it as continue

kennyapril · Post by **kennyapril** » Tue Nov 02, 2010 12:34 pm

so to get the rejected rows....Is it continue for the two conditions?

nitkuar · Post by **nitkuar** » Wed Nov 03, 2010 3:10 am

kennyapril wrote:so to get the rejected rows....Is it continue for the two conditions?

it should be continue for 'Lookup Failure' dropdown. anyway if you haven't given any condition in lookup link other dropdown 'Condition not met' value won't matter.

mhester · Post by **mhester** » Wed Nov 03, 2010 6:38 am

Lookup requires a stream output, but does not require a reject link. If you only have a reject link and no stream output the compiler will complain. Send the lookup success rows to a copy with no output or a seq file stage and write to /dev/null or some such thing.