Filter in sequential file.

Christina · Post by **Christina** » Tue Mar 27, 2007 11:33 pm

My source file contains 24 months data. I have to extract first 12 months data from that file using filter command in sequential file. Which command i have to use to achive this requirement.

chulett · Post by **chulett** » Tue Mar 27, 2007 11:47 pm

What makes you think you have to use the filter command to achieve this?

That sounds more like a job for a Transformer constraint. How will you recognize these 'first 12 months' of data? What is the business rule you are attempting to implement? The answer to that question will help determine the appropriate place to do the dirty deed.

Christina · Post by **Christina** » Wed Mar 28, 2007 12:01 am

We already have a job for this with transformer constraints. Since the data volume is high it is taking more time. So we are trying for some methods to improve the performance. Is there anyother way to achive this other than transformer constraints?(which will reduce the time taken).

thanks in advance.

chulett · Post by **chulett** » Wed Mar 28, 2007 12:06 am

chulett wrote:How will you recognize these 'first 12 months' of data? What is the business rule you are attempting to implement? The answer to that question will help determine the appropriate place to do the dirty deed.

ps. Unless your constraints are horribly complex, I don't see how they could be your 'taking more time' culprit. We'd have to have a better idea of your job design in order to provide any specific help there.

Christina · Post by **Christina** » Wed Mar 28, 2007 12:19 am

constraints are not that much complex. We will be passing the first month value and will compare this with the date value in the source file and will find the month difference. Based on this month difference we are splitting this 24 months data into 2 files of 12 months data. we have nearly 100 million records. It is taking 4 to 5 hours to load this files.
can we reduce this time.

DSguru2B · Post by **DSguru2B** » Wed Mar 28, 2007 7:00 am

Make your job multiple instance. Read a subset for each instance. Depending upon the resources available, run as many instances simultaneously.
A much faster approach, bulk load the file into a work table and write a sql query to extract the data. Again two sets of 12 months each.

chulett · Post by **chulett** » Wed Mar 28, 2007 7:30 am

In other words - it's not the constraints. Seek your solution elsewhere. You've already gotten a couple of good suggestions, let us know if you need any help with them.