Droping Columns in FILTER stage

Raamc · Post by **Raamc** » Wed Apr 23, 2008 4:23 am

Hi,

I need to drop some of the columns from Input file. I am using the FILTER stage immediately after the sequential file to filter out some records.

I know that the COPY stage is the best practice for dropping the columns. Here my doubt is, instead of using another stage, can we drop columns in the FILTER stage itself.

Is it effect the performance?

ArndW · Post by **ArndW** » Wed Apr 23, 2008 5:40 am

Actually, the best stage to drop columns with is the Modify stage, and since the Filter stage won't let you drop columns I suggest that you use the Modify Stage.

ray.wurlod · Post by **ray.wurlod** » Wed Apr 23, 2008 7:04 am

The best stage to use to drop columns is the Copy stage, as its processing overheads are zero. The "transfer" is done by pointing the output column to the same address as the input column.

OddJob · Post by **OddJob** » Wed Apr 23, 2008 7:06 am

But if you're using RCP the only way to drop fields is by using a modify stage.

ray.wurlod · Post by **ray.wurlod** » Wed Apr 23, 2008 7:14 am

I never use RCP. It destroys any hope you may have had of performing lineage analysis.

OddJob · Post by **OddJob** » Wed Apr 23, 2008 7:20 am

Never say 'never', that's what I always say.

ray.wurlod · Post by **ray.wurlod** » Wed Apr 23, 2008 3:22 pm

Raamc · Post by **Raamc** » Fri Apr 25, 2008 4:28 am

Hi All,

Thanks for the Replies.

I am using the COPY stage as suggested by RAY for drop the columns.

ArndW · Post by **ArndW** » Fri Apr 25, 2008 6:20 am

I have come to love both RCP and the Bomb.