DSXchange

Hi,

I have 5 columns(say a,b,c,d,e) and i want the counts of each column ignoring null values.

My input file will be in tera bytes. Could some one suggest the best method to design in datastage in parallel version?

Thanks,
Kumar

You want to count the number of columns that do not contain nulls or do something with the counts from inside all non-null columns? Seems to me you'll still have to check each one individually for null so I'm thinking just stage variables in a transformer will be appropriate.

Aggregator stage, Non-missing values count output column calculation method. Split your data into five streams, one for each column to be counted. Eliminate all unneeded data.

Thanks for the reply.

I think in both the scenarios above, the data stage job should run in sequence mode(Not in parallel). Is my understadning correct?

Why do you think that? Are there other requirements for grouping that you haven't shared?

If there are no grouping columns you will at some point need to perform a count sequentially (otherwise you'll get one answer for each partition).
However if you have lots of data and wish to take adavantage of parallelism you can perform the initial counts in parallel and then add another stage running sequentially to sum up the counts across the partitions.

thompsonp wrote:If there are no grouping columns you will at some point need to perform a count sequentially (otherwise you'll get one answer for each partition).

Good point. That's true... unless (as you noted) there are other columns not mentioned that will allow say a hash partition so that does not become an issue.

Generate a constant and group by that.

Seems i got very good replies here. Yes i will process in parallel mode and at the end i will group based on constact value.

Thanks guys.