DSXchange

Hi,

I need the count of records in a paralled job. I separate the data based on the conditions (constraint) from the transformer stage and send them to the aggregator stage to count the records.

I had a quick question here do I need to sort the data based on the key column before sending it to the target as I am using a parallel DS job or would the job do it correctly without the sort stage.

Thanks in advance

It is always good practice to sort the data before performing any aggregation. In your case if you just want to count the records processed or filtered, I suggest you better look for the DSGetLinkInfo().

Thanks for the reply.
But i do not relie on the link count because the link count behave weird at times and give some different count I don't know why.

devidotcom wrote:Thanks for the reply.
But i do not relie on the link count because the link count behave weird at times and give some different count I don't know why.

Does anyone know why the link gives weird numbers? Is that a bug?

We have only your assertion that the numbers are "weird". What do you mean by this term? Where's the proof?

Have you tried setting the environment variable that causes player processes to report their row counts?

For getting a count, you don't need to sort. It doesn't matter how many nodes you have, an aggregator will do. It is probably the simplest of aggregator functions.

If you want the total record count then you don't have to sort. If you want the count based on a key field then sort and hash partition on the key.