DSXchange

Hi there,

I am processing a large resultset (6M + records) through the USPREP ruleset.

My question is what is the best way to run such a resultset through the standardize stage efficiently. Right now I'm processing at 1240 rec/sec.

It appears that the standardize stage is really slow. I can understand why, it's doing alot of work. Just wondering what are some of the best practices for making this run efficiently.

Thanks.

If the system is not running out of resources, use a configuration file with more nodes.

It might be possible to write a more efficient rule set, but the benefit is probably not worth the cost.

Understood.

Thanks for the reply.

Sean

It might be worth taking out any columns that are not being processed and join them again later on. It adds a bit on complexity, but if that takes a lot of records out of the standardization stage, it might be worth it.

Matt

Good point. I was thinking the same thing.

Thanks!

Good point. I was thinking the same thing.

Thanks!