Hi All,
We are currently working on a Customer MDM solution with Quality stage. Jobs are developed with Standardization, Match Specification stages. Now need to tune it to improve the performance / space requirement. Also need to estimate QS memory requirement for volume like,
History load : 150 M (one time)
Incremental load : 6-8 M (weekly)
Can you please suggest some steps to calculate the System memory requirement ? How to calculate block volume in Ruleset ? Any document / link will also be helpful.
Thanks in advance !
Ankita
Quality stage memory requirement
Re: Quality stage memory requirement
Hi ankita,
The purpose of Ruleset is to parse the input data (standardization) only.
The space for standardized data depends up on the number of rules you going to use. Test run with one million and find out the size of dataset file and make it for 150m .
Better you can remove the standardization output columns I.E. unhandled pattern or unhandled value etc which is not used by the match specification. this way you can optimize the size of dataset.
The memory needed for the match spec calculation is critical.
It depends up on your blocking factor , matching algorithm ,etc
while running you can monitor which pass taking long time and consuming more resources and you can tweak it.
make the number of nodes equal to the number of processor. its a good practice.
give us some clue about your h/w?
Regards
Vairamuthu
The purpose of Ruleset is to parse the input data (standardization) only.
The space for standardized data depends up on the number of rules you going to use. Test run with one million and find out the size of dataset file and make it for 150m .
Better you can remove the standardization output columns I.E. unhandled pattern or unhandled value etc which is not used by the match specification. this way you can optimize the size of dataset.
The memory needed for the match spec calculation is critical.
It depends up on your blocking factor , matching algorithm ,etc
while running you can monitor which pass taking long time and consuming more resources and you can tweak it.
make the number of nodes equal to the number of processor. its a good practice.
give us some clue about your h/w?
Regards
Vairamuthu
ankita wrote:Hi All,
We are currently working on a Customer MDM solution with Quality stage. Jobs are developed with Standardization, Match Specification stages. Now need to tune it to improve the performance / space requirement. Also need to estimate QS memory requirement for volume like,
History load : 150 M (one time)
Incremental load : 6-8 M (weekly)
Can you please suggest some steps to calculate the System memory requirement ? How to calculate block volume in Ruleset ? Any document / link will also be helpful.
Thanks in advance !
Ankita
vairamuthu