Data Masking Pack
Moderators: chulett, rschirm, roy
Data Masking Pack
Hello,
We are using data masking pack in 8.7 version of datastage. The job with masking stage is taking too much of time like 20 min for 100K records , with out the masking stage it takes only 2 minutes.
What is the usual runtime for 100K records with only 5 or 6 stages?
Do we need to do anything to increase the performance?
We are using data masking pack in 8.7 version of datastage. The job with masking stage is taking too much of time like 20 min for 100K records , with out the masking stage it takes only 2 minutes.
What is the usual runtime for 100K records with only 5 or 6 stages?
Do we need to do anything to increase the performance?
Thanks,
Surya
Surya
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
How long does it take Optim alone to perform this masking?
DataStage data masking acts as a wrapper that processes the data and invokes Optim data masking. So you'd expect some overhead but probably not a lot.
How complex are the masking algorithms? Keep in mind that you can require that the statistical relationships are preserved in the data, which carries quite a large overhead in its own right.
DataStage data masking acts as a wrapper that processes the data and invokes Optim data masking. So you'd expect some overhead but probably not a lot.
How complex are the masking algorithms? Keep in mind that you can require that the statistical relationships are preserved in the data, which carries quite a large overhead in its own right.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Hi Ray,
We were not using Optim for doing the masking,earlier before this pack we were doing most of the masking with in the DS stages.
But now as we have data masking pack we are using it and see this kind of performance.
Also we are using repeatable replace function in data masking pack for only 3 fields.
We were not using Optim for doing the masking,earlier before this pack we were doing most of the masking with in the DS stages.
But now as we have data masking pack we are using it and see this kind of performance.
Also we are using repeatable replace function in data masking pack for only 3 fields.
Thanks,
Surya
Surya
-
- Premium Member
- Posts: 2
- Joined: Thu Mar 28, 2013 8:23 am
- Location: United States
- Contact:
Masking Performance
You may want to consider utilizing Data Stage as just the ETL portion of your masking. You can leverage specific masking software, such as DMsuite which will allow you to perform masking at about 1 million rows per minute and does not require any coding on your part.
Joseph I Santangelo
-
- Premium Member
- Posts: 2
- Joined: Thu Mar 28, 2013 8:23 am
- Location: United States
- Contact:
Business Logic in Masking Routines
I would suggest that you do not place any business logic in your data masking routines. It is generally bad practice, it means that you are creating an entire application just to mask an existing application.
I offer a free consultation to come up with a solution to your reqreuiments. We have been doing masking for 9 years now with many large multi-nationals and we deal with many ETLs including Informatica.
I offer a free consultation to come up with a solution to your reqreuiments. We have been doing masking for 9 years now with many large multi-nationals and we deal with many ETLs including Informatica.
Joseph I Santangelo
Thank you Jisan.
We are not placing businees logic in our data masking routines.Its just the business requirements we are taking care to implement them using datastage.
Most likely all the requirments have been handled, its only the performance issue where it is taking more time in 8.5 when used data masking pack.
Please let me know if there is any performance improvement in data masking pack in the recent version 9.1 so that we will upgrade it.
Thanks again:)
We are not placing businees logic in our data masking routines.Its just the business requirements we are taking care to implement them using datastage.
Most likely all the requirments have been handled, its only the performance issue where it is taking more time in 8.5 when used data masking pack.
Please let me know if there is any performance improvement in data masking pack in the recent version 9.1 so that we will upgrade it.
Thanks again:)
Thanks,
Surya
Surya
I don't believe there are changes in the Data Masking Pack for IS 8.5, 8.7 and 9.1. You can examine the New Features and Release Notes for 9.1 (a link is available here) to see if there are any changes or improvements listed.
If you suspect a performance problem with the pack, I recommend that you contact your official support provider for assistance.
Regards,
If you suspect a performance problem with the pack, I recommend that you contact your official support provider for assistance.
Regards,
- james wiles
All generalizations are false, including this one - Mark Twain.
All generalizations are false, including this one - Mark Twain.