Masked Data

diamondabhi · Post by **diamondabhi** » Mon Jun 13, 2005 7:25 am

Hi All,
Is there any technique in DataStage that will allow us to masked Productional data to create test data? We have a project that consists of sensitive data - ssn, hourly wage, name, address, etc. We would like to use good test data that is not production data.

Thanks,
Abhi.

ArndW · Post by **ArndW** » Mon Jun 13, 2005 7:37 am

In the past I've written a number of jobs that take the "live" data and run it through a transform which either randomizes columns or masks information, depending upon what needs to be done. In many cases randomizing would ruin the correlations (i.e. if the customer name is the key for a lookup), so a simple text-based encoding would be used (i.e. subtract 13 from the ascii value of each character) in all occurrences of that name. This can end up being a lot of work if the keys are identical with the data that should be masked. Then again, the big project I was working on was with millions of credit card records and transactions, so the anonymizing step was quite important during the initial test phase.

diamondabhi · Post by **diamondabhi** » Mon Jun 13, 2005 8:08 am

ArndW,
Can you be more elaborate as to how to mask the data or if have already built some jobs can you post some codes for me.

Thanks,
Sai.

ArndW · Post by **ArndW** » Mon Jun 13, 2005 8:27 am

Diamondabhi,

I won't post any jobs or code for this, but will elaborate a bit:

If you have numerics that need to be masked use the RND() function which returns a pseudo-random number [it is important to seed it with a constant value if you need reproduceable numbers on subsequent runs]. So if you have a currency amount you want masked (i.e. 45432.00) I would either do a completely random number as in RND(9999999)/100 or RND(100)/10*In.Currency to use the original in some fashion. Text fields with names that need to stay consistant between tables (key fields) can be masked:

Code: Select all

Function Mask(InputString)
StringLen = LEN(InputString)
Ans = ''
FOR i = 1 TO StringLen
  Modulus = MOD(i,3)
          IF  Modulus=0 THEN Ans := CHAR(SEQ(InputString[i,1])-1)
  ELSE IF  Modulus=1 THEN Ans := CHAR(SEQ(InputString[i,1])+1)
                               ELSE  Ans := InputString[i,1]
NEXT i

This example is a bit too simple, but it should give you an idea of what can be done. You could even go in and use a SOUNDEX conversion, but that would give you a lot of duplicates.

Perhaps you could explain if you need a simple or complex masking done - and specifically for which type of constructs.

diamondabhi · Post by **diamondabhi** » Mon Jun 13, 2005 9:34 am

ArndW,
Thanks a lot Arnd, I need simple coding for masking the SSN and EmpNames.

Thanks,
Abhi.

Sainath.Srinivasan · Post by **Sainath.Srinivasan** » Mon Jun 13, 2005 10:23 am

If all you want is a few rows in the file, you can generate yourself using the transformer and a stage variable and make it to write them out in any file you want.

You can pass your SSN to EmpName to match them during testing.

ketfos · Post by **ketfos** » Mon Jun 13, 2005 3:02 pm

Hi,
You could use a simple Ereplace function in the transformer where you can replace occurences of string with another.

Thks
Ketfos