Data from Log File

suryadev · Post by **suryadev** » Tue Jul 16, 2013 2:25 pm

Hello,

Below is the sample data from a log file

sampleXXXXXXXXXXXXXX
SSN: 123456789
XXXX
XXXXXXXXXX
XXXXXXXXXXX

Sample2XXXXXXXXXXXX
SSN:987654321
XXXX
XXXXXXXXXX
XXXXXXXXXXXXX

There are around 100M such records in this log file and position of SSN is also different in each single paragraph

Please suggest me how to pull the SSN value from the log file and make it some number

Thank you

chulett · Post by **chulett** » Tue Jul 16, 2013 2:56 pm

Read the records as one long string field, filter out any that do not start with 'SSN:' and then use Field() to get the second field from that ":" delimited string. Convert.

suryadev · Post by **suryadev** » Tue Jul 16, 2013 2:58 pm

Thank you!

So the complete file will be a single field? and also the target should look the same as source with only changes SSN values.

Please correct me if I am wrong

suryadev · Post by **suryadev** » Tue Jul 16, 2013 3:57 pm

I did read the file with a sequential stage, the data looks god when I view it.

Now the records can be filtered where I see SSN but how can I combine the records which are dropped earlier?

Thanks again!

chulett · Post by **chulett** » Tue Jul 16, 2013 4:14 pm

You never mentioned that part in your original post, I was under the impression all you wanted were the SSN values which is why I provided the answer that I did. Can you be more specific with regards to exactly what your end result should be?

suryadev · Post by **suryadev** » Tue Jul 16, 2013 9:14 pm

Sorry for the confusion.
Below is the source
sampleXXXXXXXXXXXXXX
SSN: 123456789
XXXX
XXXXXXXXXX
XXXXXXXXXXX

Sample2XXXXXXXXXXXX
SSN:987654321
XXXX
XXXXXXXXXX
XXXXXXXXXXXXX

Below is the Target
sampleXXXXXXXXXXXXXX
SSN: 576195198
XXXX
XXXXXXXXXX
XXXXXXXXXXX

Sample2XXXXXXXXXXXX
SSN:348231890
XXXX
XXXXXXXXXX
XXXXXXXXXXXXX

I tried doing it as you said and in the transformer can use an If condition and change it as needed but my main issue was I need to pass this data to the masking stage and all I can do in the masking stage is find the position in that particular record where there is SSN and mask it but how to handle the other records in the log file...that is where I cannot move further...

Please suggest me some solution

chulett · Post by **chulett** » Tue Jul 16, 2013 9:35 pm

So that's what you meant by "make it some number". Based on your description and the subject of your post I thought you wanted to extract it as "data from the log file" and convert it from a string to a number, not mask it. Always helps to fully explain what you are trying to do sooner rather than later.

I have zero experience with the Masking stage. Does it need to be bypassed for the records that don't need masking or can you not tell it which ones are just pass through? Anyone know?

suryadev · Post by **suryadev** » Wed Jul 17, 2013 3:10 pm

If there are several fields we can give functions to only fields which need masking but in our case we have only 1 field where we need to mask for only some records where ever these is SSN.

ray.wurlod · Post by **ray.wurlod** » Wed Jul 17, 2013 4:50 pm

Do you need to be able to reverse the mapping to a masked value?

In either case, creating a routine is probably the easiest way to go. The routine needs to generate the same (?) unique replacement for each SSN.

suryadev · Post by **suryadev** » Mon Jul 29, 2013 9:36 am

Thanks very much!

Reverse the mapping? do you mean creating a seperate flow to the masking pack?

How can the routine be created? writing an own routine?
In the list of the records I need to search where ever it says SSN?

As given below

Below is the source
sampleXXXXXXXXXXXXXX
SSN: 123456789
XXXX
XXXXXXXXXX
XXXXXXXXXXX

Sample2XXXXXXXXXXXX
SSN:987654321
XXXX
XXXXXXXXXX
XXXXXXXXXXXXX

Below is the Target
sampleXXXXXXXXXXXXXX
SSN: 576195198
XXXX
XXXXXXXXXX
XXXXXXXXXXX

Sample2XXXXXXXXXXXX
SSN:348231890
XXXX
XXXXXXXXXX
XXXXXXXXXXXXX

Can field function be used to find the position for SSN as it will in the same position when searched as records but not sure where in will be as columns as now it is in second line , sometimes third line.....

Please suggest how to use it to search for word SSN in the records?

chulett · Post by **chulett** » Mon Jul 29, 2013 10:10 am

By 'reverse the mapping to a masked value' Ray was asking if you had a need to be able to undo or 'unmask' the masked value back to its original value. Hopefully not as masking is a one-way process.

Not seeing the problem with SSN, as noted earlier all it seeems you need to do from the examples you posted is check to see if the first four characters are 'SSN:'.

suryadev · Post by **suryadev** » Tue Jul 30, 2013 9:49 am

Thank you!

It is not possible to unmask back again...as the masking pack is irreversible.

I got an idea of generating sequence number to all the records and just send the records with SSN to masking pack and after that use the two links(SSN records and Non SSN records) to funnel stage and sort based on sequence number so that the records order remains the same. hope that works!