Pattern Matching

danddmrs · Post by **danddmrs** » Mon Nov 15, 2010 1:57 pm

A job using pattern matching now fails to match patterns.
There are 2 stage variable:
svMaskPattern jprFormatMasks
svNoMatchFound inLink.FieldName Matches svMaskPattern

value of jprFormatMasks is 0:Char(253):"10N'C'":Char(253):"'UAFA'9N":Char(253):"'UAFA'8N'C'"

input has mixture of matches and non matches, constraint is svNoMatchFound = 0 with a reject added for testing. If I strip everything out of the parameter and just pass 1 mask it works, 10N'C' for example allows values with 10 numbers and last character of C to pass. As soon as I add multiple masks none are found.

To test my masks I set up a routine that passed my input as arg1 then tested the multiple masks:
If Arg1 Matches 0:Char(253):"10N'C'":Char(253):"'UAFA'9N":Char(253):"'UAFA'8N'C'"
Then 1
Else 0
and that works (matches return 1, non-matches 0).
When I passed the same mask as Arg2 it failed
If Arg1 Matches Arg2
Then 1
Else 0
all return 0.

Any thoughts would be appreciated. And yes this did work at one time.

Thanks!

ray.wurlod · Post by **ray.wurlod** » Mon Nov 15, 2010 2:03 pm

How do you construct Arg2?

danddmrs · Post by **danddmrs** » Tue Nov 16, 2010 8:21 am

Using a copy of the testing Routine I cut 0:Char(253):"10N'C'":Char(253):"'UAFA'9N":Char(253):"'UAFA'8N'C'" and replaced it with Arg2. Compiled and tested, pasting 0:Char(253):"10N'C'":Char(253):"'UAFA'9N":Char(253):"'UAFA'8N'C'" as Arg2.

I thought this would work and was planning on replacing my Stage Variable with a routine but it has the same problem.

Also, in a copy of the original job I replaced the parameter in the Matches function with the actual patterns and that works...

Replaced inLink.FileName Matches svMaskPatterns with
inLink.FileName Matches 0:Char(253):"10N'C'":Char(253):"'UAFA'9N":Char(253):"'UAFA'8N'C'"
and the results were as expected.

Thanks.

danddmrs · Post by **danddmrs** » Tue Nov 16, 2010 8:36 am

First Routine:

If FolderID Matches 0:Char(253):"10N'C'":Char(253):"'UAFA'9N":Char(253):"'UAFA'8N'C'"
Then Ans = 0
Else Ans = 1
Results in 0 for matches, 1 for non

Second Routine with 2 args
If FolderID Matches Masks
Then Ans = 0
Else Ans = 1
Result is 1 for all. Mask value was cut and pasted from first routine.

When I put only 1 pattern in the Mask arg it works.
FolderID = A1000P1234567
Mask = 'A1000P'7N
Ans = 0

Passing the same mask with multiple masks fails
FolderID = A1000P1234567
Mask = 0:Char(253):"'A1000P'7N":Char(253):"'A2000X'5N"
Ans = 1

danddmrs · Post by **danddmrs** » Tue Nov 16, 2010 12:59 pm

For clarifaction...

1. The job that was working was a proof of concept effort that had the masks hard coded in the transformer. Only when attempting to pass the masks into the job using a parameter did I run into problems. Passing the masks into the program has never worked.

2. The reason for passing the masks is that they are derived from business masks maintained by users. There are also different businesses using a variety of masks that need to be used to validate their specific data.

If it makes any difference masks are created as follows:

seq1 --> xfm1 -- seq2 --> xfm2 --> hf1 -- xfm3 -- seq3

Seq1 = Business Masks
xfm1 = Transform into DataStage Basic Masks
seq2 = Not really needed but used for validating the conversion
xfm2 = Using a dummy key flatten all the masks into one record with Char(253) inserted between each mask
hf1 = Hashed file for flattening the file
xfm3 = Copy the one record Hashed file to a sequential file

If the masks from seq3 are copied and pasted into the Transformer using the matches function it works.
If the masks are copied and pasted into a job parameter it fails to find a match.
If seq3 is loaded into the parameter from a user variable it also fails to find a match.

ray.wurlod · Post by **ray.wurlod** » Tue Nov 16, 2010 1:56 pm

You can not pass a function as, or as part of, an argument. Replace all references to Char(254) with the system variable @FM.

danddmrs · Post by **danddmrs** » Tue Nov 16, 2010 3:30 pm

Tried @FM and @VM.

@FM doesn't work at all. @VM works only when hardcoded, not passed.

Tried loading the sv directly from the hashed file. Same result.

Looking for a new direction.

Thanks.

danddmrs · Post by **danddmrs** » Tue Nov 16, 2010 3:37 pm

What about loading the Patterns to a hashed file with a dummy key. Then using a Universe Stage and Multi-row lookup with Stage Variables to exit when a match is found? That way the evaluation would occur pattern by pattern.

10M rows, 200 patterns. A performance problem in the making?

ray.wurlod · Post by **ray.wurlod** » Tue Nov 16, 2010 7:31 pm

How/where are you calling this function? That is, where are you setting its arguments' values?

danddmrs · Post by **danddmrs** » Wed Nov 17, 2010 7:26 am

ODBC --> Transformer --> SeqFile

In the Derivation for Stage Variable svNoMatchFound, inLink.Folder_ID Matches jprFormatMasks
Constraint for the output is svNoMatchFound = 1

ray.wurlod · Post by **ray.wurlod** » Wed Nov 17, 2010 1:44 pm

Now show the expression used to generate the value for the job parameter jprFormatMasks. I presume this is done in a sequence.

Also show us the entry from the job log showing the value passed to the job via that parameter.

danddmrs · Post by **danddmrs** » Thu Nov 18, 2010 9:07 am

Sample of Business Masks input
nnnnnnnnnnC
UAFAnnnnnnnnn
UBFEMnnnnnnnn
A1010nnnnnnnn
A1000Pnnnnnnn
00040lnnnnG
UAFGRPnnnnnn-nnnnnnnnn
A10023nnnnnnn
nnaaaaaaaaD
A111109DED0NE

Routine to convert

Code: Select all

      vBusMask = Trim(Arg1," ","T")
      vSub = 1
      vNodeType1 =' '
      vNode1 = ''
      vNodeType2 =' '
      vNode2 = ''
      vNodeType3 =' '
      vNode3 = ''
      vNodeType4 =' '
      vNode4 = ''
      vNodeType5 =' '
      vNode5 = ''
      vFinal1 = ''
      vFinal2 = ''
      vFinal3 = ''
      vFinal4 = ''
      vFinal5 = ''
      If vBusMask[vSub,1] = ''
      Then
         vNodeType1 = ' '
      End
      Else
         If vBusMask[vSub,1] = 'n'
         Then
            vNodeType1 = 'N'
            Loop
               vNode1 = vNode1:vBusMask[vSub,1]
               vSub = vSub + 1
            While vBusMask[vSub,1] = 'n' and vSub < 30
            Repeat
         End
         Else
            If vBusMask[vSub,1] = 'o'
            Then
               vNodeType1 = 'Nv'
               Loop
                  vNode1 = vNode1:vBusMask[vSub,1]
                  vSub = vSub + 1
               While vBusMask[vSub,1] = 'o' and vSub < 30
               Repeat
            End
            Else
               If vBusMask[vSub,1] = 'l'
               Then
                  vNodeType1 = 'A'
                  Loop
                     vNode1 = vNode1:vBusMask[vSub,1]
                     vSub = vSub + 1
                  While vBusMask[vSub,1] = 'l' and vSub < 30
                  Repeat
               End
               Else
                  If vBusMask[vSub,1] = 'm'
                  Then
                     vNodeType1 = 'Av'
                     Loop
                        vNode1 = vNode1:vBusMask[vSub,1]
                        vSub = vSub + 1
                     While vBusMask[vSub,1] = 'm' and vSub < 30
                     Repeat
                  End
                  Else
                     If vBusMask[vSub,1] = 'a'
                     Then
                        vNodeType1 = 'X'
                        Loop
                           vNode1 = vNode1:vBusMask[vSub,1]
                           vSub = vSub + 1
                        While vBusMask[vSub,1] = 'a' and vSub < 30
                        Repeat
                     End
                     Else
                        If vBusMask[vSub,1] = 'b'
                        Then
                           vNodeType1 = 'Xv'
                           Loop
                              vNode1 = vNode1:vBusMask[vSub,1]
                              vSub = vSub + 1
                           While vBusMask[vSub,1] = 'b' and vSub < 30
                           Repeat
                        End
                        Else

                           If vBusMask[vSub,1] = Upcase(vBusMask[vSub,1])
                           Then
                              vNodeType1 = 'U'
                              Loop
                                 vNode1 = vNode1:vBusMask[vSub,1]
                                 vSub = vSub + 1
                              While vBusMask[vSub,1] = Upcase(vBusMask[vSub,1]) and vSub < 30
                              Repeat
                           End
                        End
                     End
                  End
               End
            End
         End
      End
      If vBusMask[vSub,1] = ''
      Then
         vNodeType2 = ' '
      End
      Else
         If vBusMask[vSub,1] = 'n'
         Then
            vNodeType2 = 'N'
            Loop
               vNode2 = vNode2:vBusMask[vSub,1]
               vSub = vSub + 1
            While vBusMask[vSub,1] = 'n' and vSub < 30
            Repeat
         End
         Else
            If vBusMask[vSub,1] = 'o'
            Then
               vNodeType2 = 'Nv'
               Loop
                  vNode2 = vNode2:vBusMask[vSub,1]
                  vSub = vSub + 1
               While vBusMask[vSub,1] = 'o' and vSub < 30
               Repeat
            End
            Else
               If vBusMask[vSub,1] = 'l'
               Then
                  vNodeType2 = 'A'
                  Loop
                     vNode2 = vNode2:vBusMask[vSub,1]
                     vSub = vSub + 1
                  While vBusMask[vSub,1] = 'l' and vSub < 30
                  Repeat
               End
               Else
                  If vBusMask[vSub,1] = 'm'
                  Then
                     vNodeType2 = 'Av'
                     Loop
                        vNode2 = vNode2:vBusMask[vSub,1]
                        vSub = vSub + 1
                     While vBusMask[vSub,1] = 'm' and vSub < 30
                     Repeat
                  End
                  Else
                     If vBusMask[vSub,1] = 'a'
                     Then
                        vNodeType2 = 'X'
                        Loop
                           vNode2 = vNode2:vBusMask[vSub,1]
                           vSub = vSub + 1
                        While vBusMask[vSub,1] = 'a' and vSub < 30
                        Repeat
                     End
                     Else
                        If vBusMask[vSub,1] = 'b'
                        Then
                           vNodeType2 = 'Xv'
                           Loop
                              vNode2 = vNode2:vBusMask[vSub,1]
                              vSub = vSub + 1
                           While vBusMask[vSub,1] = 'b' and vSub < 30
                           Repeat
                        End
                        Else

                           If vBusMask[vSub,1] = Upcase(vBusMask[vSub,1])
                           Then
                              vNodeType2 = 'U'
                              Loop
                                 vNode2 = vNode2:vBusMask[vSub,1]
                                 vSub = vSub + 1
                              While vBusMask[vSub,1] = Upcase(vBusMask[vSub,1]) and vSub < 30
                              Repeat
                           End
                        End
                     End
                  End
               End
            End
         End
      End
      If vBusMask[vSub,1] = ''
      Then
         vNodeType3 = ' '
      End
      Else

         If vBusMask[vSub,1] = 'n'
         Then
            vNodeType3 = 'N'
            Loop
               vNode3 = vNode3:vBusMask[vSub,1]
               vSub = vSub + 1
            While vBusMask[vSub,1] = 'n' and vSub < 30
            Repeat
         End
         Else
            If vBusMask[vSub,1] = 'o'
            Then
               vNodeType3 = 'Nv'
               Loop
                  vNode3 = vNode3:vBusMask[vSub,1]
                  vSub = vSub + 1
               While vBusMask[vSub,1] = 'o' and vSub < 30
               Repeat
            End
            Else
               If vBusMask[vSub,1] = 'l'
               Then
                  vNodeType3 = 'A'
                  Loop
                     vNode3 = vNode3:vBusMask[vSub,1]
                     vSub = vSub + 1
                  While vBusMask[vSub,1] = 'l' and vSub < 30
                  Repeat
               End
               Else
                  If vBusMask[vSub,1] = 'm'
                  Then
                     vNodeType3 = 'Av'
                     Loop
                        vNode3 = vNode3:vBusMask[vSub,1]
                        vSub = vSub + 1
                     While vBusMask[vSub,1] = 'm' and vSub < 30
                     Repeat
                  End
                  Else
                     If vBusMask[vSub,1] = 'a'
                     Then
                        vNodeType3 = 'X'
                        Loop
                           vNode3 = vNode3:vBusMask[vSub,1]
                           vSub = vSub + 1
                        While vBusMask[vSub,1] = 'a' and vSub < 30
                        Repeat
                     End
                     Else
                        If vBusMask[vSub,1] = 'b'
                        Then
                           vNodeType3 = 'Xv'
                           Loop
                              vNode3 = vNode3:vBusMask[vSub,1]
                              vSub = vSub + 1
                           While vBusMask[vSub,1] = 'b' and vSub < 30
                           Repeat
                        End
                        Else

                           If vBusMask[vSub,1] = Upcase(vBusMask[vSub,1])
                           Then
                              vNodeType3 = 'U'
                              Loop
                                 vNode3 = vNode3:vBusMask[vSub,1]
                                 vSub = vSub + 1
                              While vBusMask[vSub,1] = Upcase(vBusMask[vSub,1]) and vSub < 30
                              Repeat
                           End
                        End
                     End
                  End
               End
            End
         End
      End
      If vBusMask[vSub,1] = ''
      Then
         vNodeType4 = ' '
      End
      Else

         If vBusMask[vSub,1] = 'n'
         Then
            vNodeType4 = 'N'
            Loop
               vNode4 = vNode4:vBusMask[vSub,1]
               vSub = vSub + 1
            While vBusMask[vSub,1] = 'n' and vSub < 30
            Repeat
         End
         Else
            If vBusMask[vSub,1] = 'o'
            Then
               vNodeType4 = 'Nv'
               Loop
                  vNode4 = vNode4:vBusMask[vSub,1]
                  vSub = vSub + 1
               While vBusMask[vSub,1] = 'o' and vSub < 30
               Repeat
            End
            Else
               If vBusMask[vSub,1] = 'l'
               Then
                  vNodeType4 = 'A'
                  Loop
                     vNode4 = vNode4:vBusMask[vSub,1]
                     vSub = vSub + 1
                  While vBusMask[vSub,1] = 'l' and vSub < 30
                  Repeat
               End
               Else
                  If vBusMask[vSub,1] = 'm'
                  Then
                     vNodeType4 = 'Av'
                     Loop
                        vNode4 = vNode4:vBusMask[vSub,1]
                        vSub = vSub + 1
                     While vBusMask[vSub,1] = 'm' and vSub < 30
                     Repeat
                  End
                  Else
                     If vBusMask[vSub,1] = 'a'
                     Then
                        vNodeType4 = 'X'
                        Loop
                           vNode4 = vNode4:vBusMask[vSub,1]
                           vSub = vSub + 1
                        While vBusMask[vSub,1] = 'a' and vSub < 30
                        Repeat
                     End
                     Else
                        If vBusMask[vSub,1] = 'b'
                        Then
                           vNodeType4 = 'Xv'
                           Loop
                              vNode4 = vNode4:vBusMask[vSub,1]
                              vSub = vSub + 1
                           While vBusMask[vSub,1] = 'b' and vSub < 30
                           Repeat
                        End
                        Else

                           If vBusMask[vSub,1] = Upcase(vBusMask[vSub,1])
                           Then
                              vNodeType4 = 'U'
                              Loop
                                 vNode4 = vNode4:vBusMask[vSub,1]
                                 vSub = vSub + 1
                              While vBusMask[vSub,1] = Upcase(vBusMask[vSub,1]) and vSub < 30
                              Repeat
                           End
                        End
                     End
                  End
               End
            End
         End
      End

      If vBusMask[vSub,1] = ''
      Then
         vNodeType5 = ' '
      End
      Else
         If vBusMask[vSub,1] = 'n'
         Then
            vNodeType5 = 'N'
            Loop
               vNode5 = vNode5:vBusMask[vSub,1]
               vSub = vSub + 1
            While vBusMask[vSub,1] = 'n' and vSub < 30
            Repeat
         End
         Else
            If vBusMask[vSub,1] = 'o'
            Then
               vNodeType5 = 'Nv'
               Loop
                  vNode5 = vNode5:vBusMask[vSub,1]
                  vSub = vSub + 1
               While vBusMask[vSub,1] = 'o' and vSub < 30
               Repeat
            End
            Else
               If vBusMask[vSub,1] = 'l'
               Then
                  vNodeType5 = 'A'
                  Loop
                     vNode5 = vNode5:vBusMask[vSub,1]
                     vSub = vSub + 1
                  While vBusMask[vSub,1] = 'l' and vSub < 30
                  Repeat
               End
               Else
                  If vBusMask[vSub,1] = 'm'
                  Then
                     vNodeType5 = 'Av'
                     Loop
                        vNode5 = vNode5:vBusMask[vSub,1]
                        vSub = vSub + 1
                     While vBusMask[vSub,1] = 'm' and vSub < 30
                     Repeat
                  End
                  Else
                     If vBusMask[vSub,1] = 'a'
                     Then
                        vNodeType5 = 'X'
                        Loop
                           vNode5 = vNode5:vBusMask[vSub,1]
                           vSub = vSub + 1
                        While vBusMask[vSub,1] = 'a' and vSub < 30
                        Repeat
                     End
                     Else
                        If vBusMask[vSub,1] = 'b'
                        Then
                           vNodeType5 = 'Xv'
                           Loop
                              vNode5 = vNode5:vBusMask[vSub,1]
                              vSub = vSub + 1
                           While vBusMask[vSub,1] = 'b' and vSub < 30
                           Repeat
                        End
                        Else

                           If vBusMask[vSub,1] = Upcase(vBusMask[vSub,1])
                           Then
                              vNodeType5 = 'U'
                              Loop
                                 vNode5 = vNode5:vBusMask[vSub,1]
                                 vSub = vSub + 1
                              While vBusMask[vSub,1] = Upcase(vBusMask[vSub,1]) and vSub < 30
                              Repeat
                           End
                        End
                     End
                  End
               End
            End
         End
      End
      Begin Case
         Case vNodeType1 = 'U'
            vFinal1 = "'":vNode1:"'"
         Case vNodeType1 = 'N'
            vFinal1 = Len(vNode1):vNodeType1
         Case vNodeType1 = 'Nv'
            vFinal1 = Len(vNode1):vNodeType1:':@VM:'
         Case vNodeType1 = 'A'
            vFinal1 = Len(vNode1):vNodeType1
         Case vNodeType1 = 'Av'
            vFinal1 = Len(vNode1):vNodeType1:':@VM:'
         Case vNodeType1 = 'X'
            vFinal1 = Len(vNode1):vNodeType1
         Case vNodeType1 = 'Xv'
            vFinal1 = Len(vNode1):vNodeType1:':@VM:'
         Case vNodeType1 = ' '
            vFinal1 = ''
      End Case

      Begin Case
         Case vNodeType2 = 'U'
            vFinal2 = "'":vNode2:"'"
         Case vNodeType2 = 'N'
            vFinal2 = Len(vNode2):vNodeType2
         Case vNodeType2 = 'Nv'
            vFinal2 = Len(vNode2):vNodeType2[1,1]:':@VM:'
         Case vNodeType2 = 'A'
            vFinal2 = Len(vNode2):vNodeType2
         Case vNodeType2 = 'Av'
            vFinal2 = Len(vNode2):vNodeType2[1,1]:':@VM:'
         Case vNodeType2 = 'X'
            vFinal2 = Len(vNode2):vNodeType2
         Case vNodeType2 = 'Xv'
            vFinal2 = Len(vNode2):vNodeType2[1,1]:':@VM:'
         Case vNodeType2 = ' '
            vFinal2 = ''
      End Case

      Begin Case
         Case vNodeType3 = 'U'
            vFinal3 = "'":vNode3:"'"
         Case vNodeType3 = 'N'
            vFinal3 = Len(vNode3):vNodeType3
         Case vNodeType3 = 'Nv'
            vFinal3 = Len(vNode3):vNodeType3[1,1]:':@VM:'
         Case vNodeType3 = 'A'
            vFinal3 = Len(vNode3):vNodeType3
         Case vNodeType3 = 'Av'
            vFinal3 = Len(vNode3):vNodeType3[1,1]:':@VM:'
         Case vNodeType3 = ' '
            vFinal3 = ''
      End Case

      Begin Case
         Case vNodeType4 = 'U'
            vFinal4 = "'":vNode4:"'"
         Case vNodeType4 = 'N'
            vFinal4 = Len(vNode4):vNodeType4
         Case vNodeType4 = 'Nv'
            vFinal4 = Len(vNode4):vNodeType4[1,1]:':@VM:'
         Case vNodeType4 = 'A'
            vFinal4 = Len(vNode4):vNodeType4
         Case vNodeType4 = 'Av'
            vFinal4 = Len(vNode4):vNodeType4[1,1]:':@VM:'
         Case vNodeType4 = ' '
            vFinal4 = ''
      End Case

      Begin Case
         Case vNodeType5 = 'U'
            vFinal5 = "'":vNode5:"'"
         Case vNodeType5 = 'N'
            vFinal5 = Len(vNode5):vNodeType5
         Case vNodeType5 = 'Nv'
            vFinal5 = Len(vNode5):vNodeType5[1,1]:':@VM:'
         Case vNodeType5 = 'A'
            vFinal5 = Len(vNode5):vNodeType5
         Case vNodeType5 = 'Av'
            vFinal5 = Len(vNode5):vNodeType5[1,1]:':@VM:'
         Case vNodeType5 = ' '
            vFinal5 = ' '
      End Case

      Ans = vFinal1:vFinal2:vFinal3:vFinal4:vFinal5

Resulting Patterns
0:Char(253):"10N'C'":Char(253):"'UAFA'9N":Char(253):"'UBFEM'8N":Char(253):"'A1010'8N":Char(253):"'A1000P'7N":Char(253):"'00040'1A4N'G'":Char(253):"'UAFGRP'6N'-'9N":Char(253):"'A10023'7N":Char(253):"2N8X'D'":Char(253):"'A111109DED0NE'"

From the log of the job using the Patterns
Starting Job jsFold_ID_Validation.
jprFormatMasks = 0:Char(253):"10N'C'":Char(253):"'UAFA'9N":Char(253):"'UBFEM'8N":Char(253):"'A1010'8N":Char(253):"'A1000P'7N":Char(253):"'00040'1A4N'G'":Char(253):"'UAFGRP'6N'-'9N":Char(253):"'A10023'7N":Char(253):"2N8X'D'":Char(253):"'A111109DED0NE'"
jprDirectory = n:\imagetemp

If the value from the log is cut and pasted into the job with the Matches function it runs successfully.

Thanks.

ray.wurlod · Post by **ray.wurlod** » Thu Nov 18, 2010 4:39 pm

Yes but, as noted, you can not use Char(any) in the value of a routine argument. This needs to have been pre-evaluated. Char(254) is pre-evaluated into the @FM system variable and Char(253) is pre-evaluated into the @VM system variable.

But it's more difficult than that. You can not use any form of expression at all in a routine argument. The entire expression must be pre-evaluated into a string and that string passed to the routine as its argument.

danddmrs · Post by **danddmrs** » Fri Nov 19, 2010 2:13 pm

I don't know what you mean by pre-evaluated into a string.

How would I pre-evaluate 0:Char(253):"'A1000P'7N":Char(253):"'U20'9N" into a string to pass as the argument?

Thanks.

ray.wurlod · Post by **ray.wurlod** » Fri Nov 19, 2010 2:30 pm

If it were a job sequence you could use exactly that expression.

In dsjob you have to do what you can do in UNIX. Something like

Code: Select all

0\xFD\"\'A100P\'\"\xFD\"\'U20\'\"