assigning increment number, logic needed

[smohd1338]

Hi,

I have multiple files reading in using file pattern method,along with filename, in the output i am having batch header record(BH) and in that record, the second field should represent the number of the file. what logic shall be written in transformer(4 node transformer),

example, if i have 6 finput iles , then the output should be like this
BH|000001|X|Y|Z
DL|....
DS|...
BT|....
BH|000002|X|Y|Z
DL|...
DS|...
BT|.... like wise for the third file i lll be having 000003 in the BH record,

we also have at output file level File header record(FH) and FileTrailer record(FT) as shown below


FH|A|B
BH|000001|X|Y|Z
DL|....
DS|...
BT|....
BH|000002|X|Y|Z
DL|...
DS|...
BT|....
FT|....

[chulett]

So... out of all that, you just want to know how to count the files? If so, then we're just talking about 'group change detection' where you track the current and previous value of the group key (whatever that is) in stage variables and increment a counter when it changes, otherwise you leave it alone by setting it to its current value. For example:

Code: Select all

svCurrFile = Link.Filename
svCounter = If svCurrFile = svPrevFile then svCounter else svCounter + 1
svPrevfile = svCurrFile

The order is critical as stage variables are evaluated top down, in order. That's what allows one to compare the value on the current row to one you've stashed from the previous row. Initial values are important as well - the counter should be initialized to zero.

You'll also need to worry about partitioning so that all records for a given file end up in the same partition. And you could have issues with the file numbers matching up to the input order... why four nodes if all you are doing is reading and writing files? Save yourself headaches and run this on one node. Or use a Server job.

[smohd1338]

thanks chulett, i think i should give node constraint.