Custom built parallel stages
Posted: Thu Oct 25, 2018 12:01 am
Hi all,
I have come across a custom built parallel stage within our environment that does a better job at MD5 hashing that the built-in Checksum stage because it does not use the trailing pipe character. The latter happens to be a problem for us because there are existing hashed values (produced by SQL) that have not had trailing pipe character when provided as an input.
After some discussions I have found out that custom written MD5 stage have not been used from the beginning because it was written in C++ and as such not an out of box product. Instead, the projects have used hashing command within DB. There was a fear that custom written stage may not work in future versions of DataStage.
Whilst it is a fair comment from the platform owners, my argument is that some of these custom written stages will outlive some (and may have already) some of the built-in stages that have been deprecated over the years.
Does anyone know of any documentation or is happy to provide some sensible arguments that would support the use of custom written hashing stage?
It has been has been thoroughly tested and producing the correct results. Might need to adjust it to produce upper cased hash results (again to match the current outputs produced by SQL) but otherwise the stage works as expected.
It would greatly simplify design of future ETL jobs.
Cheers,
Novak
I have come across a custom built parallel stage within our environment that does a better job at MD5 hashing that the built-in Checksum stage because it does not use the trailing pipe character. The latter happens to be a problem for us because there are existing hashed values (produced by SQL) that have not had trailing pipe character when provided as an input.
After some discussions I have found out that custom written MD5 stage have not been used from the beginning because it was written in C++ and as such not an out of box product. Instead, the projects have used hashing command within DB. There was a fear that custom written stage may not work in future versions of DataStage.
Whilst it is a fair comment from the platform owners, my argument is that some of these custom written stages will outlive some (and may have already) some of the built-in stages that have been deprecated over the years.
Does anyone know of any documentation or is happy to provide some sensible arguments that would support the use of custom written hashing stage?
It has been has been thoroughly tested and producing the correct results. Might need to adjust it to produce upper cased hash results (again to match the current outputs produced by SQL) but otherwise the stage works as expected.
It would greatly simplify design of future ETL jobs.
Cheers,
Novak