Need Advice on Build Stage

zulfi123786 · Post by **zulfi123786** » Mon Apr 21, 2014 5:30 am

Hi,

I have implemented encryption and decryption in parallel routines but the process runs slow as it has to pick up the key from disk also the transformer does not free the memory of returned pointer so the job aborts on very high volume. To enhance the speed I am planning to have a build stage where the per record disk read would be eliminated by a single read before the per record loop.

However there is a small challenge where I would need an expert advice.

1. Using the routines, the user may encrypt multiple columns in one transformer how to allow multiple columns to be encrypted in build stage as the metadata for the input port is static.

2. The encryption routine takes input argument of any datatype and the decryption routine give back exact same datatype and no explicit type conversion are required, how to have this in build stage as the metadata is predefined

Thanks

eostic · Post by **eostic** » Mon Apr 21, 2014 5:50 am

Can you accomplkish what you need to via java? The java integration stage is far more generic in the ability to determine column names at run time.

Alternatively, a complete custom Operator, perhaps.....

Ernie

zulfi123786 · Post by **zulfi123786** » Mon Apr 21, 2014 6:29 am

Hi Ernie,

Thanks for the suggestion, Custom operator approach might be perceivable.

At a high level could you please highlight the steps involved in building the custom operator so that I may get direction to proceed further.

Thanks