Message Digest C code

just4geeks · Post by **just4geeks** » Tue Aug 14, 2007 7:31 am

I am looking for a message digest function in C to be used in a custom stage. One of my jobs involves checking for duplicates in a million record file each record being of varchar(1000) datatype. When I used the remove duplicate stage, the scratch space would become full and the job would abort. I then used a CRC32 C code in a custom stage. It worked fine until I came across two distinct records with same CRC32 code.

I searched for similar posts on message digest and this is the only one that comes close to my issue. viewtopic.php?t=110300&highlight=message+digest

The solution mentioned in the post by Craig Hulett involves using a database table and a generated surrogate key. I was wondering if something could be done on the fly in DataStage only.

Any help will be appreciated. Thanks for your time.

chulett · Post by **chulett** » Tue Aug 14, 2007 8:03 am

Well... nothing about that technique itself requires a database table, just some form of persistent storage. I chose a table because of my need to include my ids in other sql operations. And the scope of the persistence would be whatever was appropriate for you needs - just the life of the process, 'forever', etc.

just4geeks · Post by **just4geeks** » Tue Aug 14, 2007 9:39 am

chulett wrote:Well... nothing about that technique itself requires a database table, just some form of persistent storage. I chose a table because of my need to include my ids in other sql operations. And the scope of the persistence would be whatever was appropriate for you needs - just the life of the process, 'forever', etc.

Thanks Craig for the reply. I was however able to get a working MD5 C code and it worked.

cgarciadesc · Post by **cgarciadesc** » Tue Aug 10, 2010 9:13 am

just4geeks wrote: Thanks Craig for the reply. I was however able to get a working MD5 C code and it worked.

Could you please share your code???

Thanks in advance!