CRC32 in PX

nvalia · Post by **nvalia** » Tue Jan 30, 2007 5:27 pm

Hi,

Has anyone tried using CRC32 in Parallel version by wrapping the code in a BuildOp since we do have this function in PX (only in server edition)?

NV

kumar_s · Post by **kumar_s** » Tue Jan 30, 2007 6:03 pm

viewtopic.php?t=99274&postdays=0&postor ... 32&start=0

You can just have look in to this thread.

ray.wurlod · Post by **ray.wurlod** » Tue Jan 30, 2007 6:33 pm

The data volumes I would expect to process with a parallel job would make the risks involved in using CRC32 too great.

CRC32 has about a one in four million chance of generating false positives.

Parallel jobs have their own, quite efficient, change detection mechanisms; the Difference stage, the Compare stage and the Change Capture stage.

snt_ds · Post by **snt_ds** » Tue Jan 30, 2007 7:37 pm

Thanks.

What are the major differnces bewteen the 3 parallel job stages used for data comparision?

ray.wurlod · Post by **ray.wurlod** » Tue Jan 30, 2007 7:50 pm

When you read the three pertinent chapters in the Parallel Job Developer's Guide what did you discern the differences to be?

I would be happy to read these chapters for you, in return for my usual fee.

snt_ds · Post by **snt_ds** » Tue Jan 30, 2007 7:56 pm

I would be more than happy to pay for it!!

Jokes apart, U have been of great help thru ur replies.
Thanks Ray.[/b]

kduke · Post by **kduke** » Tue Jan 30, 2007 8:19 pm

What about viewtopic.php?t=91683&highlight=crc32

ray.wurlod · Post by **ray.wurlod** » Tue Jan 30, 2007 11:59 pm

snt_ds wrote:I would be more than happy to pay for it!![/b]

You haven't seen my fees!

Learning where to look is half the battle. Reading manuals and readme files is how I keep up to date.

One day I'll retire, or even take a vacation, and then you'll all have to stand on your own just that little bit more.

I'm very pleased to see the growth in this community of folks who are posting answers, sometimes very good answers. It means I can retire with no problems for my conscience. One day.

DSguru2B · Post by **DSguru2B** » Wed Jan 31, 2007 7:46 am

ray.wurlod wrote:
I'm very pleased to see the growth in this community of folks who are posting answers, sometimes very good answers. It means I can retire with no problems for my conscience. One day.

chulett · Post by **chulett** » Wed Jan 31, 2007 7:50 am

And when that happens, it will still take years to catch up to whatever post level high water mark Ray leaves behind.

kumar_s · Post by **kumar_s** » Thu Feb 01, 2007 5:49 am

Wish that 'One day' is after several decades.

sud · Post by **sud** » Thu Feb 01, 2007 7:29 am

chulett wrote:... post level high water mark ...

Craig, I seriously think that needs some transformations.

Ray, thanks for letting us know that you won't be there someday and that we need to come of age soon...

By the way, don't get scared by Ray's rates, call me for reading assignments. I sure can READ

vmcburney · Post by **vmcburney** » Thu Feb 01, 2007 2:13 pm

I only ever use the Change Data Capture stage and ignore the other two stages. I think the CDC stage was developed as an improvement to the Difference stage though I don't know where I originally read this.

ray.wurlod · Post by **ray.wurlod** » Thu Feb 01, 2007 4:51 pm

It's mainly a functional question. Do you need direction of difference? Then you use the Compare stage. Do you need the source columns separated (Difference) or as subrecords (Change Capture)?

kumar_s · Post by **kumar_s** » Thu Feb 01, 2007 6:29 pm

ChageCapture can yield only after data set. Difference stage uses the diff operator, which internally uses the diff command at OS level, which does the check on before and after dataset record by records and can result before or after dataset.