Page 1 of 2
CRC32 in PX
Posted: Tue Jan 30, 2007 5:27 pm
by nvalia
Hi,
Has anyone tried using CRC32 in Parallel version by wrapping the code in a BuildOp since we do have this function in PX (only in server edition)?
NV
Posted: Tue Jan 30, 2007 6:03 pm
by kumar_s
Posted: Tue Jan 30, 2007 6:33 pm
by ray.wurlod
The data volumes I would expect to process with a parallel job would make the risks involved in using CRC32 too great.
CRC32 has about a one in four million chance of generating false positives.
Parallel jobs have their own, quite efficient, change detection mechanisms; the Difference stage, the Compare stage and the Change Capture stage.
Posted: Tue Jan 30, 2007 7:37 pm
by snt_ds
Thanks.
What are the major differnces bewteen the 3 parallel job stages used for data comparision?
Posted: Tue Jan 30, 2007 7:50 pm
by ray.wurlod
When you read the three pertinent chapters in the Parallel Job Developer's Guide what did you discern the differences to be?
I would be happy to read these chapters for you, in return for my usual fee.
Posted: Tue Jan 30, 2007 7:56 pm
by snt_ds
I would be more than happy to pay for it!!
Jokes apart, U have been of great help thru ur replies.
Thanks Ray.[/b]
Posted: Tue Jan 30, 2007 8:19 pm
by kduke
Posted: Tue Jan 30, 2007 11:59 pm
by ray.wurlod
snt_ds wrote:I would be more than happy to pay for it!![/b]
You haven't seen my fees!
Learning where to look is half the battle. Reading manuals and readme files is how I keep up to date.
One day I'll retire, or even take a vacation, and then you'll all have to stand on your own just that little bit more.
I'm very pleased to see the growth in this community of folks who are posting answers, sometimes very good answers. It means I can retire with no problems for my conscience. One day.
Posted: Wed Jan 31, 2007 7:46 am
by DSguru2B
ray.wurlod wrote:
I'm very pleased to see the growth in this community of folks who are posting answers, sometimes very good answers. It means I can retire with no problems for my conscience. One day.
Posted: Wed Jan 31, 2007 7:50 am
by chulett
And when that happens, it will still take
years to catch up to whatever post level high water mark Ray leaves behind.
Posted: Thu Feb 01, 2007 5:49 am
by kumar_s
Wish that 'One day' is after several decades.
Posted: Thu Feb 01, 2007 7:29 am
by sud
chulett wrote:... post level high water mark ...
Craig, I seriously think that needs some transformations.
Ray, thanks for letting us know that you won't be there someday and that we need to come of age soon...
By the way, don't get scared by Ray's rates, call me for reading assignments. I sure can READ
Posted: Thu Feb 01, 2007 2:13 pm
by vmcburney
I only ever use the Change Data Capture stage and ignore the other two stages. I think the CDC stage was developed as an improvement to the Difference stage though I don't know where I originally read this.
Posted: Thu Feb 01, 2007 4:51 pm
by ray.wurlod
It's mainly a functional question. Do you need direction of difference? Then you use the Compare stage. Do you need the source columns separated (Difference) or as subrecords (Change Capture)?
Posted: Thu Feb 01, 2007 6:29 pm
by kumar_s
ChageCapture can yield only after data set. Difference stage uses the diff operator, which internally uses the diff command at OS level, which does the check on before and after dataset record by records and can result before or after dataset.