CRC32 in PX
Moderators: chulett, rschirm, roy
CRC32 in PX
Hi,
Has anyone tried using CRC32 in Parallel version by wrapping the code in a BuildOp since we do have this function in PX (only in server edition)?
NV
Has anyone tried using CRC32 in Parallel version by wrapping the code in a BuildOp since we do have this function in PX (only in server edition)?
NV
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
The data volumes I would expect to process with a parallel job would make the risks involved in using CRC32 too great.
CRC32 has about a one in four million chance of generating false positives.
Parallel jobs have their own, quite efficient, change detection mechanisms; the Difference stage, the Compare stage and the Change Capture stage.
CRC32 has about a one in four million chance of generating false positives.
Parallel jobs have their own, quite efficient, change detection mechanisms; the Difference stage, the Compare stage and the Change Capture stage.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
When you read the three pertinent chapters in the Parallel Job Developer's Guide what did you discern the differences to be?
I would be happy to read these chapters for you, in return for my usual fee.
I would be happy to read these chapters for you, in return for my usual fee.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
You haven't seen my fees!snt_ds wrote:I would be more than happy to pay for it!![/b]
Learning where to look is half the battle. Reading manuals and readme files is how I keep up to date.
One day I'll retire, or even take a vacation, and then you'll all have to stand on your own just that little bit more.
I'm very pleased to see the growth in this community of folks who are posting answers, sometimes very good answers. It means I can retire with no problems for my conscience. One day.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Craig, I seriously think that needs some transformations.chulett wrote:... post level high water mark ...
Ray, thanks for letting us know that you won't be there someday and that we need to come of age soon...
By the way, don't get scared by Ray's rates, call me for reading assignments. I sure can READ
It took me fifteen years to discover I had no talent for ETL, but I couldn't give it up because by that time I was too famous.
-
- Participant
- Posts: 3593
- Joined: Thu Jan 23, 2003 5:25 pm
- Location: Australia, Melbourne
- Contact:
I only ever use the Change Data Capture stage and ignore the other two stages. I think the CDC stage was developed as an improvement to the Difference stage though I don't know where I originally read this.
Certus Solutions
Blog: Tooling Around in the InfoSphere
Twitter: @vmcburney
LinkedIn:Vincent McBurney LinkedIn
Blog: Tooling Around in the InfoSphere
Twitter: @vmcburney
LinkedIn:Vincent McBurney LinkedIn
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
It's mainly a functional question. Do you need direction of difference? Then you use the Compare stage. Do you need the source columns separated (Difference) or as subrecords (Change Capture)?
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
ChageCapture can yield only after data set. Difference stage uses the diff operator, which internally uses the diff command at OS level, which does the check on before and after dataset record by records and can result before or after dataset.
Impossible doesn't mean 'it is not possible' actually means... 'NOBODY HAS DONE IT SO FAR'