delta processing
Moderators: chulett, rschirm, roy
delta processing
I NEED TO KNOW HOW DO I DO THIS
NEED TO DEVELOP A DELTA PROCESS THAT WILL DETERMINE THE DIFFRENCE BY COMPARING THE DATA FROM THE FILE RECIEVED FROM ONE TABLE WHAT IS CURRENTLY STORED IN ANOTHER TABLE. ALL THE ATTRIBUTES RECIEVED SHOULD BE COMPARED.
THE UNMATCHED RECORD SHOULD BE UPDATED OR UPSERTED INTO THE TABLE AND THE DATE ATRIBUTE IS SET TO SYSTEM DATE.
PLEASE I AM A IN A HURRY.
CAN ANYONE TELL ME HOW DO I DO IT.
NEED TO DEVELOP A DELTA PROCESS THAT WILL DETERMINE THE DIFFRENCE BY COMPARING THE DATA FROM THE FILE RECIEVED FROM ONE TABLE WHAT IS CURRENTLY STORED IN ANOTHER TABLE. ALL THE ATTRIBUTES RECIEVED SHOULD BE COMPARED.
THE UNMATCHED RECORD SHOULD BE UPDATED OR UPSERTED INTO THE TABLE AND THE DATE ATRIBUTE IS SET TO SYSTEM DATE.
PLEASE I AM A IN A HURRY.
CAN ANYONE TELL ME HOW DO I DO IT.
First things First.
1. Post in the right forum. There is no way we would be able to tell which job you are trying to develop..server or parallel and what your target databases are etc etc. and the answer depends totally on it.
Give a nice detailed description of your Job and your source and target. I don't understand when you say data coming from a file received from one table. Is your source a table or a file?
1. Post in the right forum. There is no way we would be able to tell which job you are trying to develop..server or parallel and what your target databases are etc etc. and the answer depends totally on it.
Give a nice detailed description of your Job and your source and target. I don't understand when you say data coming from a file received from one table. Is your source a table or a file?
Kris
Where's the "Any" key?-Homer Simpson
Where's the "Any" key?-Homer Simpson
A couple of etiquettes that need to be kept in mind.
First of all you have posted in the wrong forum. We have absolutely no idea what job type it is, what version of datastage you are using and what is your OS type. Answer to your query will vary depending upon the above.
Secondly, please do not use all caps as it is a sign of shouting, as far as i remember.
Thirdly, please do not indicate, in your post, in any way, that your query is urgent, or you are in a hurry. No one does urgent here. This is a voluntary site and posters answer at their convenience.
I hope i did not sound rude, and conveyed my point gracefully.
Now as far as your query is concerned, you can used a hashed file lookup by declaring all columns as keys and detect the change, if your job is server type. If PX, then the CDC stage would be helpful. You can also load your data into a temp table and run sql to detect change.
Regards,
First of all you have posted in the wrong forum. We have absolutely no idea what job type it is, what version of datastage you are using and what is your OS type. Answer to your query will vary depending upon the above.
Secondly, please do not use all caps as it is a sign of shouting, as far as i remember.
Thirdly, please do not indicate, in your post, in any way, that your query is urgent, or you are in a hurry. No one does urgent here. This is a voluntary site and posters answer at their convenience.
I hope i did not sound rude, and conveyed my point gracefully.
Now as far as your query is concerned, you can used a hashed file lookup by declaring all columns as keys and detect the change, if your job is server type. If PX, then the CDC stage would be helpful. You can also load your data into a temp table and run sql to detect change.
Regards,
Creativity is allowing yourself to make mistakes. Art is knowing which ones to keep.
-
- Premium Member
- Posts: 62
- Joined: Tue Jun 14, 2005 7:17 pm
- Location: Australia
- Contact:
There are ways to compare the current and previous data. ChanegCapture stage, or Join-Transformer Combination will help you.
But what is the requirement after getting the detail about, update and new.
Is it you want to update the existing record with the new records or you need to insert with an indicator saying it has been update. Basically what type of SCD you what to implement?
But what is the requirement after getting the detail about, update and new.
Is it you want to update the existing record with the new records or you need to insert with an indicator saying it has been update. Basically what type of SCD you what to implement?
Impossible doesn't mean 'it is not possible' actually means... 'NOBODY HAS DONE IT SO FAR'
-
- Premium Member
- Posts: 62
- Joined: Tue Jun 14, 2005 7:17 pm
- Location: Australia
- Contact:
I'm more concerned about why the CDC cannot read the Teradata source. At it's basic level, shouldn't the job design be:
(Of course, once the CDC is complete and records are tagged as Insert or Update they can be sent to load ready datasets, etc. And, the CDC stage can be replaced by a Join/Transformer combination as kumar_s has noted.)
Code: Select all
Teradata
Existing
Records
|
V
Teradata Change Teradata
New --> Data --> Insert/
Records Capture Update
u r absolutely right doma thats what i need to do, although i tried to do so with cdc but i guess the metadata is not matching.
so what i plan to do is get each teradata into a lookup and then i can compare in a transformer and do inserts and update. i still didnt do it. but thats what i planned.
so what i plan to do is get each teradata into a lookup and then i can compare in a transformer and do inserts and update. i still didnt do it. but thats what i planned.
Daddy Doma wrote:I'm more concerned about why the CDC cannot read the Teradata source. At it's basic level, shouldn't the job design be:
(Of course, once the CDC is complete and records are tagged as Insert or Update they can be sent to load ready datasets, etc. And, the CDC stage can be replaced by a Join/Transformer combination as kumar_s has noted.)Code: Select all
Teradata Existing Records | V Teradata Change Teradata New --> Data --> Insert/ Records Capture Update