Hi,
How to implement the Incremental Aggregation Loading in Server Jobs?
Any assistance can be appreciated.
thanks in advance
Incremental Aggregation loading
Moderators: chulett, rschirm, roy
Ravij,
it is very important when posing sweeping questions like this to explain what you've already tried, otherwise we will assume that you haven't tried anything and are just looking for an easy answer.
The concept of incrementatl aggregation is a relatively new one. Much, if not most, of the code for such an incremental aggregation needs to be tailored to the specific environment and there is no single answer. The core is that you need to know which data you've already processed; and determining this is different when you have direct database access or just see export flat files. The approach is almost the same as for CDC.
it is very important when posing sweeping questions like this to explain what you've already tried, otherwise we will assume that you haven't tried anything and are just looking for an easy answer.
The concept of incrementatl aggregation is a relatively new one. Much, if not most, of the code for such an incremental aggregation needs to be tailored to the specific environment and there is no single answer. The core is that you need to know which data you've already processed; and determining this is different when you have direct database access or just see export flat files. The approach is almost the same as for CDC.
</a>
HI,
As Arnd specifed you need to be specific.
The way to do the incremental aggregation is only possible by determining the incremetal load (data). So basically you need to do a change detection with the current and previous run file.
You have built in stages available to assist you.
Chage capture/Chagne apply.
Based on the captured code, you can aggregate the data.
-Kumar
As Arnd specifed you need to be specific.
The way to do the incremental aggregation is only possible by determining the incremetal load (data). So basically you need to do a change detection with the current and previous run file.
You have built in stages available to assist you.
Chage capture/Chagne apply.
Based on the captured code, you can aggregate the data.
-Kumar
-
Sreenivasulu
- Premium Member
- Posts: 892
- Joined: Thu Oct 16, 2003 5:18 am
Kumar,
I believe the question posed by ravi is with respect to server jobs and not parallel jobs. The change capture stage and change apply stage referred are present in PX.
Ravij,
Incremental aggregation can be done using a date window for extraction.
You load only a particular window's data from the source systems and
then store in a different partition (one partition for each window).
Beofre storing the in the partition do the aggregation as usual and it would
be an incremental one since the extraction was for a small window.
Regards
Sreeni
I believe the question posed by ravi is with respect to server jobs and not parallel jobs. The change capture stage and change apply stage referred are present in PX.
Ravij,
Incremental aggregation can be done using a date window for extraction.
You load only a particular window's data from the source systems and
then store in a different partition (one partition for each window).
Beofre storing the in the partition do the aggregation as usual and it would
be an incremental one since the extraction was for a small window.
Regards
Sreeni
-
ray.wurlod
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact: