Adding a condition to the job which runs everyday

[shivan]

hi,
I have a job which runs everyday. There are like 60,000 records, so it takes like 15 mins to run. This job has to run everyday. I am thinking if i add a condition to the job like update=update(date,-1) so that it will fetch only those records which are updated and inserted yesterday. So that it doesnt have to fetch 60000 records every day. It can only fetch the new or updated records.
I am not too confident about my idea of doing this way. So i want some advice on this like is this a right way or not?

thanks
shivan

[DeepakCorning]

Do u have a timestamp in the target table??

hi,
I have a job which runs everyday. There are like 60,000 records, so it takes like 15 mins to run. This job has to run everyday. I am thinking if i add a condition to the job like update=update(date,-1) so that it will fetch only those records which are updated and inserted yesterday. So that it doesnt have to fetch 60000 records every day. It can only fetch the new or updated records.
I am not too confident about my idea of doing this way. So i want some advice on this like is this a right way or not?

thanks
shivan

[chucksmith]

Keep the date of your last good run in a file or table. That advantage of the macro DSJobStartTimestamp.

Select all source rows that have an update date >= your last good run date.

When your run finishes, update the last good run date.

This way, if you miss a day, your job still works fine. I call this "self healing".

[shivan]

hi,
the way it is working is like this: I have a update timestamp column. So if there is an update, then it will overwrite the update timestamp column with a current date. My question is that, whether i m doing this right or not. Because i m going against only the update and new columns which may be 1000, instead of running the whole job of 60000 rows everyday. Because generally i saw that , ETL persons run the job everyday. They never apply conditions like this.
correct me if i m wrong.
thanks
shivan

Keep the date of your last good run in a file or table. That advantage of the macro DSJobStartTimestamp.

Select all source rows that have an update date >= your last good run date.

When your run finishes, update the last good run date.

This way, if you miss a day, your job still works fine. I call this "self healing".