batching the records

[srai]

Hi All,

We are in process of building a DataStage job which would call a ALSB web service to update the salesforce instance.
Potentially Salesforce.com would accept 200 records in a single request.
Is this something that can easily be accomplished using DataStage? For instance, if the process extracts 500 records, we would need to invoke the ALSB service 3 times, sending requests containing 200, 200 and 100 records.


Thanks
Santosh

[Pagadrai]

Hi All,

We are in process of building a DataStage job which would call a ALSB web service to update the salesforce instance.
Potentially Salesforce.com would accept 200 records in a single request.
Is this something that can easily be accomplished using DataStage? For instance, if the process extracts 500 records, we would need to invoke the ALSB service 3 times, sending requests containing 200, 200 and 100 records.


Thanks
Santosh

Hi,
I havent worked with a webservice in datastage, but here is what you can try:
you have option to call DS job in a loop either using sequence or using server routine. For a sequence, you might have to predetermine how many times you are calling the job.
But one thing am not clear is that you also need to have a mechanism wherein you mark the request records as processed or unprocessed.

[srai]

Hi All,

We are in process of building a DataStage job which would call a ALSB web service to update the salesforce instance.
Potentially Salesforce.com would accept 200 records in a single request.
Is this something that can easily be accomplished using DataStage? For instance, if the process extracts 500 records, we would need to invoke the ALSB service 3 times, sending requests containing 200, 200 and 100 records.


Thanks
Santosh

Hi,
I havent worked with a webservice in datastage, but here is what you can try:
you have option to call DS job in a loop either using sequence or using server routine. For a sequence, you might have to predetermine how many times you are calling the job.
But one thing am not clear is that you also need to have a mechanism wherein you mark the request records as processed or unprocessed.

Hi,

But here the no of records coming from source is dynamic.
Its not fixed. So no of records coming from source would vary for each run. My problem is how to decide about the loop like how many times we have to run the job in loop.

Thanks
Santosh

[Pagadrai]

Hi Santosh
What is your source ?
Is it possible for you to get the count of source records before you call the DS job?

[srai]

Hi Santosh
What is your source ?
Is it possible for you to get the count of source records before you call the DS job?

Hi my source is Oracle. We can get count of record before every run.

[Pagadrai]

Hi Santosh,
In that case, you can use a routine to decide how many times you want to call the particular DS job.
Say if u have 500 records and you want the job to handle 200 at a time,
the routine can decide that u have to call the job 3 times and then invoke it in a loop.
( but am assuming here that the DS job can pick up only 200 at one time and after the records are processed you flag them appropriately so that they are not picked up again )

either you can call the job from routine itself or you can just get the count and handle the loop activity in the sequence.

* One more option you have is to have the whole looping logic done in a script instead of a routine. If you are comfortable using scripts, you can try this.

Let me know if you have questions.

[ray.wurlod]

U did not ask the question. U is one of our members. It's a Burmese name.

The second person personal pronoun in English is spelled "you", not "u".

Please use a professional standard of English on DSXchange, not least because "proper" English is hard enough for those whose first language is not English, and also because DSXchange is not an SMS device.

[Pagadrai]

U did not ask the question. U is one of our members. It's a Burmese name.

The second person personal pronoun in English is spelled "you", not "u".

Please use a professional standard of English on DSXchange, not least because "proper" English is hard enough for those whose first language is not English, and also because DSXchange is not an SMS device.

Hi Ray,
I will make a note of it.
Thank you.