Design for real time reference data update.
Posted: Wed May 25, 2011 1:15 pm
We have created a webservice for customer search (both deterministic and probabilistic matches). The customer will be searched on a reference data of 200 million records.
Currently we have designed the webservice for static reference data which needs to be updated for realtime scenario.
Current design:
Static data is loaded into a lookup fileset and also frequency dataset will be created at the start of the day. We have created one job and exposed as a webservice, which expects an input message for customer and the customer details in the input message are used for searching the data on the lookup file set. This is working as expected.
Next steps:
Now we want to update the reference data in real time. The reference data stored in a database will be provided in a file at the start of the day. After that the real time updates on the database (inserts and updates) are provided through an MQ.
Now we should make the real time updates availble to the webservice. We tried to appened the real time data to lookup file set. But there is no option for append mode. And even if we appened, we cannot update the existing the lookup fileset.
Second option we tried is using a dataset. We were able to append the data in real time but cannot update the dataset.
Third option, we thought is of using a database. But if we use a database it will affect the performance of the webservice as we need to read the entire data into DataStage for probabilistic matching (match specifications in QualityStage).
Could you please provide your inputs on solving this real time update issue.
Please let me know if you need any further information.
Thanks
Raghav
Currently we have designed the webservice for static reference data which needs to be updated for realtime scenario.
Current design:
Static data is loaded into a lookup fileset and also frequency dataset will be created at the start of the day. We have created one job and exposed as a webservice, which expects an input message for customer and the customer details in the input message are used for searching the data on the lookup file set. This is working as expected.
Next steps:
Now we want to update the reference data in real time. The reference data stored in a database will be provided in a file at the start of the day. After that the real time updates on the database (inserts and updates) are provided through an MQ.
Now we should make the real time updates availble to the webservice. We tried to appened the real time data to lookup file set. But there is no option for append mode. And even if we appened, we cannot update the existing the lookup fileset.
Second option we tried is using a dataset. We were able to append the data in real time but cannot update the dataset.
Third option, we thought is of using a database. But if we use a database it will affect the performance of the webservice as we need to read the entire data into DataStage for probabilistic matching (match specifications in QualityStage).
Could you please provide your inputs on solving this real time update issue.
Please let me know if you need any further information.
Thanks
Raghav