DSXchange

If you think about it, turning off combination just means that more processes are doing the work. If your job (with combination) uses a lot of CPU then you might have been bottlenecked there and by disabling combination the load is spread across more processes and if you have enough physical CPUs o...

Obviously one million rows is still below the "large" threshold. Hi Ray, I've just created a job that has source of 8 million records doing lookup to lookup file sets that also contain 8 million records. With another job that has source of 8 million, performing outer join with reference d...

Hi Arnd, Was wondering, cause I have also encountered the same error as you. My job looks like this: Source-> lookup-> lookup-> target. Both lookup file sets are the same and contains 8 mil data. Initially when did a comparison when my lookup file set has 1.6 million data with combinality turned on ...

Hi all, I created a job with estimated 8-10 million record coming from source. And does a lookup to lookup file set that contains nearly 1+ million records. However was told that if the data volume on both sides (source and lookup reference) are huge, it's better to use a join stage (In my case an o...

Basically I have 2 questions regarding look up file sets. 1) I have 2 jobs that I intend to use to populate into a particular lookup file set named abc.fs Each job obtains source data from different source files -> perform a particular business mapping onto the data then populating into said lookup ...

Parallel job Sort and Aggregator stages are far more robust with large volumes of data. They are NOT based upon their server job counterparts, but have been written specifically to take advantage of the new environment. Thanks Ray. I've managed to scrape out 4 million ++ records source and the sort...

Hi all was wondering, Previously I've had some experience with Server job's Sort & Aggregator stage(s) but they do not handle well data volume of nearly 1 million records ++. Abortion was likely the case when reaching that amount. Was wondering if this limitation is apparent in PX's Sort & A...

Normal look up has to load data into memory first and then provide a look up facility from the memory. SQL in Normal look up stage will first load all the data it fetched into memory for this. So SQL used in Normal stage will be not be able to accept/recognize ORCHESTRATE.age, for the simple reason...

The format "ORCHESTRATE.age" is the correct one. What happens if you mark age as key and let DS auto-generate the SQL - does that compile and run correctly? Perhaps it is the BETWEEN statement and syntax and not the use of your token that is generating your error. I've changed my sql to l...

My job design goes like this: Source file -> Lookup file stage (With db2 stage as reference input) -> Output sequential file. From my source file I have 2 fields: name, age. DB2 stage sql query goes like this: SELECT AGE_ID, MIN_AGE, MAX_AGE FROM AGE_TABLE WHERE ???? BETWEEN MIN_AGE and MAX_AGE. The...

Hey ray, and all,

Thanks fo much for the suggestions. =) It's working fine now with the tail stage.

Hi all, Thanks for your reply. =) Actually, Just found a solution to this but not sure if they are suitable ones. I either use a Tail Stage/Duplicate stage to obtain the last record. Since I am only concern on the row count (First field) value which is the biggest... But performance wise... do the a...

Good day all, I am having the below predicament. row_count, emp_no, emp_name 1 123 Sean 2 128 Aileen 3 132 Xuan I have these records going through into the PX transformer. Question is, issit possible to specify just outputing the last record within the transformer itself? My goal is to obtain the va...

Hi Ray, In my db2 stage I set the server as db2inst2 whereas my client instance name is db2inst1. I guess that's the cause. But I was wondering how this works. Please correct if me if my understanding is wrong. The client instance name is set as db2inst1 in my project (ETL Server) /home/db2inst1/sql...

Hi All, I found a number of posts in forum related to ds_ipcopen() but most of them are file not found and space issue. For my case, I'm using link collectors as outputs of four shared containers so I use Enable row buffer option. At the begining of job (before link partitioner-collector), there are...