my job requires lookup from teradata table for each row of source, the teradata table has 200+ millions records. how can i pass the input column name in the where clause of select statement in the teradata table.
Teradata Table
|
|
|
source file--------->Lookup stage ------------>target
teradata table lookup
Moderators: chulett, rschirm, roy
-
- Premium Member
- Posts: 783
- Joined: Mon Jan 16, 2006 10:17 pm
- Location: Sydney, Australia
-
- Premium Member
- Posts: 783
- Joined: Mon Jan 16, 2006 10:17 pm
- Location: Sydney, Australia
-
- Participant
- Posts: 3593
- Joined: Thu Jan 23, 2003 5:25 pm
- Location: Australia, Melbourne
- Contact:
If you are using an early 7.x release your key lookup column and your stream columns need to have exactly the same name before you can select them for a lookup key field. If you are using 7.5.1 then a normal lookup stage will let you drag and drop the stream field to the lookup key field. With 200 million records it will provide poor performance as it is too large to load into memory. A join stage may work better.
If your stream data is quite small then a sparse lookup is possible as a row is retrieved from the database for each input row instead of the full 200 million being returned. In this case your stream lookup key field needs to have exactly the same name as your joining stream field and they will be automatically linked together, you will not see the link within the Lookup stage.
If your stream data is quite small then a sparse lookup is possible as a row is retrieved from the database for each input row instead of the full 200 million being returned. In this case your stream lookup key field needs to have exactly the same name as your joining stream field and they will be automatically linked together, you will not see the link within the Lookup stage.
Certus Solutions
Blog: Tooling Around in the InfoSphere
Twitter: @vmcburney
LinkedIn:Vincent McBurney LinkedIn
Blog: Tooling Around in the InfoSphere
Twitter: @vmcburney
LinkedIn:Vincent McBurney LinkedIn
-
- Premium Member
- Posts: 783
- Joined: Mon Jan 16, 2006 10:17 pm
- Location: Sydney, Australia
i didn't find any sparse or normal lookup type for Teradata Table stage.
how can i pass input column in the where clause of the select query in teradat stage.
"SELECT
PRTY_ID, CDBOR_CIDN APP_KEY_VAL_1
FROM
RTW4B_TDW_D01_Data.TR_PRTY
where CDBOR_CIDN=ORCHESTRATE.APP_KEY_VAL_1"
this query doesn't work
how can i pass input column in the where clause of the select query in teradat stage.
"SELECT
PRTY_ID, CDBOR_CIDN APP_KEY_VAL_1
FROM
RTW4B_TDW_D01_Data.TR_PRTY
where CDBOR_CIDN=ORCHESTRATE.APP_KEY_VAL_1"
this query doesn't work
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Sparse lookup is only available for DB2 and Oracle.
You need either to pre-load a Lookup File Set with the data in your lookup table, or involve a virtual Data Set. In the latter case you really need to have enough memory available such that the entire reference data set can be resident.
You need either to pre-load a Lookup File Set with the data in your lookup table, or involve a virtual Data Set. In the latter case you really need to have enough memory available such that the entire reference data set can be resident.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
-
- Premium Member
- Posts: 783
- Joined: Mon Jan 16, 2006 10:17 pm
- Location: Sydney, Australia
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
If both sources are Teradata, can you perform the join within Teradata?
If not, can you reverse the logic, so that your lookup table only contains 200000 rows?
Otherwise, do consider a server job, which can perform lookups directly against the (large) Teradata table. With only 200000 records to process this may be an adequate solution.
If not, can you reverse the logic, so that your lookup table only contains 200000 rows?
Otherwise, do consider a server job, which can perform lookups directly against the (large) Teradata table. With only 200000 records to process this may be an adequate solution.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.