Hi to all.
Im trying to merge data from two different transformer stages, the metadata
in both stages is equal, and i would like to merge them without using any
kind of files or database tables, is it possible?
Tanks for your patient.
_________________________________________________________________
Get your FREE download of MSN Explorer at http://explorer.msn.com/intl.asp
Merging data
No, youve been misled. You have a primary input
stream, everything else is a reference. You cannot
reference a stream of data as suggested to you. Your
situation is your desire to only do a reference on
condition.
Questions:
1. Is your reference lookup so detrimental to
performance that it needs to be avoided?
Solution #1: If its not impacting performance
that much, who cares?
2. If your lookup is so detrimental, why is it? Is
it a complex SQL statement in an ODBC/OCI stage?
Solution #1: Optimize the query by restructuring
SQL, using hints, look at indexes, server load, etc.
Solution #2: Stick the lookup in a hash file for
faster reference and potential preload to memory.
3. If your lookup is in a hash file, why is your
lookup performance bad?
Solution #1: Look at how much data you put into
your hash. Does the hash file contain more columns
than necessary. Remember, hash files are faster with
less data columns because the entire row is read from
disk or memory.
Solution #2: If your data is small, like less than
100,000 rows, write a DS Routine/Function to load the
data (output the table to a sequential text file) you
want to lookup up in to an array and then you can use
this Routine/Function in the derivation, and can do a conditional search. This type of function would use a common memory block and initialize a variable the first time the function is called. It will load the file into an array the first time called, and bypass the load thereafter. It then can locate the passed value in the array or not and return anything you want. This can be exceptionally fast.
Good luck!
-Ken
--- pedro santos wrote:
> Hi to all.
> Im trying to merge data from two different
> transformer stages, the metadata
> in both stages is equal, and i would like to merge
> them without using any
> kind of files or database tables, is it possible?
> Tanks for your patient.
>
>
_________________________________________________________________
> Get your FREE download of MSN Explorer at
> http://explorer.msn.com/intl.asp
>
__________________________________________________
Do You Yahoo!?
Find a job, post your resume.
http://careers.yahoo.com
stream, everything else is a reference. You cannot
reference a stream of data as suggested to you. Your
situation is your desire to only do a reference on
condition.
Questions:
1. Is your reference lookup so detrimental to
performance that it needs to be avoided?
Solution #1: If its not impacting performance
that much, who cares?
2. If your lookup is so detrimental, why is it? Is
it a complex SQL statement in an ODBC/OCI stage?
Solution #1: Optimize the query by restructuring
SQL, using hints, look at indexes, server load, etc.
Solution #2: Stick the lookup in a hash file for
faster reference and potential preload to memory.
3. If your lookup is in a hash file, why is your
lookup performance bad?
Solution #1: Look at how much data you put into
your hash. Does the hash file contain more columns
than necessary. Remember, hash files are faster with
less data columns because the entire row is read from
disk or memory.
Solution #2: If your data is small, like less than
100,000 rows, write a DS Routine/Function to load the
data (output the table to a sequential text file) you
want to lookup up in to an array and then you can use
this Routine/Function in the derivation, and can do a conditional search. This type of function would use a common memory block and initialize a variable the first time the function is called. It will load the file into an array the first time called, and bypass the load thereafter. It then can locate the passed value in the array or not and return anything you want. This can be exceptionally fast.
Good luck!
-Ken
--- pedro santos wrote:
> Hi to all.
> Im trying to merge data from two different
> transformer stages, the metadata
> in both stages is equal, and i would like to merge
> them without using any
> kind of files or database tables, is it possible?
> Tanks for your patient.
>
>
_________________________________________________________________
> Get your FREE download of MSN Explorer at
> http://explorer.msn.com/intl.asp
>
__________________________________________________
Do You Yahoo!?
Find a job, post your resume.
http://careers.yahoo.com