About Hashed File Memory Usage

Post questions here relative to DataStage Server Edition for such areas as Server job design, DS Basic, Routines, Job Sequences, etc.

Moderators: chulett, rschirm, roy

Post Reply
suresh.narasimha
Premium Member
Premium Member
Posts: 81
Joined: Mon Nov 21, 2005 4:17 am
Location: Sydney, Australia
Contact:

About Hashed File Memory Usage

Post by suresh.narasimha »

Hi Everyone,

Good Afternoon.

I'm confused with the way hashed file reads the data.

1. If i use a hashed file as a lookup and i donot check option " Preload file to memory " ....does it pull all the data to memory or reads data from the physical file location for each row ?

2. If it reads data from the physical location for every row...which will give me a better performance between hashed file and OCI..if i have 1,00,000 rows.

Please suggest..

Thanks ..
SURESH NARASIMHA
saikir
Participant
Posts: 92
Joined: Wed Nov 08, 2006 12:25 am
Location: Minneapolis
Contact:

Post by saikir »

Hi,

If i understand correctly you want to do a look up in a server edition. In a server edition you can do a lookup using only a hashed file.

Pre-load into memory: Once enabled the hashed file will be read into memory once the job is run. It can certainly improve the performance.

However, if you are hashed file is exceeding your memory then you will have a problem.

Sai
suresh.narasimha
Premium Member
Premium Member
Posts: 81
Joined: Mon Nov 21, 2005 4:17 am
Location: Sydney, Australia
Contact:

About Hashed File Memory Usage

Post by suresh.narasimha »

Hi Saikir,

Thanks.

My question is if i donot enable preload file to memory how does a hashed file behave ?

Regards
SURESH NARASIMHA
saikir
Participant
Posts: 92
Joined: Wed Nov 08, 2006 12:25 am
Location: Minneapolis
Contact:

Post by saikir »

Hi,

If you dont have the option enabled, it will not load the entire hashed file into memory. It will read from the hashed file row by row.

However, if you have a large amount of data to do lookups on and if you can create an intermediate table in your database then load the data into a table and do a equi join. As join in a database is a set based operation and will be much faster.

Sai
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

If you do not enable pre-load to memory each record is logically read from disk. In practice very few of these I/Os will be physical I/Os, because the relevant pages are likely to be in memory - either main memory or in the disk controller's memory.

Even so a hashed file is likely to out-perform even a local Oracle instance because of the way each finds its records. A hashed file uses a hashing algorithm to calculate the actual direct address of the key it needs to retrieve. Oracle (and every other database that don't use hashed file terminology) has the overhead of having to search a primary key index to determine that location.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
suresh.narasimha
Premium Member
Premium Member
Posts: 81
Joined: Mon Nov 21, 2005 4:17 am
Location: Sydney, Australia
Contact:

About Hashed File Memory Usage

Post by suresh.narasimha »

Thanks Ray.

Now i'm clear the way how hashed file behaves.

Regards
SURESH NARASIMHA
Post Reply