DSXchange

Hi,

We know that there is a possibility of making use of Server routines in Parallel Jobs. We are planning to write a server routine where we would be reading data in a Hashed File.

If this is possible, then using this server routine, we should be able to read through the data in a Hashed File in a Parallel Job. Please let us know if this is a plausible approach.

Thanks.

vnspn wrote:Hi,

We know that there is a possibility of making use of Server routines in Parallel Jobs. We are planning to write a server routine where we would be reading data in a Hashed File.

If this is possible, then using this server routine, we should be able to read through the data in a Hashed File in a Parallel Job. Please let us know if this is a plausible approach.

Thanks.

It is possible to call a server routine using a BASIC transformer but it will work only on SMP systems.

Or a Server Shared Container, but that's not a very 'PX' solution. Why a routine to read the hashed file? Why involve hashed files in your PX design at all?

Not a plausible approach. Redesign it so that the contents of a hashed file are now present in a dataset and use a px job to read the dataset. If you could be more clear on what needs to be done, someone here can point you in the right direction.

1) So, is DataSet the best equivalent of Hashed File in parallel Jobs to do a lookup?

2) Okay, here is the kind of situation that we are into, because of which we were thinking of making use of Hashed Files. We are planning to load a table's data into Hashed File. Then, in another job, call a routine, were we traverse through this Hashed File and do some logic and return the output as the output argument of a routine.

Dataset, lookup fileset are all viable options.
This logic that you are talking about, can't it be done at the database level. This eradicates the need to create a snapshot of the table.

Why we didn't want to have this done in the database level is, we want this logic to be done for each incoming records. So, we didn't want to establish a connection with database for each incoming record.

That was the reason we thought of loading the records into Hashed File once and then making use of the data in the Hashed File as a table.

You dont have to open a connection for each record. A connection remains open and all processing can be done within that one connection. Anywho, I gave you my 2 cents and along with others, provided plausible workarounds. Its your call.

nick.bond wrote: 2B, I wouldn't even say it's a workaround, it's just using EE as it should be used.

Just being polite I guess.
If the OP really feels the dire need to use a hashed file then no problem in using a server job to do it. Its just that, in order to achieve the degree of parallism EE provides, needs to be explicitly incorporated using multiple instances and parallel runs.

If it's to be a reusable reference table, a Lookup File Set may be preferable to a Data Set, as the index (hash table) on the key gets pre-built when the Lookup File Set is created.

Thanks for all your comments. I would try to make use of parallel stages as much as possible to exploit the capabilities of EE.

Would get back here if I need something more.

Thanks.