Java Pack Limitations

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
har
Participant
Posts: 118
Joined: Tue Feb 17, 2004 6:23 pm
Location: cincinnati
Contact:

Java Pack Limitations

Post by har »

Hi,
I'm bit curious abt the limitations of Java Pack.
Say for ex i have to process 10 million records or more through Java Client Stage or Transformer stage,does it handle.
The reason why i'm asking is Web Services cant handle more than 10000 records,usually we see lot of errors like heap size or memory issues.
Does the Java Packs works the same way or differently
Apprecite you comments

Thanks,
Har
eostic
Premium Member
Premium Member
Posts: 3838
Joined: Mon Oct 17, 2005 9:34 am

Post by eostic »

There is no comparison....none whatsoever. Web Services Pack is a tool for invoking Web Services.... you are at the mercy of what the Web Service is doing..... WSPack can handle some pretty large payloads......why someone would put really large payloads in a SOAP based service is a whole other discussion....

JavaPack is a tool for extending DataStage....you get the ability to invoke your class, and then it (your class) does whatever it wants to.....reading and writing row by row into your job, inspecting meta data, going out to other sources, calling other java bits, whatever........the limitations there being only whatever limitations you introduce into your own java code... I suppose your class could ultimately blow up the JVM, but still, so could a lot of things....

Ernie
Ernie Ostic

blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
lstsaur
Participant
Posts: 1139
Joined: Thu Oct 21, 2004 9:59 pm

Post by lstsaur »

Actually the Java pack is the only tool that I have been using to solve the limitations of the DataStage, e.g., using JDBC, handling BLOB datatype.

Do you know why your application ended up to choose the Java stage to process 100 millions records? It's all depend on your hardware's capacity to determine whether these 100 million records can be handled.
Roopanwita
Participant
Posts: 125
Joined: Mon Sep 11, 2006 4:22 am
Location: India

Java Pack.

Post by Roopanwita »

Can anybody please tell me how this javapack work? Does it come with DS PX?
I am new to it.Can anybody give me brief idea (function & how to use) of it.
Also can suggest me so site for it.
Thanks
eostic
Premium Member
Premium Member
Posts: 3838
Joined: Mon Oct 17, 2005 9:34 am

Post by eostic »

Hi...

Check out the recent thread on the Java API...

viewtopic.php?t=117195&highlight=

lstsaur placed a very good example and instructions there. I also put an example on the blog below...

Ernie
Ernie Ostic

blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
Post Reply