Missing resource bundle and Too many files on child node

AKUMAR21 · Post by **AKUMAR21** » Sat Mar 15, 2008 8:08 am

I have a sequence job that is running 25 jobs in parallel. Individual jobs run fine but while running the sequencer, many of the jobs abort with the following two errors for some of the transformer, column import and funnel stages; The errors are:
1. Missing resource bundle : /opt/node1/......nls/message/TFIO
2. Too many files in system on node tedscdu2.

Different jobs abort in subsequent runs.

Is it some related with config file, node partitioning etc.
How to resolve these errors?

PS: I dont have dsadmn rights.

chulett · Post by **chulett** » Sat Mar 15, 2008 8:50 am

Sounds to me like you should run fewer jobs at the same time.

AKUMAR21 · Post by **AKUMAR21** » Sun Mar 16, 2008 11:28 pm

I'm supposed to run all the 25 jobs in parallel. Thats the business requirement. Cant change that

Do we really have a solution for this on a developer's level or architectural level?

ArndW · Post by **ArndW** » Mon Mar 17, 2008 2:04 am

Have you used $APT_DUMP_SCORE to see how many PIDs you are firing off at the same time? If this number is in the thousands and the requirement remains that all the jobs need to run at the same time:

(a) optimize the jobs to use less processes
(b) use less nodes in your APT_CONFIG_FILE
(c) What hardware/OS are you running on? You will most likely need to reconfigure you kernel to cater for this type of use

ray.wurlod · Post by **ray.wurlod** » Mon Mar 17, 2008 2:49 am

I find myself wondering whether "resource bundle" refers to resources mentioned in the configuration file. Does the configuration file refer to resources that are not available on the node(s) in question?

Too many files is self-explanatory, but why? It may be that the solution is to run with MORE nodes, so that there are fewer files per node.

AKUMAR21 · Post by **AKUMAR21** » Mon Mar 17, 2008 3:57 am

I tried to change APT_GRID_PARTITIONS from 4 to 2 and this time none of the jobs gave the above erros of missing resource bundle. Not sure how was this param related to the error. However many jobs now got aborted due to virtual ckts error 8024.
Note: I'm using a TPUMP stage loading data into Teradata. There are around 123 TPUMP stages in total. I reduced the min max sessions to 1,1 but got the same error that all virtual ckts are currently in use

ray.wurlod · Post by **ray.wurlod** » Mon Mar 17, 2008 5:04 am

That's the first you've mentioned that you're on a grid - that would have been invaluable knowledge to have in the initial post.

I presume that "ckts" is supposed to be an abbreviation for "circuits", though I fail to see where the "k" comes from.

There is clearly an upper limit on the number of virtual circuits that can be open into Teradata at any one time. You need to find a compromise design that uses fewer than 123 TPump stages simultaneously.

chulett · Post by **chulett** » Mon Mar 17, 2008 6:37 am

How about you post the actual error rather than some made up abbreviated version?

AKUMAR21 wrote:I'm supposed to run all the 25 jobs in parallel. Thats the business requirement. Cant change that

Baloney. What would 'the business' say if that's not physically possible given the constraints of your system? I would certainly think that could be amended to 'run as many jobs as possible in parallel'.

AKUMAR21 also wrote:Note: I'm using a TPUMP stage loading data into Teradata. There are around 123 TPUMP stages in total. I reduced the min max sessions to 1,1 but got the same error that all virtual ckts are currently in use.

Another little nugget that would have been good to note up front. Have you worked with your Teradata DBA, determined if that many simultaneous connections are even possible with your hardware?

Google wrote:Virtual circuits come from the Teradata gateway connections to the PE's on your system. Each PE can support 120 Sessions.

How many Parsing Engines are configured?

AKUMAR21 · Post by **AKUMAR21** » Mon Mar 31, 2008 9:49 am

The exact error is
RDBMS error: 8024, All virtual circuits are in use.
And yes I'm working in Grid environment.
Well I'm not exactly sure how many PEs are there but its less than 3. To attack the sessions constraints I gave TPUMP sessions min,max as 1,1 for all the TPUMP stages through sequencer. Initially I didnt get any virtual circuits error. But next day for the next frequent runs I'm getting the virutal circuits error; but missing resource bundle error is gone for the time being.