Standardization:Using the seq file extractd from xml

divstands · Post by **divstands** » Tue Jun 16, 2009 2:43 pm

While running the above job, I am getting the following errors:

Data,0: Internal Error: (size <= blocksize_): iomgr/iocomm.C: 1736
node_node1: Player 1 terminated unexpectedly. [processmgr/player.C:149]
main_program: APT_PMsectionLeader(1, node1), player 1 - Unexpected exit status 1. [processmgr/slprocess.C:368]
SelectMX,0: Internal Error: (size <= blocksize_): iomgr/iocomm.C: 1736
node_node1: Player 3 terminated unexpectedly. [processmgr/player.C:149]
main_program: APT_PMsectionLeader(1, node1), player 3 - Unexpected exit status 1. [processmgr/slprocess.C:368]
APT_PMsectionLeader(1, node1), player 4 - Unexpected exit status 1. [processmgr/slprocess.C:368]
main_program: Step execution finished with status = FAILED. [sc/sc_api.C:242]

However, I do not get any error( the job runs successfully) when I take the input data as a sequential file with data extracted from the SQL database.

What should I modify to run the above mentioned job perfectly.

eostic · Post by **eostic** » Tue Jun 16, 2009 2:49 pm

tell us lots more. What are the sources/targets, what stage types, what are you trying to do, etc..... there's not enough context here to know what might be the solution.

Ernie

divstands · Post by **divstands** » Tue Jun 16, 2009 2:54 pm

eostic wrote:tell us lots more. What are the sources/targets, what stage types, what are you trying to do, etc..... there's not enough context here to know what might be the solution.

Ernie

I converted an xml to a sequential file using the job which looks like :
Sequential File(Input as xml)  XML Input Stage  Sequential file(Output File B)
Later on I tried to standardize the home address data in the Output File B, where the job looks like:
Sequential File(Input as File B)  Standardize(Country)  Filter(Select Country) Standardize(Home Address using MXADDR.set) Sequential File

While running the above job, I am getting the following errors:

Data,0: Internal Error: (size <= blocksize_): iomgr/iocomm.C: 1736
node_node1: Player 1 terminated unexpectedly. [processmgr/player.C:149]
main_program: APT_PMsectionLeader(1, node1), player 1 - Unexpected exit status 1. [processmgr/slprocess.C:368]
SelectMX,0: Internal Error: (size <= blocksize_): iomgr/iocomm.C: 1736
node_node1: Player 3 terminated unexpectedly. [processmgr/player.C:149]
main_program: APT_PMsectionLeader(1, node1), player 3 - Unexpected exit status 1. [processmgr/slprocess.C:368]
APT_PMsectionLeader(1, node1), player 4 - Unexpected exit status 1. [processmgr/slprocess.C:368]
main_program: Step execution finished with status = FAILED. [sc/sc_api.C:242]

However, I do not get any error( the job runs successfully) when I take the input data as a sequential file with data extracted from the SQL database.

What should I modify to run the above mentioned job perfectly.

ray.wurlod · Post by **ray.wurlod** » Tue Jun 16, 2009 4:33 pm

Please delete your duplicate post before anyone responds to it.

eostic · Post by **eostic** » Tue Jun 16, 2009 7:49 pm

How much have you tested the first job? Is the file you created exactly as you expected? Are the formats everything that you need? Sounds like the file is not getting created as you expected, or that the metadata in the second Job is not correct when reading that file. Have you used the Sequential stage before in a job like your second one? Have you looked at the file in an editor, and tried it with just a few rows? Can you create the file for the second job by hand and get that to work?

The problem (if it is a problem) may be in the first Job, but it's not clear if it's a problem, or just a mismatch between what the first job is doing and what you are trying to do with that data in the second.

Ernie

divstands · Post by **divstands** » Wed Jun 17, 2009 11:51 am

eostic wrote:How much have you tested the first job? Is the file you created exactly as you expected? Are the formats everything that you need? Sounds like the file is not getting created as you expected, or that the metadata in the second Job is not correct when reading that file. Have you used the Sequential stage before in a job like your second one? Have you looked at the file in an editor, and tried it with just a few rows? Can you create the file for the second job by hand and get that to work?

The problem (if it is a problem) may be in the first Job, but it's not clear if it's a problem, or just a mismatch between what the first job is doing and what you are trying to do with that data in the second.

Ernie

Yes I am getting the file in the expected format ( I suppose! For the xpath /TCRMService/RequestControl/DWLControl/requesterName , Output column is like <requesterName>MDMUSER</requesterName>
And for xpath
/TCRMService/RequestControl/DWLControl/requesterName/text(), output column is like MDMUSER)

Yes, I have used the Sequential stage before in a job like my second one. However there is no difference in the metadata used in the first and second job, because ,
a). I have tried bifurcating the job into Job 1 and 2 as explained
b). I made the job as one and ran it

For both the cases the Error is same:
Case 1:
main_program: IBM WebSphere DataStage Enterprise Edition 8.1.0.5040
main_program: conductor uname: -s=AIX; -r=3; -v=5; -n=bcbeta052; -m=00C6BDFF4C00
main_program: orchgeneral: loaded
main_program: Echo:
main_program: Explanation:
main_program: Dump:
main_program: APT configuration file:
main_program: This step has 3 datasets:
main_program: Schemas:
Data_Set_80,0: Internal Error: (size <= blocksize_): iomgr/iocomm.C: 1736
node_node1: Player 1 terminated unexpectedly. [processmgr/player.C:149]
main_program: APT_PMsectionLeader(1, node1), player 1 - Unexpected exit status 1.
Copy_of_StandardizeCNTRY,0: Failure during execution of operator logic. [api/operator_rep.C:399]
Copy_of_StandardizeCNTRY,0: Input 0 consumed 0 records.
Copy_of_StandardizeCNTRY,0: Output 0 produced 0 records.

Case 2:
main_program: IBM WebSphere DataStage Enterprise Edition 8.1.0.5040
main_program: conductor uname: -s=AIX; -r=3; -v=5; -n=bcbeta052; -m=00C6BDFF4C00
main_program: orchgeneral: loaded
main_program: Echo:
main_program: Explanation:
main_program: Dump:
main_program: APT configuration file:
main_program: This step has 3 datasets:
main_program: Schemas:
StandardizeCNTRY,0: Internal Error: (size <= blocksize_): iomgr/iocomm.C: 1736
node_node1: Player 1 terminated unexpectedly. [processmgr/player.C:149]
main_program: APT_PMsectionLeader(1, node1), player 1 - Unexpected exit status 1.
SelectMX,0: Internal Error: (size <= blocksize_): iomgr/iocomm.C: 1736
xStan,0: Internal Error: (size <= blocksize_): iomgr/iocomm.C: 1736
main_program: Step execution finished with status = FAILED. [sc/sc_api.C:242]

In both the above case, there are few warnings like
When checking operator: When binding input interface field "ValorPiso_MXADDR" to field "ValorPiso_MXADDR": Implicit conversion from source type "ustring[max=10]" to result type "string[max=10]": Converting ustring to string using codepage ISO-8859-1. [api/interface_rep.C:6183]

For few fields generated out of standardize stage, but I have used the same job for
a). data extracted from a database to a sequential file.
It worked well. Only when I am extracting the data
b). From an xml file to a sequential file/data set,
the above errors are coming.

What can be the issue?

I also tried to change the environment variables for the
APT_DEFAULT_TRANSPORT_BLOCK:SIZE from 131072 to 262144 and
APT_DEFAULT_TRANSPORT_BLOCK:SIZE from 1048576 to 2097136

But because of the above changes few other parallel jobs stopped working. Hence I switched to default values.

eostic · Post by **eostic** » Wed Jun 17, 2009 12:00 pm

Hard to say what to advise you here, except that you need to start breaking things down to find out where it works and where it blows up. Reduce it dramatically. Follow your XML stage with a transformer and exclude all but maybe 3 columns and maybe 1 row. Be certain that they are 100% character. See if that works...then add columns and rows back in...and so forth until you see what triggers the problem.

Ernie

divstands · Post by **divstands** » Wed Jun 17, 2009 4:45 pm

eostic wrote:Hard to say what to advise you here, except that you need to start breaking things down to find out where it works and where it blows up. Reduce it dramatically. Follow your XML stage with a transformer and exclude all but maybe 3 columns and maybe 1 row. Be certain that they are 100% character. See if that works...then add columns and rows back in...and so forth until you see what triggers the problem.

Ernie

Hi Ernie

Its solved now. In the env variables i changed

APT_DEFAULT_TRANSPORT_BLOCK_SIZE = APT_MAX_TRANSPORT_BLOCK_SIZE = 1048576

and it worked!

ray.wurlod · Post by **ray.wurlod** » Wed Jun 17, 2009 5:52 pm

What were their previous values, who advised these changes, and (did they explain) why changing these values fixed the particular problem? For that matter, what was the real, underlying, original problem?

divstands · Post by **divstands** » Thu Jun 18, 2009 10:48 am

ray.wurlod wrote:What were their previous values, who advised these changes, and (did they explain) why changing these values fixed the particular problem? For that matter, what was the real, underlying, original problem?

The APT_DEFAULT_TRANSPORT_BLOCK_SIZE was 131072 initially which was not in line with the requisite value according to the formula for the block size. You can find the formula in the documentation. As of now i am not able to access the support site to retrieve that.