Matching Error

[vairus]

hi all,

I had 7 passes in matching. i got error while its processing 6th pass.
i got this error in OSHERRORLOG.txt.
Could anybody help me to figure out what is this error exactly?


##I IIS-DSEE-TFCN-00001 13:07:47(000) <main_program>
IBM WebSphere DataStage Enterprise Edition 8.0.1.4665
Copyright (c) 2001, 2005-2007 IBM Corporation. All rights reserved



##I IIS-DSEE-TUTL-00031 13:07:47(001) <main_program> The open files limit is 2000; raising to 2147483647.
##I IIS-DSEE-TOSH-00002 13:07:47(002) <main_program> orchgeneral: loaded
##I IIS-DSEE-TOSH-00002 13:07:47(003) <main_program> orchsort: loaded
##I IIS-DSEE-TOSH-00002 13:07:47(004) <main_program> orchstats: loaded
##I IIS-QSEE-POP*-00001 13:07:47(009) <Mat> Creating sub-operator: <QSmatStats -workDir /opt/IBM/InformationServer/Server/Projects/ELF/QSScratch/Mat -f Mat.MAT -trace all -lvl 1>

##I IIS-QSEE-POP*-00001 13:07:47(010) <Mat> Creating sub-operator: <copy>

##I IIS-QSEE-POP*-00001 13:07:47(011) <Mat> Creating sub-operator: <copy>

##I IIS-QSEE-POP*-00001 13:07:47(012) <Mat> Creating sub-operator: <QSmat -n 1 -workDir /opt/IBM/InformationServer/Server/Projects/ELF/QSScratch/Mat -f Mat.MAT -trace all -lvl 1>

##I IIS-QSEE-POP*-00001 13:07:47(013) <Mat> Creating sub-operator: <tsort -flagKey -stable -key MatchPrimaryWord1NYSIIS_ZANAME -ci -key MatchPrimaryWord2NYSIIS_ZANAME -ci -key MatchPrimaryWord3NYSIIS_ZANAME -ci -collation_sequence en -strength secondary>

##I IIS-QSEE-POP*-00001 13:07:47(014) <Mat> Creating sub-operator: <tsort -key qsMatchDataIDA -key qsMatchPassNumber -collation_sequence en -strength secondary>

##I IIS-QSEE-POP*-00001 13:07:47(015) <Mat> Creating sub-operator: <copy>

##I IIS-QSEE-POP*-00001 13:07:47(016) <Mat> Creating sub-operator: <innerjoin -key qsMatchDataID>

##I IIS-QSEE-POP*-00001 13:07:47(017) <Mat> Creating sub-operator: <tsort -key qsMatchDataID -collation_sequence en -strength secondary>

##I IIS-QSEE-POP*-00001 13:07:47(018) <Mat> Creating sub-operator: <QSmat_join -U -workDir /opt/IBM/InformationServer/Server/Projects/ELF/QSScratch/Mat -f Mat.MAT -trace all -lvl 1>

##I IIS-QSEE-POP*-00001 13:07:47(019) <Mat> Creating sub-operator: <copy>

##I IIS-DSEE-TDOD-00080 13:07:47(021) <ODBC_Enterprise_0> Process meta data not available in database
##I IIS-DSEE-TDOD-00080 13:07:47(023) <ODBC_Enterprise_1> Process meta data not available in database
##I IIS-DSEE-TFSC-00001 13:07:47(025) <main_program> APT configuration file: /opt/IBM/InformationServer/Server/Configurations/default.apt
##W IIS-DSEE-TFOR-00073 13:07:48(000) <APT_CopyOperator(1) in Mat> When checking operator: A sequential operator cannot preserve the partitioning
of the parallel data set on input port 0.
##W IIS-DSEE-TFIP-00095 13:07:48(001) <APT_CopyOperator(3) in Mat> When checking operator: The modify operator drops field "qsMatchMasterFlag" which is not present in output interface.
##W IIS-DSEE-TFIP-00072 13:07:48(002) <APT_CopyOperator(3) in Mat> When checking operator: When binding output schema variable "outRec": When binding output interface field "qsMatchSetID" to field "qsMatchSetID": Implicit conversion from source type "uint64" to result type "int64": Possible range limitation.
##W IIS-DSEE-TFIP-00072 13:07:48(003) <APT_CopyOperator(3) in Mat> When checking operator: When binding output schema variable "outRec": When binding output interface field "qsMatchDataID" to field "qsMatchDataID": Implicit conversion from source type "uint64" to result type "int64": Possible range limitation.
##W IIS-DSEE-TFIP-00072 13:07:48(004) <QSM_MatStatsOp in Mat> When checking operator: When binding output interface field "qsMatchStatValue" to field "qsMatchStatValue": Implicit conversion from source type "uint64" to result type "int64": Possible range limitation.
##W IIS-DSEE-TFOR-00073 13:07:48(005) <APT_CopyOperator(1) in Mat> When checking operator: A sequential operator cannot preserve the partitioning
of the parallel data set on input port 0.
##I IIS-QSEE-MATF-00001 13:15:29(000) <QSM_MatOp in Mat,0>
Variable: MatchPrimaryWord

##I IIS-QSEE-MATF-00001 13:15:29(001) <QSM_MatOp in Mat,0> New MISSING weight for ALL values: 0.00

##I IIS-QSEE-MATF-00001 13:15:29(002) <QSM_MatOp in Mat,0>
ALL VALUES INSIDE TABLE

##I IIS-QSEE-MATF-00001 13:15:29(003) <QSM_MatOp in Mat,0>
Columns are: Value, Frequency, MAgree, MPart, mprob, uprob, type, Agree weight, Disagree weight

##I IIS-QSEE-MATF-00001 13:15:29(004) <QSM_MatOp in Mat,0> SALCOR 1 0 0 1.00 1.00 D 0.01 0.00

##I IIS-QSEE-MATF-00001 13:15:29(005) <QSM_MatOp in Mat,0>
Variable: MatchPrimaryWord

##I IIS-QSEE-MATF-00001 13:15:29(006) <QSM_MatOp in Mat,0> New MISSING weight for ALL values: 0.00

##I IIS-QSEE-MATF-00001 13:15:29(007) <QSM_MatOp in Mat,0>
ALL VALUES INSIDE TABLE

##I IIS-QSEE-MATF-00001 13:15:29(008) <QSM_MatOp in Mat,0>
Columns are: Value, Frequency, MAgree, MPart, mprob, uprob, type, Agree weight, Disagree weight

##I IIS-QSEE-MATF-00001 13:15:29(009) <QSM_MatOp in Mat,0> INGENIEURSWERKE 1 0 0 1.00 1.00 D 0.01 0.00

##I IIS-QSEE-MATF-00001 13:15:29(010) <QSM_MatOp in Mat,0>
Variable: MatchPrimaryWord

##I IIS-QSEE-MATF-00001 13:15:29(011) <QSM_MatOp in Mat,0> New MISSING weight for ALL values: 0.00

##I IIS-QSEE-MATF-00001 13:15:29(012) <QSM_MatOp in Mat,0>
ALL VALUES INSIDE TABLE

##I IIS-QSEE-MATF-00001 13:15:29(013) <QSM_MatOp in Mat,0>
Columns are: Value, Frequency, MAgree, MPart, mprob, uprob, type, Agree weight, Disagree weight

##I IIS-QSEE-MATF-00001 13:15:29(014) <QSM_MatOp in Mat,0> 1 0 0 1.00 1.00 D 0.01 0.00

##I IIS-QSEE-MATM-00005 13:15:29(015) <QSM_MatOp in Mat,0> Frequency table(s) will be used

##I IIS-QSEE-MATM-00007 13:15:29(016) <QSM_MatOp in Mat,0> Default weights calculated for values OUTSIDE table

##I IIS-QSEE-MATM-00007 13:15:29(017) <QSM_MatOp in Mat,0> Columns are: A Var Name, User mprob, Revised mprob, User uprob, Revised uprob, Agree weight, Disagree weight, Missing weight

##I IIS-QSEE-MATM-00007 13:15:29(018) <QSM_MatOp in Mat,0> MatchPrimaryWord 1.00 0.9999 1.00 0.9999 0.00 0.00 0.00

##I IIS-QSEE-MATM-00007 13:15:29(019) <QSM_MatOp in Mat,0> MatchPrimaryWord 1.00 0.9999 1.00 0.9999 0.00 0.00 0.00

##I IIS-QSEE-MATM-00007 13:15:29(020) <QSM_MatOp in Mat,0> MatchPrimaryWord 1.00 0.9999 1.00 0.9999 0.00 0.00 0.00

##I IIS-QSEE-MATM-00008 13:15:29(021) <QSM_MatOp in Mat,0> <Pass 1> Blocks processed: 1501845

##I IIS-QSEE-MATM-00008 13:15:29(022) <QSM_MatOp in Mat,0> Data records: 3233804 read, 2790709 residuals written

##I IIS-QSEE-MATM-00008 13:15:29(023) <QSM_MatOp in Mat,0> Duplicates: 443095 probable, 0 possible

##E IIS-DSEE-TFIP-00024 13:15:29(000) <QSM_MatStatsOp in Mat,0> Un-handled conversion error on field "qsMatchStatValue " from source type "uint64" to destination type "int64":
source value="18446744071790405731"; the result is non-nullable and there is no handle_null to specify a default value.
##I IIS-QSEE-PMSO-00001 13:15:29(001) <QSM_MatStatsOp in Mat,0> ** Output Statistics For UNDUPLICATE **
*

##I IIS-QSEE-PMSO-00001 13:15:29(002) <QSM_MatStatsOp in Mat,0> ****************************** PASS 1 *******************************

##I IIS-QSEE-PMSO-00001 13:15:29(003) <QSM_MatStatsOp in Mat,0> *
* A records read: 3233804

##I IIS-QSEE-PMSO-00001 13:15:29(004) <QSM_MatStatsOp in Mat,0> * Blocks processed: 1501845
* Overflow blocks: 0

##I IIS-QSEE-PMSO-00001 13:15:29(005) <QSM_MatStatsOp in Mat,0> * Maximum A block size (including overflow): 1743

##I IIS-QSEE-PMSO-00001 13:15:29(006) <QSM_MatStatsOp in Mat,0> * Average A block size (not including overflow): 1.32

##I IIS-QSEE-PMSO-00001 13:15:29(007) <QSM_MatStatsOp in Mat,0> * Pseudo matches: 271733
* Duplicates: 443095

##I IIS-QSEE-PMSO-00001 13:15:29(008) <QSM_MatStatsOp in Mat,0> * EXACT duplicates: 360988
* Clerical duplicates: 0

##I IIS-QSEE-PMSO-00001 13:15:29(009) <QSM_MatStatsOp in Mat,0> * A residuals (including MISSING): 2790709

##I IIS-QSEE-PMSO-00001 13:15:29(010) <QSM_MatStatsOp in Mat,0> *
* Cost Element Allocation Statistics

##I IIS-QSEE-PMSO-00001 13:15:29(011) <QSM_MatStatsOp in Mat,0> * Maximum elements used on a case: 753378

##I IIS-QSEE-PMSO-00001 13:15:29(012) <QSM_MatStatsOp in Mat,0> * Average elements used on a case: 24184521890252.90

##I IIS-QSEE-PMSO-00001 13:15:29(013) <QSM_MatStatsOp in Mat,0> *
***********************************************************

##I IIS-QSEE-PMSO-00001 13:15:29(014) <QSM_MatStatsOp in Mat,0> *
* H I S T O G R A M
*

##I IIS-QSEE-PMSO-00001 13:15:29(015) <QSM_MatStatsOp in Mat,0> * Distribution of observed weights for all LINKED records ONLY

##I IIS-QSEE-PMSO-00001 13:15:29(016) <QSM_MatStatsOp in Mat,0> * Scale based on mean frequency of: 443095.00

##I IIS-QSEE-PMSO-00001 13:15:29(017) <QSM_MatStatsOp in Mat,0> *
* For weights with a frequency greater than the mean the histogram shows a '>' in the last column

##I IIS-QSEE-PMSO-00001 13:15:29(018) <QSM_MatStatsOp in Mat,0> *
* Weight Frequency
*

##I IIS-QSEE-PMSO-00001 13:15:29(019) <QSM_MatStatsOp in Mat,0> * 0.00 443095 ****************************************************************************************************

##I IIS-QSEE-PMSO-00001 13:15:29(020) <QSM_MatStatsOp in Mat,0> *
* TOTAL LINKED COMPARISONS: 443095
*

##E IIS-DSEE-TFOR-00001 13:15:29(021) <QSM_MatStatsOp in Mat,0> Failure during execution of operator logic.
##I IIS-DSEE-TFOR-00163 13:15:29(022) <QSM_MatStatsOp in Mat,0> Input 0 consumed 3 records.
##I IIS-DSEE-TFOR-00094 13:15:29(023) <QSM_MatStatsOp in Mat,0> Output 0 produced 4 records.
##F IIS-DSEE-TFDR-00036 13:15:29(024) <QSM_MatStatsOp in Mat,0> Fatal Error: Virtual data set.; output of "QSM_MatStatsOp in Mat": un-handled error converting output field.
##E IIS-DSEE-TFPM-00192 13:15:29(000) <node_node3> Player 3 terminated unexpectedly.
##E IIS-DSEE-TFPM-00338 13:15:29(000) <main_program> APT_PMsectionLeader(3, node3), player 3 - Unexpected exit status 1.
##E IIS-DSEE-TFOR-00001 13:15:29(000) <ODBC_Enterprise_1,0> Failure during execution of operator logic.
##I IIS-DSEE-TFOR-00163 13:15:29(001) <ODBC_Enterprise_1,0> Input 0 consumed 0 records.
##E IIS-DSEE-TFOR-00001 13:15:29(000) <ODBC_Enterprise_1,2> Failure during execution of operator logic.
##I IIS-DSEE-TFOR-00163 13:15:29(001) <ODBC_Enterprise_1,2> Input 0 consumed 0 records.
##F IIS-DSEE-TFIO-00126 13:15:29(002) <ODBC_Enterprise_1,2> Fatal Error: waitForWriteSignal(): Premature EOF on node CA001L15 Socket operation on non-socket
##E IIS-DSEE-TFPM-00192 13:15:29(001) <node_node3> Player 4 terminated unexpectedly.
##E IIS-DSEE-TFPM-00338 13:15:29(001) <main_program> APT_PMsectionLeader(3, node3), player 4 - Unexpected exit status 1.
##E IIS-DSEE-TFPM-00338 13:15:29(002) <main_program> APT_PMsectionLeader(4, node4), player 3 - Unexpected exit status 1.
##E IIS-DSEE-TFPM-00338 13:15:29(003) <main_program> APT_PMsectionLeader(2, node2), player 3 - Unexpected exit status 1.
##E IIS-DSEE-TFPM-00338 13:15:29(004) <main_program> APT_PMsectionLeader(1, node1), player 3 - Unexpected exit status 1.
##E IIS-DSEE-TFSC-00011 13:15:34(000) <main_program> Step execution finished with status = FAILED.
##I IIS-DSEE-TCOS-00026 13:15:34(001) <main_program> Startup time, 0:09; production run time, 7:38.


Thanks in Advance guys

[ray.wurlod]

In version 7 there is a limit of five passes. I don't know, however, whether this limit still applies in version 8. If it does, it might explain a fatal error on the sixth pass.

[vairus]

Hi ,

thanks for your reply RAY . In Earlier version 7.5 , the maximum number of pass is 7. i hope it should be more than 7 or atleast 7 in version8 .

I tested the same 7 passes with 20,000 records and i got the result.

Am running now with 4million records ,12GB Ram,500GB disk space.

it create datasets upto 5 passes and while processing 6th pass it gives me error.

i increased the block overflow to 1million.

I couldnt figure out exactly whats the problem

I hope now you will get idea on the problem..

thanks

In version 7 there is a limit of five passes. I don't know, however, whether this limit still applies in version 8. If it does, it might explain a fatal error on the sixth pass. ...