DSXchange

Hi,

My requirement is to extract data from different tables using ODBC stage using MySQL. We have multiple tables with around 50 fields and 40 million records each.

My job design is simple: ODBC -> Copy -> Dataset.

RCP is enabled and no metadata is defined in any of the stages.

But, when I'm executing the job, sometimes the job fails and sometimes it's starts processing records after a long time, almost after 45 minutes.

That raised some questions in my mind as to how RCP works.

Does Datastage scan through all the records and determine the metadata in realtime or what.

One more observation is that when I expected the same queries on a sample table, which contains 1000 records, the job runs in 1 min or less.

Please shed some light on this issue.

Best Regards,
Sam

DataStage does not do a full table scan to get the metadata.

You mention that the jobs occasionally fail. With what error message(s)?

Hi ArndW,

These are the messages from the log.

"APT_CombinedOperatorController,0: Fatal Error: waitForWriteSignal(): Premature EOF on node xxxx-yyyy-zzzz-360etl-x1 Socket operation on non-socket"

"APT_CombinedOperatorController,1: Fatal Error: waitForWriteSignal(): Premature EOF on node xxxx-yyyy-zzzz-360etl-x1 Socket operation on non-socket"


Best Regards,
Sam

What do you infer from "waitForWriteSignal"?

Hi Ray,

Searched around and this could be due to number of issues.

According to IBM, could be an issue with using 'CIO' option while mounting the scratch and resource disks.

Someone in dsxchange was referring that this could be an issue with compiler version used.

Currently we are using Datastage 11.3.1.2.

Interestingly, I just got information from my colleague that he did not run into such issues even when he was extracting data from a table that contains around 350 million records. The table I'm currently extracting from contains only 38 million records.

Best Regards,
Sam