One of the fields is used as a Natural Key passed into a Shared Container to do a lookup and retrieve the corresponding Surrogate Key.
When I run the job, the 8 million rows run through the initial transformer which passes the Natural Key into the Shared container. The rows are then passed through a tmp file before moving to a 2nd transformer stage where the Surrogate key is passed into. The problem seems to be that once around 4 million rows have been passed through the container, the following error occurs.
Do you think that I will need to stream my jobs to pass smaller volumes through or should DS be able to cope with 8 million rows at a time ?UKWHBMISCAxxInbGLMTfrm1Job..ARRG_ID_Lkup.SurrKey_Write_Tfm: [IBM][CLI Driver] SQL30081N A communication error has been detected. Communication protocol being used: "TCP/IP". Communication API being used: "SOCKETS". Location where the error was detected: "128.8.123.16". Communication function detecting the error: "recv". Protocol specific error code(s): "*", "*", "0". SQLSTATE=08001
SQLExecute: Error executing statement 'SELECT next value for DWHD019.DWH_ARRG_XREF_SURRID,-1 FROM SYSIBM.SYSDUMMY1'. See following DB2 message for details.