Good ol' sequential file stage warning with a twist!!!!

abhilashnair · Post by **abhilashnair** » Mon Mar 21, 2011 10:52 pm

Parallel job joining two tables using Join Stage. Data loaded into target ODBC which has a reject link. The rejects are being captured in sequential file.

Preserve Partitioning is cleared in ODBC. Input Partitioning is Round Robin in Sequential Stage...Getting warning

"When checking operator: A sequential operator cannot preserve the partitioning of the parallel data set on input port 0"

I did an exact search and tried all the workarounds. Even cleared the partitioning in the transformer stage preceding the ODBC stage, but to no avail. Pls Help

Ravi.K · Post by **Ravi.K** » Tue Mar 22, 2011 3:09 am

Check at Sequential file stage (Reject link) Preserve Partitioning, Is it Clear or Set. if it is set then make it to clear and try.

abhilashnair · Post by **abhilashnair** » Tue Mar 22, 2011 3:23 am

Ravi.K wrote:Check at Sequential file stage (Reject link) Preserve Partitioning, Is it Clear or Set. if it is set then make it to clear and try.

It is clear. As I already mentioned in my above post, the ODBC stage preceding the Seq file stage has Preserve Partitioning as Clear.

jwiles · Post by **jwiles** » Tue Mar 22, 2011 10:54 am

"When checking operator: A sequential operator cannot preserve the partitioning of the parallel data set on input port 0"

The warning is there because an operator running in sequential mode cannot preserve the partitioned output of an operator running in parallel mode. It says so in the warning message! Notice that it didn't say "Sequential File Stage"...it said sequential operator. It is talking about the operating mode of an operator.

Which operator is producing the warning (which ones are running sequentially? The job score can show you) You could specify a collection method for the sequentially running operator, or you can use a message handler to take care of the message.

Regards,

abhilashnair · Post by **abhilashnair** » Tue Mar 22, 2011 11:26 am

Actually the error contains the stage name of the sequential file, I edited it out while pasting here. So it is confirmed that it is the operator for the sequential stage

jwiles · Post by **jwiles** » Tue Mar 22, 2011 11:40 am

Ok. I had the job in another order in my mind (seqfile -> ODBC).

SInce it's on the reject link of the ODBC stage, run the reject to a Copy stage then to the SeqFile stage. Set the Clear Partitioning option on the output of the Copy stage.

Regards,

abhilashnair · Post by **abhilashnair** » Tue Mar 22, 2011 10:21 pm

Just curious to know why it is not working with the Clear Partitioning option on the output of ODBC. Why do I need to add another stage just for the sake of it ?

ray.wurlod · Post by **ray.wurlod** » Tue Mar 22, 2011 11:06 pm

On both outputs of the ODBC stage?

abhilashnair · Post by **abhilashnair** » Tue Mar 22, 2011 11:08 pm

The ODBC stage has just one reject link. There is no other output

ray.wurlod · Post by **ray.wurlod** » Tue Mar 22, 2011 11:08 pm

abhilashnair · Post by **abhilashnair** » Tue Mar 22, 2011 11:17 pm

abhilashnair · Post by **abhilashnair** » Wed Mar 23, 2011 3:44 am

Changed the Partitioning in target ODBC to Same. It was hash before. And the warning went away. Still not sure what caused this in the first place. Anyway the issue is resloved now