Bug in pivot stage ???

chulett · Post by **chulett** » Tue Jun 09, 2009 9:31 pm

nagarjuna wrote:Thank you very much craig for testing .Hope you have given datatype as Bigint in both i/p,o/p of pivot and test data is also Big int ( value greater than int ).

Yes indeed.

It might help if you explained your troublesome job design in full - stages, processing, etc so we can perhaps do a more apples-to-apples comparison / test. You said earlier that if (basically) all you had was a pivot it "worked fine" so wondering what other stages are involved when it doesn't work fine.

And please confirm we're talking about the Pivot Enterprise stage here, yes?

nagarjuna · Post by **nagarjuna** » Tue Jun 09, 2009 9:46 pm

Pivot enterprise stage ?? I am using a pivot stage ( whose documentation is in server guide ) ...So you mean to say that there is another pivot stage in parallel jobs ?? I dont know about it.Right now i am not at my work place so i dont have access

...Any ways my parallel job design is as follows :

Sequential file ------> Pivot ( server stage ) -----> Sequential File .

From Source to target , I have given a field ID ( as BigInt ) .Its failing for Bigint data with a warning arthimetic overflow .

ray.wurlod · Post by **ray.wurlod** » Tue Jun 09, 2009 9:53 pm

There is indeed a Pivot Enterprise stage. It was introduced in version 8.1. It primarily offers a graphical tool for selecting the pivot key and pivoting columns and to include a "pivot index" column (auto-numbering of the rows for each pivot key value). The first of these means that you do not have to type in a comma-separated list of column names.

The other major differences are that the Pivot stage default execution mode is sequential while the Pivot Enterprise stage default execution mode is parallel, and in a future version the Pivot Enterprise will be able to support vertical pivots (rows to columns).

In the title bar the Pivot stage in parallel jobs shows as "PivotPX Stage" while the Pivot Enterprise stage shows as "PXPivot Stage". Their icons on the design area are identical.

ray.wurlod · Post by **ray.wurlod** » Tue Jun 09, 2009 9:54 pm

As a test, change BigInt to VarChar throughout, to see whether this really is the cause of the arithmetic overflow.

nagarjuna · Post by **nagarjuna** » Tue Jun 09, 2009 9:58 pm

Thanks ray for your reply ....when i changed it as varchar job is running fine . Its not throwing any error .For , BigInt only its giving problem.

As i am working in 8.0.1 version , i am using server pivot stage in parallel jobs ...Thanks ray for your info about pivot enterprise stage in 8.1 version .

chulett · Post by **chulett** » Tue Jun 09, 2009 10:02 pm

Both stages are in the 8.1 version, and it's not the "server pivot stage" - that's completely different, so perhaps I should say all three stages. It is the original PX pivot stage rather than the exciting new Enterprise version introduced in 8.1.

I'll see if I can try the other one tomorrow.

nagarjuna · Post by **nagarjuna** » Tue Jun 09, 2009 10:07 pm

Thanks for your time craig ...If you get any chance request you to try with pivot ( which is present in server jobs ) ....

Anyways , I am running that particular job with varchar datatype .Just want to make sure with experts in dsxchange about BigInt datatype in pivot stage .

chulett · Post by **chulett** » Tue Jun 09, 2009 10:09 pm

So... are you looking for someone to test this in a Server job? That's the only way one could use the pivot stage "which is present in server jobs".

nagarjuna · Post by **nagarjuna** » Tue Jun 09, 2009 10:13 pm

No craig ....I am using a parallel job .But , i took pivot stage from the folder server type .As far as i know , we have only one pivot stage in 8.0.1 .Tomorrow i will confirm the same and avoid further ambiguity .

chulett · Post by **chulett** » Wed Jun 10, 2009 7:01 am

nagarjuna wrote:No craig ....I am using a parallel job .But , i took pivot stage from the folder server type .As far as i know , we have only one pivot stage in 8.0.1 .

That could very well be your problem right there. Do double-check.

nagarjuna · Post by **nagarjuna** » Wed Jun 10, 2009 8:33 am

Hi Craig ,

Just now i have checked the job .when i looked at the stage tab ( double clicked the pivot stage ) Stagetype : PivotPX .

chulett · Post by **chulett** » Wed Jun 10, 2009 9:07 am

Good. When I get a chance I'll try using that stage and see if anything comes of it. For whatever that is worth.

chulett · Post by **chulett** » Wed Jun 10, 2009 11:45 am

8.1 on Linux, swapping in the 'PX' pivot in place of the Enterprise Pivot gets me a "Error: Arithmetic overflow" warning before the fatal errors start to be logged:

Error occurred in call to ORPHCallActivePluginRun() with input link '0'
The runLocally() of the operator failed.

All reported by the pivot stage. Also found this earlier warning interesting:

main_program: Invalid character(s) ([xA9]) found converting string (code point(s): IBM Confidential, OCO Source Materials, [xA9] Copyright IBM Corp. 2006. All Rights R...) from codepage UTF-8 to Unicode, substituting.

A problem with the copyright symbol it seems but not sure how that string gets pulled into the mixture.

chulett · Post by **chulett** » Wed Jun 10, 2009 12:37 pm

Note that it works fine as long as none of the values exceed the range of an Integer, once a single value goes beyond that into BigInt territory - overflow. This... 'limitation' does not exist in the Enterprise stage, btw.

nagarjuna · Post by **nagarjuna** » Wed Jun 10, 2009 2:23 pm

Hi Craig ,
Thank you very much for confirming the error in pivot stage .