Dataset column metatdata quirk

bidsadw · Post by **bidsadw** » Thu May 19, 2011 9:30 am

Hi

I have a job taht unloads data from a udb table into a dataset.

job layout as follows

DB2Enterprise ( Uses custom SQL )
Stage
|
|
|
Transformer ( reformats timestamps )
Stage
|
|
|
Dataset Stage

Note that in the project I am workiing in, RCP is turmed off at the project level

The job is MULTI-INSTANCE

Job runs clena with no warmoings or informational messages.

The layout of the columns in the Dataset Stage is expected to be as follows

ColA Integer 16
ColB Decimal (18,0)
ColC timestamp[microseconds]

I got to Dataset Managemnt and see this instead

ColC timestamp[microseconds]
ColA Integer 16
ColB Decimal (18,0)

I view the data in the in the dataset and it is diplayed as follwos

ColA ColB ColC
1000 123456. 2008-06-20 10:52:06.000000

I have looked at everything I can think off.

I even created a brand new job on a totally new PX canavas, same result.

The only reason I am concerned wit this is taht I am tryiny to append to this dataset in a different job and I get this warning:

XfmTXXCA09: When checking operator: On output data set 0: Data set (File data set, file "{0}".; output of "APT_TransformOperatorImplV0S22_XfmTXXCA09_Part2_Refresh_XfmTXXCA09 in XfmTXXCA09") has a preset schema that differs from the propagated schema;
propagated fields not in the existing schema will be dropped;
fields in the existing schema but not propagated will be defaulted.

I did search in this forum for this warning and followed the various suggestions but no joy.

Someone in my area did suggest that the Transformer may be the culprit but I cannot see how.

Any help would be appreciated

ray.wurlod · Post by **ray.wurlod** » Thu May 19, 2011 4:59 pm

Manually delete the Data Set before the next run.

bidsadw · Post by **bidsadw** » Thu May 19, 2011 5:06 pm

Tried deleting the dataset and rerun the jobs, nothing changes.

When you say manulaly delete, are you suggesting not using Dataset Managment to delet the dataset?

ray.wurlod · Post by **ray.wurlod** » Fri May 20, 2011 3:46 pm

Yes. That would remove any inconsistent record schema in any Data Set created earlier.

bidsadw · Post by **bidsadw** » Sun May 22, 2011 8:07 am

Partial resolution on the warning that was generated on the append to the dataset in question

XfmTXXCA09: When checking operator: On output data set 0: Data set (File data set, file "{0}".; output of "APT_TransformOperatorImplV0S22_XfmTXXCA09_Part2_Refresh_XfmTXXCA09 in XfmTXXCA09") has a preset schema that differs from the propagated schema;
propagated fields not in the existing schema will be dropped;
fields in the existing schema but not propagated will be defaulted.

Turns out that even though RCP was disabled at the project level, it did not disable it in the jobs that was actually appending to te dataset.

I had the RCP turned on at the project level, went in to job properties and disbaled RCP there.

Dataset can no be appended to with out issues.

I did as ray.wurlod suggested and issued an rm *.ds command in the dataset directory and rearn the job. The colimn metedata is still messed up.

I was under the impression that deleteing datasets was only supposed to be done via the Dataset Mgmt tool. Is this correct?

pandeesh · Post by **pandeesh** » Sun May 22, 2011 10:25 am

Yes.

Using rm command to remove the Dataset simply removes the descriptor file but not tthe larger data files.

So it's well and good to use Dataset management tool.