Read XML file in PX

[Mat01]

Hi All,

I am trying to use the XML stages to read and transform XML data. I was able to read my file in Server (Thanks Kim for your ETLstats examples) but I haven't found a way to do this in PX. The main problem seems to be that I cannot read an entire file as one single field in PX. Hence, The XML input stage cannot parse the XML data. I have tried passing the stage a file name as specified in the XML source tag but this doesn't work either (job succesful, but empty output).

Has anybody found a way to read XML files with PX?

Thanks,

MAT

[jatinrheen]

Hi,

We have Transactional MQ from which we read the input XML , and then use COLUMN IMPORT in order to shift this input to a column name .Then Use XML Input / Tranformer stage in order to parse it .

Let me know if you have any problems so that I can send you the dsx.

Regards

Hi All,

I am trying to use the XML stages to read and transform XML data. I was able to read my file in Server (Thanks Kim for your ETLstats examples) but I haven't found a way to do this in PX. The main problem seems to be that I cannot read an entire file as one single field in PX. Hence, The XML input stage cannot parse the XML data. I have tried passing the stage a file name as specified in the XML source tag but this doesn't work either (job succesful, but empty output).

Has anybody found a way to read XML files with PX?

Thanks,

MAT

[Mat01]

Thank you for your reply,

I'm not sure I follow you...How do you the use of the column import? I'd like to see an example if you can.

Thanks

Mat

[michaeld]

This is how I did it:

Using a sequential file stage:

1)

Create a longvarchar (or longnvarchar) column that will hold your xml.

2)set format properties as follows:

record level->Record type = implicit
field level->Delimiter=none

NOTE: Make sure that you remove all other record level or field level properties!

This will put all of the data in the source file to a single column in a single row.

no charge:-)

[vmcburney]

IBM have published a tutorial on using the XML input stage in parallel jobs. See my blog post for details: 7 DataStage Tutorials.

[JoshGeorge]

To avoid "record is too big to fit in a block" error, APT_DEFAULT_TRANSPORT_BLOCK_SIZE need to be set to the double of maximum possible message size expected.

The main problem seems to be that I cannot read an entire file as one single field in PX. Hence, The XML input stage cannot parse the XML data

[chulett]

I have tried passing the stage a file name as specified in the XML source tag but this doesn't work either (job succesful, but empty output).

I'll wager you read the XML file just fine with that setup. Stuff in with no stuff out is generally a sign of improper XPath expressions in the XML Input stage.

[vmcburney]

I haven't got it to work just passing the file name, I've usually passed the contents as well. You might need to tweak your imported XML definition such as switching the key field. You might need to test on smaller files to see if it is valid XML and not a volume problem.

[chulett]

Really? It certainly can be done in a Server job, don't see why that aspect of this would be different in PX. [shrug]

It's the exact same 'XML Input' stage in both products, yes? Not a PX specific flavor that I am aware of, so sure seems like you should be able to make that work the same way.

Make sure the filename is fully qualified, of course.

[balajisr]

URL/File Path option of XML Input Stage works in PX. But my xml size was small say about 2KB. I need to verify with bigger XML.

[rajesh223]

I have tried to ready the xml by passing the xml path to the sequential fila and downstream to xml_input stage, then followed by the transformer stage and finally to the output sequential file.

I am able to read the data successfully, but there is one problem. As per the standard, all the file name and the location should be parameterized. In this case how can we pass the xml file location into the flat file.

I tried using rowgenerator>> transformer>> xml_input stage

I passed job parameter in the transformer as xml file path, then downstream to xml_input. I am unable to run the job, there is no error every thing looks fine. The data are flowing till transformer and not proceding after xml_inputstage. The xml_input stage unable to read the data from the prior stage (transformer). But it works for sequential file.

any thoughts on this ....

-Rajesh

[eostic]

This subject continues to be a small nightmare. I share everyone's frustration here. Still another way to approach this, and the best solution that I've found so far is to use the ExternalSource Stage, choose "specific program" as the source, and then send in a subdirectory name in job parameter to an "ls"...... here's the property value for Source Program right from a working job:
ls #xmlSubDirectory#*.xml | sort ....that is then passed to a Transformer where (on Windows) I had to manually concatenate a C:.

We need a Stage that does exactly this, or an implementation of the Folder Stage with size flexibility.

Ernie

[ray.wurlod]

There are three different kinds of custom stage you can create in (for) parallel jobs. Any volunteers?

[Maveric]

Ray, can custom, Build and Wrapped be classified as the three different kinds of custom stage? A wild guess.

[ray.wurlod]

Yes, that's them.