How to load large XML files with External Source Stage

[jpnrd]

Hi

I am trying to create one parallel job where we will read one XML file as input into a XML input stage for further processing.
I used one sequential file stage where we read that entire XML file "test.xml" as a stream input into one single column "XML Record" of that sequential file with Record Type as implicit and Field Delimiter as none in Format page.
Then we have fed this column into XML Input Stage.

But the problem which I am facing is if the "test.xml" file size is huge (say more than 500 bytes) the input xml data is getting truncated into two rows in the sequential file and hence we are not able to feed those data into XML Input Stage.

Now I am trying to use the External Source stage to resolve this problem. Can anybody tell me how to use external source stage in present scenario.
What are the settings I need to make in the External Source stage?

[*Note - Title changed to be more descriptive - Andy*]

[chulett]

Have you looked here yet?

[jpnrd]

Craig,

I looked at it. Its really Helpful. I dont why its not working for me. My sever is in D:/, is that the problem. If thats the problem Please tell me a solution.

[jpnrd]

I kept all xml files in a shred folder and tried using shared folder. I can get the link out of the External Source Stage and passed it to XML input stage. The job ran well but 0 record are imported. It has only one warning


XML_Input_3,0: Warning: XMLTest4.XML_Input_3: XML input document parsing failed. Reason: Xalan fatal error (publicId: , systemId: , line: 0, column: 0): An exception occurred! Type:RuntimeException, Message:The primary document entity could not be opened. Id=D:\IBM\InformationServer\Server\Projects\ETL_DEV///skfb234/sharedata/EnterpriseDataArchitecture/ODS/PolicyCenterXMLdata/KFB_Liability_APIPCov.xml


Can you please help me with this warning.

[eostic]

Do some careful debugging of the "list command" that you are using in the External Source Stage. I am surprised to see it appending the DS Project directory....... run the External Source to a flat file instead of XML...what do you get? You should have a complete list of your xml docs..nothing else........

Also..check the check box on the input link side of the xml stage...when you use External Source you must check URI/URL, not XML Content....perhaps, if you were using your old job, you still have XML Content checked......

Ernie

[arunkumarmm]

Hi

I am trying to create one parallel job where we will read one XML file as input into a XML input stage for further processing.
I used one sequential file stage where we read that entire XML file "test.xml" as a stream input into one single column "XML Record" of that sequential file with Record Type as implicit and Field Delimiter as none in Format page.
Then we have fed this column into XML Input Stage.

But the problem which I am facing is if the "test.xml" file size is huge (say more than 500 bytes) the input xml data is getting truncated into two rows in the sequential file and hence we are not able to feed those data into XML Input Stage.

Now I am trying to use the External Source stage to resolve this problem. Can anybody tell me how to use external source stage in present scenario.
What are the settings I need to make in the External Source stage?

[*Note - Title changed to be more descriptive - Andy*]

You can use sequential file stage and define your column as LongVarChar - 999999999. This will work.

[chulett]

No... not really. At least not well.