Processing XML files

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
mydsworld
Participant
Posts: 321
Joined: Thu Sep 07, 2006 3:55 am

Processing XML files

Post by mydsworld »

If I need to FTP (get) a XML file from remote server and process, how should my job look like :

FTP Ent -> XML Input -> Seq File
eostic
Premium Member
Premium Member
Posts: 3838
Joined: Mon Oct 17, 2005 9:34 am

Post by eostic »

I haven't tried it, but I suspect it "might" work with some documents but fail with others because the FTP stages in DataStage have historically been "streaming" mechanisms. You need to send a "whole" document into XMLInput....if in streaming, the FTP Stage "thinks" it has a single record (if it sees what it things is an end-of-row indicator), it will send it. If that's not the entire file, then XMLInput will fail. Perhaps someone with a lot more experience with the FTP ENterprise Stage can chime in....it's the behavior of that Stage that will have a large impact here.

Ernie
Ernie Ostic

blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

So a two-job solution may be preferable, or a script to do the FTP to a local file that you can then process through the XML Input stage.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
rameshrr3
Premium Member
Premium Member
Posts: 609
Joined: Mon May 10, 2004 3:32 am
Location: BRENTWOOD, TN

Post by rameshrr3 »

Use a Before Job Subroutine and call and automated FTP or SCP script to copy to a local folder, then use External Source to cat the file contents to its output link and pass this output link to XML input
mydsworld
Participant
Posts: 321
Joined: Thu Sep 07, 2006 3:55 am

Post by mydsworld »

I have requirement where Datastage is just needed to FTP a XML file from some source server to target server and do no processing of the XML.

Whats the most efficient way to do this. What should I use in FTP stage metadata. Will it be XSDL definition or just a huge size varchar column ?
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

Ugh. Use DataStage to run command line ftp for this... or "just a huge size varchar column".
-craig

"You can never have too many knives" -- Logan Nine Fingers
Post Reply