Get File name in External source stage

[Roopanwita]

Hi,

I am using External Source stage to read XML file through DataStage. My question is: when we are reading any(CSV/TXT) file using Sequential File stage, we can set Source File Column ='True' and get source file name , is there any such option to read source file name while reading XML file?

I am trying to implement this logic DataStage, is there any work around other than scripts .

Thank you in advance .

[chulett]

A Server job using the Folder stage can get that for your XML, that or a Server Shared Container with the same in your parallel job.

[eostic]

I'm confused. The technique to read xml using the External Source "is" using the filename --- that's how it works...you issue an operating system command like unix list (ls) and send only the filenames downstream.....

...is that what you want?

Ernie

[Roopanwita]

Thank you for your reply chulett.

Becuase of some function requirement, I can't go for server job. Just wanted to check if it can be implemented in Parallel job.

Thank you,

[chulett]

As Ernie noted, you have the name of the file as that is what the ESS passes to the XML Input stage. Have you tried simply passing it on?

[Roopanwita]

Thank you for response.

I am reading XML files (multiple files) using ESS and in Transformer, I got file names using count function.

I was mapping Trnasformer to XML Input stage (I loaded the XSD).

What do I need to pass there in Output column derivatio ? As I am unable to pass that file name value here, as all XML value field I am mapping XSD name like (/xml:test/xml:col1/text())

Thanks,

[chulett]

OK... not sure why you would use count to get the filenames but regardless you have them. You must also have it in the Transformer as you need to pass it to the XML Input stage so it knows what file to read, yes?

Clarify for us what exactly it is you need to do with this filename, where it needs to go. You can create a field in the XML Input stage for it and leave the Description / XPath Expression blank which tells the stage to just pass it through. From there it can go wherever you like.

Does that help?

[srinivas.nettalam]

Apparently from ESS stage you are counting the '/' and taking out filename part only and passing to the XML input stage. But the usual way is to use 'ls /path/filename.xml' and pass the absolute path of the filename to the XML input stage.

[Roopanwita]

Hi,

Thank you for response.

We do get File_name with path ESS stage, so used count to find 1st \ and get only file name.

My design

ESS --> Transformer --> XML Input stage--> Peek

ESS: reading file using specific programme (ls -l)
Transformer : 2 fields , 1 input file_name(getting from ESS) and 2nd getting only filename(created in transformer)
XML input : has 2 fields from Transformer and out columns are as per XSD with XPath , and also File_name from Input (without any derivation)

Unable to get File_name field( going as NULL)

Is there any property in XML Input stage to propagate the value from source .All fileds mapped from XSD are getting populated correctly.

Thank you .

[eostic]

I can't think of a reason why it wouldn't work....the filename column (or any other column) on the input link is available just like any other for normal mapping at the output step.

As a test, just to see if it has something to do with your xsd structure, create a whole new job that just passes the filename into the xml stage...use only an input step and an output step. Map the filename column from the transformer to the output step and then go to peek or some other target.

I just tried one like that it and it worked fine (I'm using 9.1) --- move into something more complex from there or compare it to what you have now.

Ernie

[chulett]

Your XML is being parsed correctly so the full pathname must be coming in from the ESS correctly or the XML Input stage wouldn't be reading it. I suspect this bit:

Transformer : 2 fields , 1 input file_name(getting from ESS) and 2nd getting only filename(created in transformer)

I would take a hard look at whatever it is you are doing in the transformer to pull the filename from the full path. Worst case post your derivation here.

[Roopanwita]

Thank for reply.

I tried to remove all other XML fields just keeping File_name filed from input to output of XML Input Stage. Job is failing with below error The link contains no derivation rules

But Immediate after ESS from Transformer I have added a Peek , it shows filename corrcetly.

Thank You

[srinivas.nettalam]

Would you post the sample value of the filename you are getting from ESS and the derivation in the transformer??

[Roopanwita]

Hi,

I am able to get file name in ES stage , but actually I am unable to parse it through XML input stage. In XML input stage's output , I have created a field FILE_NAME (same as Input column name) , it doesn't have any derivation (1st field) then I have all XML fileds (created by XSD). It is able to parse XML data , but FILE_NAME is going NULL (empty, i.e unable to parse file_name)

Can you please let me know what is the issue here .

Thank you in advance.

[srinivas.nettalam]

...I have created a field FILE_NAME (same as Input column name) , it doesn't have any derivation (1st field)....

It doesn't have any derivation means you are passing '' or simply mapping the input file_name column?