Get File name in External source stage
Moderators: chulett, rschirm, roy
-
- Participant
- Posts: 125
- Joined: Mon Sep 11, 2006 4:22 am
- Location: India
Get File name in External source stage
Hi,
I am using External Source stage to read XML file through DataStage. My question is: when we are reading any(CSV/TXT) file using Sequential File stage, we can set Source File Column ='True' and get source file name , is there any such option to read source file name while reading XML file?
I am trying to implement this logic DataStage, is there any work around other than scripts .
Thank you in advance .
I am using External Source stage to read XML file through DataStage. My question is: when we are reading any(CSV/TXT) file using Sequential File stage, we can set Source File Column ='True' and get source file name , is there any such option to read source file name while reading XML file?
I am trying to implement this logic DataStage, is there any work around other than scripts .
Thank you in advance .
I'm confused. The technique to read xml using the External Source "is" using the filename --- that's how it works...you issue an operating system command like unix list (ls) and send only the filenames downstream.....
...is that what you want?
Ernie
...is that what you want?
Ernie
Ernie Ostic
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
-
- Participant
- Posts: 125
- Joined: Mon Sep 11, 2006 4:22 am
- Location: India
-
- Participant
- Posts: 125
- Joined: Mon Sep 11, 2006 4:22 am
- Location: India
Thank you for response.
I am reading XML files (multiple files) using ESS and in Transformer, I got file names using count function.
I was mapping Trnasformer to XML Input stage (I loaded the XSD).
What do I need to pass there in Output column derivatio ? As I am unable to pass that file name value here, as all XML value field I am mapping XSD name like (/xml:test/xml:col1/text())
Thanks,
I am reading XML files (multiple files) using ESS and in Transformer, I got file names using count function.
I was mapping Trnasformer to XML Input stage (I loaded the XSD).
What do I need to pass there in Output column derivatio ? As I am unable to pass that file name value here, as all XML value field I am mapping XSD name like (/xml:test/xml:col1/text())
Thanks,
OK... not sure why you would use count to get the filenames but regardless you have them. You must also have it in the Transformer as you need to pass it to the XML Input stage so it knows what file to read, yes?
Clarify for us what exactly it is you need to do with this filename, where it needs to go. You can create a field in the XML Input stage for it and leave the Description / XPath Expression blank which tells the stage to just pass it through. From there it can go wherever you like.
Does that help?
Clarify for us what exactly it is you need to do with this filename, where it needs to go. You can create a field in the XML Input stage for it and leave the Description / XPath Expression blank which tells the stage to just pass it through. From there it can go wherever you like.
Does that help?
-craig
"You can never have too many knives" -- Logan Nine Fingers
"You can never have too many knives" -- Logan Nine Fingers
-
- Participant
- Posts: 134
- Joined: Tue Jun 15, 2010 2:10 am
- Location: Bangalore
-
- Participant
- Posts: 125
- Joined: Mon Sep 11, 2006 4:22 am
- Location: India
Hi,
Thank you for response.
We do get File_name with path ESS stage, so used count to find 1st \ and get only file name.
My design
ESS --> Transformer --> XML Input stage--> Peek
ESS: reading file using specific programme (ls -l)
Transformer : 2 fields , 1 input file_name(getting from ESS) and 2nd getting only filename(created in transformer)
XML input : has 2 fields from Transformer and out columns are as per XSD with XPath , and also File_name from Input (without any derivation)
Unable to get File_name field( going as NULL)
Is there any property in XML Input stage to propagate the value from source .All fileds mapped from XSD are getting populated correctly.
Thank you .
Thank you for response.
We do get File_name with path ESS stage, so used count to find 1st \ and get only file name.
My design
ESS --> Transformer --> XML Input stage--> Peek
ESS: reading file using specific programme (ls -l)
Transformer : 2 fields , 1 input file_name(getting from ESS) and 2nd getting only filename(created in transformer)
XML input : has 2 fields from Transformer and out columns are as per XSD with XPath , and also File_name from Input (without any derivation)
Unable to get File_name field( going as NULL)
Is there any property in XML Input stage to propagate the value from source .All fileds mapped from XSD are getting populated correctly.
Thank you .
I can't think of a reason why it wouldn't work....the filename column (or any other column) on the input link is available just like any other for normal mapping at the output step.
As a test, just to see if it has something to do with your xsd structure, create a whole new job that just passes the filename into the xml stage...use only an input step and an output step. Map the filename column from the transformer to the output step and then go to peek or some other target.
I just tried one like that it and it worked fine (I'm using 9.1) --- move into something more complex from there or compare it to what you have now.
Ernie
As a test, just to see if it has something to do with your xsd structure, create a whole new job that just passes the filename into the xml stage...use only an input step and an output step. Map the filename column from the transformer to the output step and then go to peek or some other target.
I just tried one like that it and it worked fine (I'm using 9.1) --- move into something more complex from there or compare it to what you have now.
Ernie
Ernie Ostic
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
Your XML is being parsed correctly so the full pathname must be coming in from the ESS correctly or the XML Input stage wouldn't be reading it. I suspect this bit:
I would take a hard look at whatever it is you are doing in the transformer to pull the filename from the full path. Worst case post your derivation here.Roopanwita wrote:Transformer : 2 fields , 1 input file_name(getting from ESS) and 2nd getting only filename(created in transformer)
-craig
"You can never have too many knives" -- Logan Nine Fingers
"You can never have too many knives" -- Logan Nine Fingers
-
- Participant
- Posts: 125
- Joined: Mon Sep 11, 2006 4:22 am
- Location: India
-
- Participant
- Posts: 134
- Joined: Tue Jun 15, 2010 2:10 am
- Location: Bangalore
-
- Participant
- Posts: 125
- Joined: Mon Sep 11, 2006 4:22 am
- Location: India
Hi,
I am able to get file name in ES stage , but actually I am unable to parse it through XML input stage. In XML input stage's output , I have created a field FILE_NAME (same as Input column name) , it doesn't have any derivation (1st field) then I have all XML fileds (created by XSD). It is able to parse XML data , but FILE_NAME is going NULL (empty, i.e unable to parse file_name)
Can you please let me know what is the issue here .
Thank you in advance.
I am able to get file name in ES stage , but actually I am unable to parse it through XML input stage. In XML input stage's output , I have created a field FILE_NAME (same as Input column name) , it doesn't have any derivation (1st field) then I have all XML fileds (created by XSD). It is able to parse XML data , but FILE_NAME is going NULL (empty, i.e unable to parse file_name)
Can you please let me know what is the issue here .
Thank you in advance.
-
- Participant
- Posts: 134
- Joined: Tue Jun 15, 2010 2:10 am
- Location: Bangalore