Read a network request and response file

chetan.c · Post by **chetan.c** » Thu Feb 02, 2012 2:31 am

kandyshandy wrote:Little difficult to understand the actual requirements. Can you post one file content and tell us what you want?

File1

StatusCode=500
StatusCodeText=Internal Server Error
RequestHeaderSize=221
RequestDataSize=1696
RequestSize=1917
ResponseHeaderSize=167
ResponseDataSize=3997
ResponseSize=4164

[urlfield]
<
xml version="1.0" encoding="utf-8"
>%0a<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelop ... sageHeader xmlns="http://csi.cingular.com/CSI/Namespaces/ ... Header.xsd" xmlns:cin="http://csi.cingular.com/CSI/Namespaces/ ... l.xsd">%0a <TrackingMessageHeader>%0a <cin:version>v52</cin:version>%0a <cin:messageId>....

File2

StatusCode=200
StatusCodeText=OK
RequestHeaderSize=305
RequestDataSize=2347
RequestSize=2652
ResponseHeaderSize=148
ResponseDataSize=6624
ResponseSize=6772

<Request>
<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/">
<SOAP-ENV:Header>
<mes:MessageHeader xmlns:mes="http://csi.cingular.com/CSI/Namespaces/ ... in:version xmlns:cin="http://csi.cingular.com/CSI/Namespaces/ ... :messageId xmlns:cin="http://csi.cingular.com/CSI/Namespaces/ ... timeToLive xmlns:cin="http://csi.cingular.com/CSI/Namespaces/ ... timeToLive>............

And so on for all files.

kandyshandy · Post by **kandyshandy** » Thu Feb 02, 2012 2:58 am

To answer Muni...

As per my logic, StageVar1 will have StatusCode|StatusCodeText|...etc (pipe symbol was missing in my expression)
Similarly, StageVar2 will be 200|OK|.... etc

I guess Chetan understood.

kandyshandy · Post by **kandyshandy** » Thu Feb 02, 2012 3:05 am

Chetan, Both the files you posted look alike. Would you like to process each file separately or all files at once?.. I am trying to understand your objective.

chetan.c · Post by **chetan.c** » Thu Feb 02, 2012 3:09 am

Yes they are similar.
I want to process all the files at once.

Thanks for your time.

kandyshandy · Post by **kandyshandy** » Thu Feb 02, 2012 3:39 am

Then i believe that it is possible though it is not "that" easy. Note that the below logic works in sequential mode only...

First you need to merge all files. While merging (using UNIX command), add a comment or symbol or trailer after each file including for the last file.

E.g. (i added **** after each file)

StatusCode=500
StatusCodeText=Internal Server Error
RequestHeaderSize=221
RequestDataSize=1696
RequestSize=1917
ResponseHeaderSize=167
ResponseDataSize=3997
ResponseSize=4164

[urlfield]
<
xml version="1.0" encoding="utf-8"
>%0a<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelop ... sageHeader xmlns="http://csi.cingular.com/CSI/Namespaces/ ... Header.xsd" xmlns:cin="http://csi.cingular.com/CSI/Namespaces/ ... l.xsd">%0a <TrackingMessageHeader>%0a <cin:version>v52</cin:version>%0a <cin:messageId>....
****
StatusCode=200
StatusCodeText=OK
RequestHeaderSize=305
RequestDataSize=2347
RequestSize=2652
ResponseHeaderSize=148
ResponseDataSize=6624
ResponseSize=6772

<Request>
<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/">
<SOAP-ENV:Header>
<mes:MessageHeader xmlns:mes="http://csi.cingular.com/CSI/Namespaces/ ... in:version xmlns:cin="http://csi.cingular.com/CSI/Namespaces/ ... :messageId xmlns:cin="http://csi.cingular.com/CSI/Namespaces/ ... timeToLive xmlns:cin="http://csi.cingular.com/CSI/Namespaces/ ... timeToLive>
****

Read each line as fixed width including XML messages.

Create as many stage variables as the number of output columns you want. (your scenario needs 11 stage variables to deal with each column)

e.g.
Define following stage variables...

StatusCode
StatusCodeText
RequestHeaderSize
....
....

Stage variable "StatusCode" -> [If Field(Col,1,'=') = 'StatusCode' Then Field(Col,2,'=') Else StatusCode]

Similarly, define all required stage variables including for XML messages.

When done, create columns in output sequential file. Define the column names similar to what you want in the output. Map each stage variable to its respective output column.

Very important... add a constraint in transfomer [Col1 = '****']

Logic: Stage variables will get respective values while each record is processed in sequence. But no record will be written to output file due to the constraint specified. When **** is encountered, a record will be written to the output file. I guess this is what you wanted.

** First line is column names will give you the header record needed

This can also be achieved by using BASIC language or by writting wrapper in C++.

chetan.c · Post by **chetan.c** » Thu Feb 02, 2012 5:05 am

Hi Kandy,

Developing the job ,will get back with the results.
Thanks for the explanation.

Thanks.

chetan.c · Post by **chetan.c** » Thu Feb 02, 2012 6:55 am

Hi Kandy,

The job is working fine and as required for the data apart from the XML content.

.
The part of getting the coulmn names was excellent.

But can you guide me on how to read the XML messages?.
As i'm not getting the logic for XML content in the stage variables.

Thanks!!

chetan.c · Post by **chetan.c** » Thu Feb 02, 2012 7:05 am

Also is there a way where i can extract the entire XML and pass it to the XML input stage itself and extract values from there?

kandyshandy · Post by **kandyshandy** » Fri Feb 03, 2012 3:10 am

Very much possible but you need to separate the XML content into a stream and input it to XML stage. You need to define Xpath expressions to get a particular XML element/attribute value.

kandyshandy · Post by **kandyshandy** » Fri Feb 03, 2012 3:12 am

XML messages can be read similar to how you read normal lines..

If it is complex, try using index and field functions to get what you want.

chetan.c · Post by **chetan.c** » Fri Feb 03, 2012 3:59 am

XML messages can be read similar to how you read normal lines..

If it is complex, try using index and field functions to get what you want.[/quote]

I have experimented with the XML input stage and am able to read an XML document.

But the problem is how to get only the XML part that is from its starting tag till the ending tag from the files.
I tried a logic where in i did a search for starting tag (Ex:<Request>) it works fine but large number of files reading it into fixed column is over running the memory.

Can you please help with this.

Thanks.

kandyshandy · Post by **kandyshandy** » Fri Feb 03, 2012 4:05 am

You need to paste a sample XML message. Do all elements appear continuosly without new line character? In that case, you can use Index, Len & Field functions to get what you want. If the data can't be held in the length you defined, change the data type to LongVarchar & increase the length. Also, read about BLOCKSIZE environment variable

If each element appears as a new line, then you should be able to find what you want easily.

chetan.c · Post by **chetan.c** » Mon Feb 06, 2012 5:28 am

<Request>
<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/">
<SOAP-ENV:Header>
<mes:MessageHeader xmlns:mes="http://csi.cingular.com/CSI/Namespaces/ ... in:version xmlns:cin="http://csi.cingular.com/CSI/Namespaces/ ... :messageId xmlns:cin="http://csi.cingular.com/CSI/Namespaces/ ... timeToLive xmlns:cin="http://csi.cingular.com/CSI/Namespaces/ ... eTimeStamp xmlns:cin="http://csi.cingular.com/CSI/Namespaces/ ... n:userName xmlns:cin="http://csi.cingular.com/CSI/Namespaces/ ... erPassword xmlns:cin="http://csi.cingular.com/CSI/Namespaces/ ... enceNumber xmlns:cin="http://csi.cingular.com/CSI/Namespaces/ ... InSequence xmlns:cin="http://csi.cingular.com/CSI/Namespaces/ ... sageHeader>
</SOAP-ENV:Header>
<SOAP-ENV:Body>
<inq:InquireFiberServiceAccountDetailsRequest xmlns:inq="http://csi.cingular.com/CSI/Namespaces/ ... ilsRequest>
</SOAP-ENV:Body>
</SOAP-ENV:Envelope>
</Request>

This is the XML.
My requirement is to have this entire XML irespective of value present or not and store this XML.

My Idea is to start extracting from the row where<Request> starts and end on row where </Request> is found.

Is this logic correct and is it possible?

chulett · Post by **chulett** » Mon Feb 06, 2012 8:17 am

That sounds like a good start. Now that we know you want to store rather than parse it, can you tell us what 'store' means here? Where / how does it need to be stored?

ray.wurlod · Post by **ray.wurlod** » Mon Feb 06, 2012 4:17 pm

The logic is correct, and you can accumulate the XML as a single string in a stage variable. Output it once the closing tag has been processed (that is, if the input value is "</Request>").