Hi,
Would like know if DSv7.x able to read CDR(call detail record) raw data which is in binary format?
Or there must be a decoding task prior to DS?
~SC Yong
Can DSv7 read CDR raw data?
Moderators: chulett, rschirm, roy
Hello SC,
the answer is a qualified "yes". "CDR" as you stated isn't an ANSI or DIN standard, so we can't really know what format it is coming in as.
Binary data as it applies to integers can be handled easily, as that is standardized (at least for positive numbers). Binary floating point in ANSI 74 format can be read as well (the exponent/mantissa formats and lengths are standardized). Other binary formats can be processed from within DataStage in user-written functions; the internal language allows you to perform actions such as
- convert a binary data string into a string with binary, octal and hex representations
- manipulate at bit & byte level
- convert it back into a binary representation or into a string numeric format
In order to give a definitive answer, you would need to explain a bit more about the binary representation and perhaps the record layout.
the answer is a qualified "yes". "CDR" as you stated isn't an ANSI or DIN standard, so we can't really know what format it is coming in as.
Binary data as it applies to integers can be handled easily, as that is standardized (at least for positive numbers). Binary floating point in ANSI 74 format can be read as well (the exponent/mantissa formats and lengths are standardized). Other binary formats can be processed from within DataStage in user-written functions; the internal language allows you to perform actions such as
- convert a binary data string into a string with binary, octal and hex representations
- manipulate at bit & byte level
- convert it back into a binary representation or into a string numeric format
In order to give a definitive answer, you would need to explain a bit more about the binary representation and perhaps the record layout.
</a>
If your CDR's are in ASN.1 format, then the best solution is to use an ASN.1 decoder, implemented in some other language (C++, Perl, Java, etc.). AFAIK, DataStage does not natively support ASN.1.
Trying to do one yourself in DataStage will likely not result in the kind of performance that CDR processing typically requires.
OTOH, if they are simply bit encoded messages directly from the switch, then you can process them by AND'ing, OR'ing, and bit shifting. It would help if you have done some assembly language recently.
Of course, a simple C program will probably run much quicker, if you are handling large volumes.
Carter
Trying to do one yourself in DataStage will likely not result in the kind of performance that CDR processing typically requires.
OTOH, if they are simply bit encoded messages directly from the switch, then you can process them by AND'ing, OR'ing, and bit shifting. It would help if you have done some assembly language recently.
Of course, a simple C program will probably run much quicker, if you are handling large volumes.
Carter
Hi,
if your binary data is complex:
IMHO, your best solution would be to build a custom stage.
I run to a situation once that handling binary data with DS basic missed some info, I think it was a 05(in hex vaule) that made the index counter advance rsulting in a skip and missing data, who knows if and what else you might come across.
the other option would be to run a utility that transforms this to an ascii format data and continue from there.
(this reminds me of past questions like can DS handle pdf files and such)
IHTH,
if your binary data is complex:
IMHO, your best solution would be to build a custom stage.
I run to a situation once that handling binary data with DS basic missed some info, I think it was a 05(in hex vaule) that made the index counter advance rsulting in a skip and missing data, who knows if and what else you might come across.
the other option would be to run a utility that transforms this to an ascii format data and continue from there.
(this reminds me of past questions like can DS handle pdf files and such)
IHTH,
Roy R.
Time is money but when you don't have money time is all you can afford.
Search before posting:)
Join the DataStagers team effort at:
http://www.worldcommunitygrid.org
Time is money but when you don't have money time is all you can afford.
Search before posting:)
Join the DataStagers team effort at:
http://www.worldcommunitygrid.org
-
dsxuserrio
- Participant
- Posts: 82
- Joined: Thu Dec 02, 2004 10:27 pm
- Location: INDIA
I think the raw data comes with different input formats. As suggested by other users create a custom stage which reads the input format as an xml file. Let the custom operator parse the input based on the xml file and produce outputs in different ports .(Output datsets from C++ runlocally method)
This way when there is a new data format , you only need to change the xml file and not the custom oerator.
Thanks
This way when there is a new data format , you only need to change the xml file and not the custom oerator.
Thanks
dsxuserrio
Kannan.N
Bangalore,INDIA
Kannan.N
Bangalore,INDIA
in my personal expreince
my personal experience.
the company I 've worked in has many CDR types, some were ANSI,BINARY (ugly ones
), and some were ASN.1 (uglyest 
)..
So how we did load them to Oracle DB?
we have build a C++ code, where each switch has a unique library/headers that read and convert it to CVS files. Moreover the c++ code will fire DS job against the decoded CDR which in its turn load it to Oracle.
we end up with multi-tier CDRs processing structure.
the company I 've worked in has many CDR types, some were ANSI,BINARY (ugly ones
), and some were ASN.1 (uglyest )..So how we did load them to Oracle DB?
we have build a C++ code, where each switch has a unique library/headers that read and convert it to CVS files. Moreover the c++ code will fire DS job against the decoded CDR which in its turn load it to Oracle.
we end up with multi-tier CDRs processing structure.