Validating the file name and column names
Moderators: chulett, rschirm, roy
Validating the file name and column names
Hello everyone
I need to read a CSV file as source and need to validate whether the file is in CSV format.
The columns should be validated whether they are in the required order or not.
The column names with should be validated with certain names.
Please suggest a way to make it doable.
Thanks
I need to read a CSV file as source and need to validate whether the file is in CSV format.
The columns should be validated whether they are in the required order or not.
The column names with should be validated with certain names.
Please suggest a way to make it doable.
Thanks
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Assuming the first line is column names, I guess you could (a) validate that all the column names are valid and (b) establish the expected order of fields in the data records when processing that first line.
The expected number of fields is also established from the first line, but counting the comma delimiters. If this is non-zero you don't have a CSV file (or, perhaps, you have a CSV file with one field - do you need to be able to handle this possibility?).
If your requirement that columns may be in arbitrary order applies within the file, then there is no generic solution. Indeed there is probably no solution at all unless there is some kind of per-row metadata (such as XML tags, for example).
The expected number of fields is also established from the first line, but counting the comma delimiters. If this is non-zero you don't have a CSV file (or, perhaps, you have a CSV file with one field - do you need to be able to handle this possibility?).
If your requirement that columns may be in arbitrary order applies within the file, then there is no generic solution. Indeed there is probably no solution at all unless there is some kind of per-row metadata (such as XML tags, for example).
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Re: Validating the file name and column names
You can design like below also.
You can use either TFM | Filter for the first one.
Code: Select all
MasterFile2ChkCol
|
|
--Col headerAlone-- --------LKUP----REJ-----TFM (Abort after 1 row)
SeqFile (Read including header) -->TFM|
--Data Alone -------PROCESS AS YOU WISH
Thanks
Ram
----------------------------------
Revealing your ignorance is fine, because you get a chance to learn.
Ram
----------------------------------
Revealing your ignorance is fine, because you get a chance to learn.
use Reject link in the Sequential file.rahul8055 wrote:Is there any way where I can check whether the file is CSV or not in datastage?
Code: Select all
SeqFile----(RejLink)---TFM-(Abort after 1 row)--peek
Thanks
Ram
----------------------------------
Revealing your ignorance is fine, because you get a chance to learn.
Ram
----------------------------------
Revealing your ignorance is fine, because you get a chance to learn.
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Existence of the file is more easily checked using the test command in an Execute Command activity in a sequence.
Validity as CSV can be done in any of the ways we've already suggested, as well as operating system commands (such as grep for any lines that do not contain any commas, piping the output of that through wc -l command and verifying that the count ($CommandOutput) is 0).
Validity as CSV can be done in any of the ways we've already suggested, as well as operating system commands (such as grep for any lines that do not contain any commas, piping the output of that through wc -l command and verifying that the count ($CommandOutput) is 0).
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.