I have one input file around 2000 fixed width columns.I have to search text from this columns.
Directly i can't search from multiple columns, so i thought of merging all columns into one single column.This i'm doing by using a ":" operator in stage variable.
E.g:
col1(5) col2(2) col3(3) StageVarcol1:col2:col3)
"abc " "xx" "yy " "abc xxyy "
But the size is so huge it's throwing an error "Run time error(6) overflow"
Is this the right way of merging all columns into one single column?
Please help me out...
Merging columns into single
Moderators: chulett, rschirm, roy
-
- Participant
- Posts: 78
- Joined: Fri Mar 02, 2007 4:54 am
- Location: CHENNAI
-
- Premium Member
- Posts: 301
- Joined: Thu Jul 14, 2005 10:27 am
- Location: Melbourne, Australia
- Contact:
I don't think attempting to concatenate 2000 columns in a stage variable is going to yield useful results - and I wouldn't want to be the guy asked to maintain that solution!
A simple solution would be to read the sequential file as a single column, although you'd have to ensure the column definition can handle the worst case scenario (i.e. the largest potential row width) which I suspect in your case would be rather large.
Another option is to use a Column Export stage to concatenate the columns for you, although you've still got a lot of metadata to manage (You'd have to import your table definition for all 2000 columns.)
Alternatively, depending upon your requirement, you may be able to achieve what you need by pre-processing the file using Unix shell commands, delivering the source row as one column and your items of interest as another (using a different delimiter to the one used in the file, of course.) You could achieve this using an external source stage, or passing the rows through an external filter.
J.
A simple solution would be to read the sequential file as a single column, although you'd have to ensure the column definition can handle the worst case scenario (i.e. the largest potential row width) which I suspect in your case would be rather large.
Another option is to use a Column Export stage to concatenate the columns for you, although you've still got a lot of metadata to manage (You'd have to import your table definition for all 2000 columns.)
Alternatively, depending upon your requirement, you may be able to achieve what you need by pre-processing the file using Unix shell commands, delivering the source row as one column and your items of interest as another (using a different delimiter to the one used in the file, of course.) You could achieve this using an external source stage, or passing the rows through an external filter.
J.
<b>John McKeever</b>
Data Migrators
<b><a href="https://www.mettleci.com">MettleCI</a> - DevOps for DataStage</b>
<a href="http://www.datamigrators.com/"><img src="https://www.datamigrators.com/assets/im ... l.png"></a>
Data Migrators
<b><a href="https://www.mettleci.com">MettleCI</a> - DevOps for DataStage</b>
<a href="http://www.datamigrators.com/"><img src="https://www.datamigrators.com/assets/im ... l.png"></a>
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
If it's fixed width there is no worst case scenario. Read the entire record as a single Char column, then use the Index() function to search for a string in that.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
-
- Participant
- Posts: 78
- Joined: Fri Mar 02, 2007 4:54 am
- Location: CHENNAI