Problem with sequential file and file name column
Moderators: chulett, rschirm, roy
Problem with sequential file and file name column
I have a strange problem when reading files in sequential stage.
I read 3 files 090309.sdi,090310.sdi and 090311.sdi with pattern file option.
I use the option file name column to associate each line with the corresponding file.
But, some lines in the 090309.sdi are affected to the 090311.sdi file.
Do you have any idea on what happen?
thanks,
Martin.
I read 3 files 090309.sdi,090310.sdi and 090311.sdi with pattern file option.
I use the option file name column to associate each line with the corresponding file.
But, some lines in the 090309.sdi are affected to the 090311.sdi file.
Do you have any idea on what happen?
thanks,
Martin.
-
- Participant
- Posts: 3337
- Joined: Mon Jan 17, 2005 4:49 am
- Location: United Kingdom
-
- Premium Member
- Posts: 783
- Joined: Mon Jan 16, 2006 10:17 pm
- Location: Sydney, Australia
Well, my logic is the following
I take one or more files in input and I sort them by date. I remove duplicate by taking the more recent line.
So, I have 2 lines with the same key in the 090309 and in the 090311, but the line in the 090309 is affected to the 090311, so my remove duplicate becomes wrong.
to Sainath.Srinivasan, effectivly the rows differ. I obtain this
File ---- row in the file ---- rows in Datastage
090309.sdi ---- 2 320 822 ---- 2 320 815 (-7)
090310.sdi ---- 2 287 527 ---- 2 287 527 (ok)
090311.sdi ---- 2 213 218 ---- 2 213 325 (+7)
so, i identified the lines. It's the last 7 lines of the file 090309.sdi that are affected to the 090311.sdi file.
I take one or more files in input and I sort them by date. I remove duplicate by taking the more recent line.
So, I have 2 lines with the same key in the 090309 and in the 090311, but the line in the 090309 is affected to the 090311, so my remove duplicate becomes wrong.
to Sainath.Srinivasan, effectivly the rows differ. I obtain this
File ---- row in the file ---- rows in Datastage
090309.sdi ---- 2 320 822 ---- 2 320 815 (-7)
090310.sdi ---- 2 287 527 ---- 2 287 527 (ok)
090311.sdi ---- 2 213 218 ---- 2 213 325 (+7)
so, i identified the lines. It's the last 7 lines of the file 090309.sdi that are affected to the 090311.sdi file.
sorry, my mother tongue is french and not english.chulett wrote:Affected to? To quote The Princess Bride: You keep using that word. I do not think it means what you think it means. Could you please clarify what exactly you mean by 'affected to' in this context?
well, in mean, in the sequential stage, I defined the option 'file name column'=FILEMAME
so, all the lines of the 090309.sdi should have the value '/pathfile/090309.sdi'
and all the lines of the 090310.sdi should have the value '/pathfile/090310.sdi' and so on...
but for 7 lines of the 090309.sdi files the FILENAME value is '/pathfile/090311.sdi'
I hope it is more clear this time.
No worries, you wouldn't want to see my French.
The example helps alot. So... does this problem seem to be coming directly from the Sequential File stage itself or is it being introduced in later processing, say the 'remove duplicates' step you mentioned? I've seen posts about odd problems with that 'filename' option, perhaps the environment variable mentioned in this post might help:
viewtopic.php?t=121263
The example helps alot. So... does this problem seem to be coming directly from the Sequential File stage itself or is it being introduced in later processing, say the 'remove duplicates' step you mentioned? I've seen posts about odd problems with that 'filename' option, perhaps the environment variable mentioned in this post might help:
viewtopic.php?t=121263
-craig
"You can never have too many knives" -- Logan Nine Fingers
"You can never have too many knives" -- Logan Nine Fingers
well, this environement is already set to TRUE. And for my test, I suppress the remove duplicate operator. It seems to become directly from the Sequential File Stage itself.chulett wrote:No worries, you wouldn't want to see my French.
The example helps alot. So... does this problem seem to be coming directly from the Sequential File stage itself or is it being introduced in later processing, say the 'remove duplicates' step you mentioned? I've seen posts about odd problems with that 'filename' option, perhaps the environment variable mentioned in this post might help:
viewtopic.php?t=121263
-
- Participant
- Posts: 3337
- Joined: Mon Jan 17, 2005 4:49 am
- Location: United Kingdom