I have two sequncial files that I need to load into a stage table. The files are both the same format, just two different subsets of data based on time. How do I load both files in one job?
Regards Jim
Two Files One Load
Moderators: chulett, rschirm, roy
-
datastagedummy
- Participant
- Posts: 56
- Joined: Thu Feb 13, 2003 6:08 pm
- Location: USA
quote:Originally posted by datastagedummy
[br]The easiest way is just to merge the files using cat command in unix and feed it to your existing job
im in a windows world. Is there a way to get datastage to do this for me? Im playing with two sequncial stages going into one bulk loader, but I am having a time with that. Any other thoughts?
Jim
[br]The easiest way is just to merge the files using cat command in unix and feed it to your existing job
im in a windows world. Is there a way to get datastage to do this for me? Im playing with two sequncial stages going into one bulk loader, but I am having a time with that. Any other thoughts?
Jim
-
ray.wurlod
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
In UNIX it's cat, in Windows it's either type or copy.
Before the job runs, execute this command (perhaps through a before-job subroutine invoking ExecDOS):
type file1 file2 > file3
Then your job processes file3, which contains all the rows from file1 and file2. If file1 and file2 have header rows, you may require a little more processing.
Or, you can use a Merge stage to read from the two files, specifying "Complete Set" as the join type.
Ray Wurlod
Education and Consulting Services
ABN 57 092 448 518
Before the job runs, execute this command (perhaps through a before-job subroutine invoking ExecDOS):
type file1 file2 > file3
Then your job processes file3, which contains all the rows from file1 and file2. If file1 and file2 have header rows, you may require a little more processing.
Or, you can use a Merge stage to read from the two files, specifying "Complete Set" as the join type.
Ray Wurlod
Education and Consulting Services
ABN 57 092 448 518
quote:Originally posted by JDionne
[br]quote:Originally posted by Ray.Wurlod
[br]In UNIX it's cat, in Windows it's either type or copy.
Before the job runs, execute this command (perhaps through a before-job subroutine invoking ExecDOS):
type file1 file2 > file3
Then your job processes file3, which contains all the rows from file1 and file2. If file1 and file2 have header rows, you may require a little more processing.
Or, you can use a Merge stage to read from the two files, specifying "Complete Set" as the join type.
Ray Wurlod
Education and Consulting Services
ABN 57 092 448 518
I have looked into the merge stage, but i cant figure out how to make it work. Do you know it well enough to give me better detail about how to use it than what is in the help file?
Thank you for all of your help
Jim
[br]quote:Originally posted by Ray.Wurlod
[br]In UNIX it's cat, in Windows it's either type or copy.
Before the job runs, execute this command (perhaps through a before-job subroutine invoking ExecDOS):
type file1 file2 > file3
Then your job processes file3, which contains all the rows from file1 and file2. If file1 and file2 have header rows, you may require a little more processing.
Or, you can use a Merge stage to read from the two files, specifying "Complete Set" as the join type.
Ray Wurlod
Education and Consulting Services
ABN 57 092 448 518
I have looked into the merge stage, but i cant figure out how to make it work. Do you know it well enough to give me better detail about how to use it than what is in the help file?
Thank you for all of your help
Jim
-
ds_developer
- Premium Member
- Posts: 224
- Joined: Tue Sep 24, 2002 7:32 am
- Location: Denver, CO USA
Don't use the Merge stage - it is worthless in my opinion. The trick is 3 sequential stages in a row: the Input of the second stage is set to overwrite with the new file name, the Input of the 3rd stage is set to append with the new file name and the Output of the 3rd stage is the new file. That should do it!
John
John
quote:Originally posted by ds_developer
[br]Don't use the Merge stage - it is worthless in my opinion. The trick is 3 sequential stages in a row: the Input of the second stage is set to overwrite with the new file name, the Input of the 3rd stage is set to append with the new file name and the Output of the 3rd stage is the new file. That should do it!
John
Having a bit of trouble with your suggestion. Let me restate it so that I know that we are talking on the same page
I have two files, lets say file A and File B
I load file A to File B appending the data, then load file B into
a new file overwriting it? If so why cant I juat Load the data from A to b and be finished with it?
Also when the job runs i get the errror that file C is already in use and then it aborts. So that is why i am thinking that we are not on the same page. Thanx for your time
Jim
[br]Don't use the Merge stage - it is worthless in my opinion. The trick is 3 sequential stages in a row: the Input of the second stage is set to overwrite with the new file name, the Input of the 3rd stage is set to append with the new file name and the Output of the 3rd stage is the new file. That should do it!
John
Having a bit of trouble with your suggestion. Let me restate it so that I know that we are talking on the same page
I have two files, lets say file A and File B
I load file A to File B appending the data, then load file B into
a new file overwriting it? If so why cant I juat Load the data from A to b and be finished with it?
Also when the job runs i get the errror that file C is already in use and then it aborts. So that is why i am thinking that we are not on the same page. Thanx for your time
Jim
quote:Originally posted by Neena
[br]Hi as ray said
Before the job runs, execute this command (perhaps through a before-job subroutine invoking ExecDOS):
type file1 file2 > file3
or
copy A+B C
both woks to merge two files in to one
Neena
....I finaly got what John was saying to work. Now the question that I have is wich way makes better bussness sence.....John's way actualy mutates the source files....your way does not. is that realy that important?......Any sugestions?
[br]Hi as ray said
Before the job runs, execute this command (perhaps through a before-job subroutine invoking ExecDOS):
type file1 file2 > file3
or
copy A+B C
both woks to merge two files in to one
Neena
....I finaly got what John was saying to work. Now the question that I have is wich way makes better bussness sence.....John's way actualy mutates the source files....your way does not. is that realy that important?......Any sugestions?
-
ds_developer
- Premium Member
- Posts: 224
- Joined: Tue Sep 24, 2002 7:32 am
- Location: Denver, CO USA
Sorry I didn't have enough time to explain further this morning. The 3 sequential stage operation does not change the source files, however it does create an extra file.
Here's how it goes -
Start with File A and File B
1st Sequential Stage:
Outputs: File A
2nd Sequential Stage:
Inputs: File C (set to overwrite)
Outputs: File B
3rd Sequential Stage:
Inputs: File C (set to append)
Outputs: File C (goes to Transformer)
If space is an issue, you can delete File C in the After-Stage Subroutine of the transformer.
John
Here's how it goes -
Start with File A and File B
1st Sequential Stage:
Outputs: File A
2nd Sequential Stage:
Inputs: File C (set to overwrite)
Outputs: File B
3rd Sequential Stage:
Inputs: File C (set to append)
Outputs: File C (goes to Transformer)
If space is an issue, you can delete File C in the After-Stage Subroutine of the transformer.
John
Regardless of the methodology used, I wouldn't advocate any method that 'mutates' the source file. At best, it is not restartable - and at worst case can be considered dangerous, as the possibility for corruption of one of the original files exists. These files are not always easily recreateable.
Create a third file by combining the original two and process the third.
My 2 cents...
-craig
Create a third file by combining the original two and process the third.
My 2 cents...
-craig