writing and reading a hash file in same stage problem
Moderators: chulett, rschirm, roy
writing and reading a hash file in same stage problem
I have a job like this
Folder stage -> XML stage -> Hash File stage(with input and output on the same file) -> OCI
The reason I use Hash file stage is to remove duplicate data. Most of times it works fine and I have used a lot in my jobs with a hash file stage having input and output link for the same file. But I experienced problem today that I had 10000 records write to the hash file and only 100 records were inserted into the table in OCI stage. When I viewed the file, it definitely had more than 100 records in the hash file. The job was completed successfully without any error or warning. Does anyone experienced the same problem?Thanks
Folder stage -> XML stage -> Hash File stage(with input and output on the same file) -> OCI
The reason I use Hash file stage is to remove duplicate data. Most of times it works fine and I have used a lot in my jobs with a hash file stage having input and output link for the same file. But I experienced problem today that I had 10000 records write to the hash file and only 100 records were inserted into the table in OCI stage. When I viewed the file, it definitely had more than 100 records in the hash file. The job was completed successfully without any error or warning. Does anyone experienced the same problem?Thanks
Regards
Jie
Jie
How many rows went "into" the OCI stage - perhaps you are getting insert errors due to constraints. Add a reject link in the job to ensure that this isn't causing your dropped records.
<a href=http://www.worldcommunitygrid.org/team/ ... TZ9H4CGVP1 target="WCGWin">
</a>
</a>
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Do you have a SELECT on your hashed file read or any warnings in the log file?
<a href=http://www.worldcommunitygrid.org/team/ ... TZ9H4CGVP1 target="WCGWin">
</a>
</a>
Also, do you re-create the hashed file each run or append to it? Is the error visible on each run or just occasionally? Do other jobs use the same hashed file? If you turn off buffering on the write does the error remain?
<a href=http://www.worldcommunitygrid.org/team/ ... TZ9H4CGVP1 target="WCGWin">
</a>
</a>
The file is used only in one job. File is recreated every job run. No write cache is used.
ArndW wrote:Also, do you re-create the hashed file each run or append to it? Is the error visible on each run or just occasionally? Do other jobs use the same hashed file? If you turn off buffering on the write d ...
Regards
Jie
Jie
Is it reproduceable now? Do other jobs use that hashed file as well and could be affecting the contents?
<a href=http://www.worldcommunitygrid.org/team/ ... TZ9H4CGVP1 target="WCGWin">
</a>
</a>
-
- Participant
- Posts: 27
- Joined: Wed Jan 03, 2007 1:29 am
Hash File Probem
Could you check like you have given the same path in the Input & output tab of Hash file. If you have given a path in input & some other path in output, it may take from the old hash file.
Hemachandra
Re: Hash File Probem
I seperate the job into 2 jobs. A simple job loading from hashed file to Oracle table. I got the following log today:
Run stopped after 100 rows
No warning.
Run stopped after 100 rows
No warning.
Regards
Jie
Jie
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Run stopped after 100 rows can only be caused by one thing; when the job run request is issued, a limit of 100 rows is imposed. Check the Limits tab on your Job Run Options dialog.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
I will check the Run Options when I get back office tomorrow morning. But I run the sequence job from the director. There are many other jobs in this sequence and only this job got stopped after 100 rows. Very werid.ray.wurlod wrote:Run stopped after 100 rows can only be caused by one thing; when the job run request is issued, a limit of 100 rows is imposed. Check the Limits tab on your Job Run Options dialog. ...
Regards
Jie
Jie