strange problem with output hashed file
Moderators: chulett, rschirm, roy
strange problem with output hashed file
Hi,
I have very strange problem with hashed files.
Simple job:
hf1--->Transformer--->hf2
Readable data in hf1.
In transformer I use three input columns as input arguments to routine. And then I use a return value from routine as one of output column.
But on my output I get about fifty records with unreadable data in that column. These wrong records start from about 100 record in HF. End about 150 record. And from that record to the end everything is good
And what solutions we find:
- the input column which is a first input arg to my routine is assigned to a stage variable and then that variable is put as first arg to that routine.
After that we get good results in output HF.
Also when we add new output from transformer to seq file it also gives a good result.
Can anybody explain what is going on? Is this some DS bug?
I can't find any rational explanation for that situation
I have very strange problem with hashed files.
Simple job:
hf1--->Transformer--->hf2
Readable data in hf1.
In transformer I use three input columns as input arguments to routine. And then I use a return value from routine as one of output column.
But on my output I get about fifty records with unreadable data in that column. These wrong records start from about 100 record in HF. End about 150 record. And from that record to the end everything is good
And what solutions we find:
- the input column which is a first input arg to my routine is assigned to a stage variable and then that variable is put as first arg to that routine.
After that we get good results in output HF.
Also when we add new output from transformer to seq file it also gives a good result.
Can anybody explain what is going on? Is this some DS bug?
I can't find any rational explanation for that situation
Best regards,
Piotrek
Polish DataStage consulting company:
<a href="http://www.etlpro.pl">ETL PRO</a>
Piotrek
Polish DataStage consulting company:
<a href="http://www.etlpro.pl">ETL PRO</a>
If you doubt the output, add an extra link to go to a Sequential file. Are you expecting sorted data in some fashion and using stage variables with COMMONs? Define unreadable data.
Kenneth Bland
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
Yes, I use COMMON variables, but not in this routine.kcbland wrote:If you doubt the output, add an extra link to go to a Sequential file. Are you expecting sorted data in some fashion and using stage variables with COMMONs? Define unreadable data. ...
I don't need sorted data, I am sorting in further step.
Ureadable data - square sign, etc., chars which cannot be access directly through key on keybord.
Best regards,
Piotrek
Polish DataStage consulting company:
<a href="http://www.etlpro.pl">ETL PRO</a>
Piotrek
Polish DataStage consulting company:
<a href="http://www.etlpro.pl">ETL PRO</a>
Spool to a file at the same time and see what the result looks like. Spool all values used in the function to the file as well as the result from the file. Look for what's wrong.
Kenneth Bland
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
But this is the issue. When I don;t have output to seq file I get wrong results in HF, but when I add seq file output I get fine results in HF and seq filekcbland wrote:Spool to a file at the same time and see what the result looks like. Spool all values used in the function to the file as well as the result from the file. Look for what's wrong.
Best regards,
Piotrek
Polish DataStage consulting company:
<a href="http://www.etlpro.pl">ETL PRO</a>
Piotrek
Polish DataStage consulting company:
<a href="http://www.etlpro.pl">ETL PRO</a>
Turn off row-buffering, write-caching, and inter-process and try again. I can't really tell you what you're doing wrong, but it could be a bug.
Kenneth Bland
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
OK, I will try this tomorrow and I will get you know about the results.kcbland wrote:Turn off row-buffering, write-caching, and inter-process and try again. I can't really tell you what you're doing wrong, but it could be a bug. ...
Best regards,
Piotrek
Polish DataStage consulting company:
<a href="http://www.etlpro.pl">ETL PRO</a>
Piotrek
Polish DataStage consulting company:
<a href="http://www.etlpro.pl">ETL PRO</a>
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
There was enabled "record level read" option in source hf. After disabling it I get good results. So is it the bug of DS? I think this option should work fine ...kcbland wrote:Turn off row-buffering, write-caching, and inter-process and try again. I can't really tell you what you're doing wrong, but it could be a bug. ...
Best regards,
Piotrek
Polish DataStage consulting company:
<a href="http://www.etlpro.pl">ETL PRO</a>
Piotrek
Polish DataStage consulting company:
<a href="http://www.etlpro.pl">ETL PRO</a>
Why did you check this is in the first place?ppalka wrote: There was enabled "record level read" option in source hf. After disabling it I get good results. So is it the bug of DS? I think this option should work fine ...
Kenneth Bland
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
I have just posted three solutionsDSguru2B wrote:Could you kindly share the solution with us, so that future posters, encountering a similar problem, will know what to do.
Two tricks:
1. add extra output link from transformer;
2. pass input argument through stage variable.
And the finest solution, I think, to disable option "Record level read" on input hashed file of my job. In that case you do not need to change any design of your job.
Best regards,
Piotrek
Polish DataStage consulting company:
<a href="http://www.etlpro.pl">ETL PRO</a>
Piotrek
Polish DataStage consulting company:
<a href="http://www.etlpro.pl">ETL PRO</a>
It wasn't me. There were two more people working on that project. I don't know what was the purpose of checking that option in.kcbland wrote:Why did you check this is in the first place?
I think it could be cause of not having complementary knowledge of DS.
Best regards,
Piotrek
Polish DataStage consulting company:
<a href="http://www.etlpro.pl">ETL PRO</a>
Piotrek
Polish DataStage consulting company:
<a href="http://www.etlpro.pl">ETL PRO</a>