Stage Variable NOT Stored Correctly
Moderators: chulett, rschirm, roy
-
- Premium Member
- Posts: 8
- Joined: Sun Feb 05, 2012 8:20 pm
- Contact:
Stage Variable NOT Stored Correctly
I am very green for the DataStage and I have just 2 days experiences for the DataStage.
I have been advised by my team-mates for having Stage Variable. However, he is out of town and I have to solve this problems within 3 days. But the Stage Variable which seems to be never stored.
I have captured most of the tasks imposed during my trials.
Step 1 - Sort Stage
Step 2 - Transformer Stage
I have been advised by my team-mates for having Stage Variable. However, he is out of town and I have to solve this problems within 3 days. But the Stage Variable which seems to be never stored.
I have captured most of the tasks imposed during my trials.
Step 1 - Sort Stage
Step 2 - Transformer Stage
Learning everyday is a must for everyone...
-
- Participant
- Posts: 597
- Joined: Fri Apr 29, 2005 6:19 am
- Location: Singapore
-
- Premium Member
- Posts: 1735
- Joined: Thu Mar 01, 2007 5:44 am
- Location: Troy, MI
:D
Yes you can do it with stage variables, you need to define a few stage variables.
Stage Variable=Derivation
svPrevTime=svCurrTime
svCurrTime=<input col>.Time_Chk
svPrevKey=svCurrKey
svCurrKey=<input col>.key1:'~':key2
svTimeDiff=compare the time here.
svConstraintYN=if svPrevKey=svCurrKey and timediff<30 then 'N' else 'Y'
then put a constraint as svConstraintYN='Y'
you need to make sure that svPrevTime and svPrevKey is defined above/before svCurrTime and svCurrKey respectively. you can use the exact order as i defined above.
Stage variables are executed sequentially from top to bottom and the same principal is being exploited here to store the last value.
Yes you can do it with stage variables, you need to define a few stage variables.
Stage Variable=Derivation
svPrevTime=svCurrTime
svCurrTime=<input col>.Time_Chk
svPrevKey=svCurrKey
svCurrKey=<input col>.key1:'~':key2
svTimeDiff=compare the time here.
svConstraintYN=if svPrevKey=svCurrKey and timediff<30 then 'N' else 'Y'
then put a constraint as svConstraintYN='Y'
you need to make sure that svPrevTime and svPrevKey is defined above/before svCurrTime and svCurrKey respectively. you can use the exact order as i defined above.
Stage variables are executed sequentially from top to bottom and the same principal is being exploited here to store the last value.
Priyadarshi Kunal
Genius may have its limitations, but stupidity is not thus handicapped.
Genius may have its limitations, but stupidity is not thus handicapped.
Welcome to DataStage!
Parallel jobs have a lot of date and time functions built in. They are documented in the appendix of the Parallel Job Developer's Guide. You should have it on your client as a PDF under the Start menu... documentation icon. Check out this one for comparing two timestamps:
SecondsSinceFromTimestamp()
Parallel jobs have a lot of date and time functions built in. They are documented in the appendix of the Parallel Job Developer's Guide. You should have it on your client as a PDF under the Start menu... documentation icon. Check out this one for comparing two timestamps:
SecondsSinceFromTimestamp()
Choose a job you love, and you will never have to work a day in your life. - Confucius
I have few doubts with respect to above post.priyadarshikunal wrote:Yes you can do it with stage variables, you need to define a few stage variables.
1)what is the intial value of svCurrTime and svCurrKey stage variables?If i did not mention anything in stage variables tab what values it will take after processing first record from input?
2)Is it necesary to mention intial values in the above requirement?
2)can i run the transformer stage in parallel mode in the above requirement?
-
- Premium Member
- Posts: 1735
- Joined: Thu Mar 01, 2007 5:44 am
- Location: Troy, MI
0 for integer and empty for varchar and so on. in stage variable tab you will see initial value column, define it there. as the stage variables can not be null.vamsi.4a6 wrote:I have few doubts with respect to above post?
1)what is the intial value of svCurrTime and svCurrKey stage variables?If i did not mention anything in stage variables tab what values it will take after processing first record from input?
2)Is it necesary to mention intial values in the above requirement?
Null should not be or better to say cannot be assigned to a stage variable.
2)can i run the transformer stage in parallel mode in the above requirement?
Yes if your data is partitioned. this will remove the records for the same key which came within 30 mins.
Priyadarshi Kunal
Genius may have its limitations, but stupidity is not thus handicapped.
Genius may have its limitations, but stupidity is not thus handicapped.
Regarding sorting and partitioning in your job:
1) DON'T select "Perform Sort" on the partitioning tab for the input link of a SORT stage. There are a few specific cases where this is useful (yours is not one of them!) and in most cases this is a waste of resources and reduces throughput.
2) Assuming that your transformer is immediately after your sort stage, you don't need to select "Perform Sort" in the input link of the transformer...you have just sorted the file and there is no need to sort it again.
3) As already mentioned in this thread, you shouldn't use the time field as a partition key...it will keep your logic from working correctly
4) Allow your transformer to run in parallel execution mode and set partitioning on the transformer to either Auto or Same...in this case, Same would probably be the better choice. You partitioned the data at the Sort stage and don't need to repartition it for your transformer.
Regards,
1) DON'T select "Perform Sort" on the partitioning tab for the input link of a SORT stage. There are a few specific cases where this is useful (yours is not one of them!) and in most cases this is a waste of resources and reduces throughput.
2) Assuming that your transformer is immediately after your sort stage, you don't need to select "Perform Sort" in the input link of the transformer...you have just sorted the file and there is no need to sort it again.
3) As already mentioned in this thread, you shouldn't use the time field as a partition key...it will keep your logic from working correctly
4) Allow your transformer to run in parallel execution mode and set partitioning on the transformer to either Auto or Same...in this case, Same would probably be the better choice. You partitioned the data at the Sort stage and don't need to repartition it for your transformer.
Regards,
- james wiles
All generalizations are false, including this one - Mark Twain.
All generalizations are false, including this one - Mark Twain.
-
- Premium Member
- Posts: 8
- Joined: Sun Feb 05, 2012 8:20 pm
- Contact:
Thanks for your hints. It has been solved. My mate has also suggested me be aware about time functions. For example, "MinutesFromTime" is just for minute and it should be converted by hours + minutes...
Also, catering for cross-date transaction is needed.
Before the end of the reply, I would like to raise my thanks to all of you.
Also, catering for cross-date transaction is needed.
Before the end of the reply, I would like to raise my thanks to all of you.
Learning everyday is a must for everyone...
-
- Premium Member
- Posts: 1735
- Joined: Thu Mar 01, 2007 5:44 am
- Location: Troy, MI
If you use the SecondsSinceFromTimestamp() function that I mentioned above, then I think you can avoid doing it "the hard way" (checking across hours and across days at midnight). By definition, the function "Returns the number of seconds between two timestamps." One check and you're done!gainfomart wrote:For example, "MinutesFromTime" is just for minute and it should be converted by hours + minutes...
Also, catering for cross-date transaction is needed.
Choose a job you love, and you will never have to work a day in your life. - Confucius