raju_chvr wrote:I would like to know if any of you have ever come accross or come up with testing methodologies for DS jobs. I am sure Kim, Kenneth, Ray and Teej may have their own ways of testing DS before exporting/moving them to Testing or Production boxes.
Hmmm....
Open Director. Run Job. Check results.
The problem here -- we have PX for 6.0, and that does not have its own debug mode available on the GUI interface. 7.0 obviously would be different for us -- the ability to step through the rows and stages would be very helpful for us.
I use rejects for the Transform stages during testing -- if anything is rejected, it usually means that I forgot to put an "If IsNull()" somewhere.
We were suggested to do an incremential build by an Ascential consultant, where we add a stage, add the requirement, then test intensively, then add another stage... Honestly, with the vast ease of building jobs in PX, I find it just a massive waste of time.
I do wish that some things are 'noticed' during compile time. For example -- lets say you are using a join stage, and you are hashing/sorting the key fields. Wouldn't it make sense that they would notice whether or not you are hashing the right fields? That tripped me up one time while testing something. I was hashing on "PARENT_WHOLESALER_NAME" and merging on "PARENT_WHOLESALER_NUMBER". No results found on an 4 node run, naturally.
We found over time that the error messages are about 90% our fault, 9.999% data fault and 0.001% DataStage's fault. Right now, we have one of those 0.001% issue.
This is the kind of error you DON'T want to see at all. But this kind of guideline help us stay focused when we are debugging our programs -- the GUI is so easy, yet so yucky in some areas, that we can easily miss some things. For example -- why is the Transform Stage always split down the middle perfectly, when the input fields are narrow, and the output fields are wide? Spread it out! Why don't the transform stage word-wrap the transformations? Therefore, it's easy to overlook an error like:
Code: Select all
If IsNull(Input.W_KEY_FIELD) Then
SetNull()
Else
Input.A_KEY_FIELD
If it's wrapped, this would be easier to spot. I only noticed it because when I check the links for the input stages, it was pointing where it wasn't supposed to be pointing to.
Color coding the links would be so NICE! Instead of using ---->, have a RED bold ===>.
But I wouldn't give this up and go back to C++ only. Ever since I worked on ETL, C++ became fun for me once again, but only in good small doses.
Plus it's easier to debug on C++ than DataStage PX.
So... testing. Really, it all varies -- I trust the tools to do its job, and I compare it with a direct SQL query on a small dataset.
-T.J.