Parameter sets, value files and sequence compilation

[IASAQ]

Hi all,

New to Datastage and I have to admit that after working with Cognos Data Manager a little bit, I find Datastage's way of passing values from job to job in a job sequence very convoluted. I try to use parameter sets since that's what seems to be recommended instead of using UserStatus. I searched the forums as much as I could on the subject.

This said, here's the scenario in the sequence.

Job1 extracts some info that will be needed to trigger Job2 and writes it in a parameter set value file. Let's call the parameter set "PS_1" and the value file is named "PSValues". In the value file, I will have a "param1=value1" entry.

In Job1's properties window, in the Trigger tab, Expression section, I have PS_1.param1 = "N" as a condition to execute Job2. That "N" comes from the execution of Job1.

The problem is that the sequence will not compile because it says that "PS_1.param1" is not defined. How do I deal with this?

Thanks for your input.

[ray.wurlod]

Should be PS_1.param1, and parameter set PS_1 needs to be added to the job/sequence that uses it.

[IASAQ]

Ray,

It is PS_1.param1, it's a typo on my part in the original post. I'll fix the post. As for PS_1 set, it is added in the sequence and jobs as well already. So why the compile error?

The sequence will know the name of the value file only when it's passed as a parameter with dsjob since it looks I can't put it in the job property window for that sequence when I add the PS_1 parameter set. The sequence doesn't know which parameter set value file I want to use at design time and when I want to compile it, I get undefined variables of course. Ah yes, it's important to note that PS_1 will have multiple value files.

As previously stated, I'm new with the product. I'm sure there are a lot of things I don't get yet.

[IASAQ]

Well, I fixed the compilation problem. Apparently, I misinterpreted how to use parameter set value files. I thought that the value file was where I put the parameters and that each value file could have different parameters.

Looks like I have to add the parameters in the parameter set and all value files I may use for that parameter set share the same parameters. Not the way I thought it worked at all. Still may end up using UserStatus after all.

[ray.wurlod]

Bonne chance!

Each values file allows you to specify a different set of default values for the parameters in the Parameter Set.

[IASAQ]

Gents, another question on the subject if I may.

If I read the parameter set value file with an "Execute Command" stage in the sequence in order to use a value to start another job activity, how do you deal with the trigger?

Let's say my parameter set value file contains this:

param1=ABC
param2=123

Using the "|" as a delimiter, my converted "$CommandOutput" will contain this:

param1=ABC|param2=123

If I use the field function in my sequence trigger to retrieve the first entry value, the result will be "param1=ABC" not "ABC", which is what I want.

How do you deal with it? Another string parsing to get the substring after the "=" character? Or am I asking too much of parameter set value files to handle sequence steps?
.
Thanks

[chulett]

Unless I missing something, I don't see a need to play any Execute Command games with the values file. The job should be using the parameter set and when you build out the value file associated with it, the job uses that automatically. Granted, I haven't been following along before this so perhaps you have some kind of special need?

Okay, perhaps I see what's going on here. Others will need to confirm as I've been away from this particular tool for quite some time but if you need to keep several different sets of "parameter file values", seems to me you should store them... elsewhere. Separately, staged and ready to go. Then before you run the job, use the Execute Command to copy the version of the values file you need over the unique one assigned to the parameter set. After that, the job will consume them without any extra shenanigans needed on your part.

Question for Those Know - or is there a way to dynamically point the parameter set to the appropriate values file just before running the job? Off to see what I can find. In any case, there really shouldn't be any need for you to pull them out and build those delimited strings of name/value pairs like that.

[IASAQ]

Unless I missing something, I don't see a need to play any Execute Command games with the values file.

Chullet,

The goal of doing this is to not have to use "UserStatus" from a job to trigger another job in the sequence. I read on the forums that it's not recommended anymore since the apparition of parameter sets.

But to tell you the truth, using parameter sets to do that is proving to be a major PITA compared to using "UserStatus" to drive my sequence..

As I said in my previous reply, it's entirely possible that I'm trying to use parameter sets value files for a purpose they were not designed for.

[chulett]

Understood and that's exactly what I'm worried about - that there is a bit of over-thinking or over-engineering going on as the use of parameter sets with value files were meant to make everyone's life easier, not harder.

USERSTATUS is an old, Server job concept that is typically fairly easy to leverage when all you need is a single value and a Server job can be used to 'fetch' it for you. A single record can be written to the USERSTATUS area that the job maintains and a Sequence job has a built in hook to move something from a job's USERSTATUS directly to a job parameter runtime value. Yes, that 'single value' can be a delimited string but then you need something post-fetch to parse that string into component pieces. I've certainly done that 'back in the day' before parameter sets and all that by leveraging the Field function in the parameter assignment - not the trigger but actually in the Job Activity stage itself. And you can always inline the Field function to get the value in your example:

Code: Select all

Field(Field($CommandOutput,'|',1,1),'=',2,1)

Forgive me if the syntax isn't exactly correct, working from memory here, but you basically take the first field from a pipe delimited string and then from that take the second equal sign delimited field.

Of course, all of this should be moot with parameter files.

[IASAQ]

And you can always inline the Field function to get the value in your example:

Code: Select all

Field(Field($CommandOutput,'|',1,1),'=',2,1)

Forgive me if the syntax isn't exactly correct, working from memory here, but you basically take the first field from a pipe delimited string and then from that take the second equal sign delimited field.

Ah. This would probably fit the bill on the trigger tab side in the sequence. I'll try it out.

Thanks, that was helpful.