Datastage Sequencer problem

rumu · Post by **rumu** » Mon Sep 22, 2008 11:14 am

hi,

I am new to datastage parallel,my requirement is to generate one surrogate key for a job while the job gets started.This id is a MetatagId for each job.
I am using surrogate key generator to create the surrogate key.
while try to generate the key,i used 1 number of records in the options in surrage key stage ,while running the job,the stage creates 2 records 1 and 101.I have chosen the option of generate from last highest value but it does not take the last value ,during 2nd run job generates 201 and 301.
Could you kindly let me know why 2 records are getting created insted of one.
is there any inbuilt function(keymgmtfunction) to generate the key like the same used in server job?
regards

rumu

ray.wurlod · Post by **ray.wurlod** » Mon Sep 22, 2008 4:15 pm

What does "parallel" mean to you? Are you running this job on two nodes?

rumu · Post by **rumu** » Tue Sep 23, 2008 1:10 am

Yes Ray,2 node configuration file is being used ...
In surrogate key generator stage the 'configurations file' options is set to default and it is disabled also.
Could you please suggest

Thanks for your support.

rumu · Post by **rumu** » Tue Sep 23, 2008 1:29 am

Hi Ray,

I added the env variable $APT_CONFIG_FILE in the job parameter and set the value to 1node configurationfil.
Now only one record is generated.

Is this the required step or anything else is required?

Many thanks for your support.

Regards

ray.wurlod · Post by **ray.wurlod** » Tue Sep 23, 2008 2:30 am

That's one "solution". Another would be to execute stage(s) in sequential mode. Another would be to execute stage(s) in a single-node node pool within a multi-node configuration. My personal preference for processing a single row is a server job, where the startup and execution overheads are far lower than for a parallel job.

rumu · Post by **rumu** » Tue Sep 23, 2008 5:23 am

I tried one option ie run the stage in sequential mode.

I made the surrogate key stage in sequential mode..my output is a sequential file stage.

I got the following error:

'Surrogate_Key_Generator_0: Error when checking operator: Input data set on port 0 has a partition method, but the operator is not parallel.'

ray.wurlod · Post by **ray.wurlod** » Tue Sep 23, 2008 5:55 am

That's no surprise. Fix it.

rumu · Post by **rumu** » Tue Sep 23, 2008 6:28 am

Hi Ray,

A stage could be sequential if the execution mode is set to sequential and node pool resource constraints made to single node.

In my stage(surrogate key generator) the above option is disabled and the reason might be that the configurations file does not have multiple nodes.

Please advise.

rumu · Post by **rumu** » Tue Sep 23, 2008 6:42 am

Ray,

In continuation to my previous reply i would like to add that,i tried putting one node 'node1' in Node map constraint options ,got the same error.

Regards,

rumu

ray.wurlod · Post by **ray.wurlod** » Tue Sep 23, 2008 7:04 am

But, in spite of all of these things, you still have a partitioning algorithm specified.

rumu · Post by **rumu** » Tue Sep 23, 2008 7:21 am

I put 'Preserve Partition' as clear but this is for the next stage.

Are you asking about 'execution mode' in job properties,i checked that the option 'force sequencial mode' is checked.

Could you please help.

ray.wurlod · Post by **ray.wurlod** » Tue Sep 23, 2008 4:04 pm

I'm not asking anything. I am observing that the job design has a partitioning algorithm specified, and that it is that which is triggering the alert message.

rumu · Post by **rumu** » Wed Sep 24, 2008 12:58 am

The job design is simple,

Surrogate key generator stage------------>Sequential File stage

surrogate key generator stage has 'sequential mode' execution.

as per my knowledge no specific partitioning method is mentioned in the job design.

Could you please enlight?

rumu

ray.wurlod · Post by **ray.wurlod** » Wed Sep 24, 2008 1:14 am

Please dump and post the job score.

rumu · Post by **rumu** » Wed Sep 24, 2008 1:32 am

Hi Ray,

here is the dump:

OSH script
# OSH / orchestrate script for Job metatagidcreate compiled at 09:28:02 24 Sep 2008
#################################################################
#### STAGE: Surrogate_Key_Generator_0
## Operator
surrogatekey
## Operator options
-sk_id '/transfer01/workdir/Metatagid1.txt'
-output_key 'Metatagid'
-asSequencer
-records 1
## General options
[ident('Surrogate_Key_Generator_0'); jobmon_ident('Surrogate_Key_Generator_0'); seq; nodemap ( node1 ) ]
## Outputs
0> [-pp; modify (
metatag:string[max=30]=Metatagid;
keep
Metatagid;
)] 'Surrogate_Key_Generator_0:DSLink2.v'
;
#################################################################
#### STAGE: Sequential_File_4
## Operator
export
## Operator options
-schema record
{final_delim=end, delim=',', quote=double}
(
metatag:string[max=30];
)
-file '/transfer01/workdir/testmeta'
-overwrite
-rejects continue
## General options
[ident('Sequential_File_4'); jobmon_ident('Sequential_File_4'); nodemap ( node1 ) ]
## Inputs
0< [] 'Surrogate_Key_Generator_0:DSLink2.v'
;
# End of OSH code

regards,

rumu