DSXchange

Sainath.Srinivasan wrote:In that case, would't the second record be discarded by the stage ?

Actually first record should be discarded always. But in some runs first record is discarded and in some other runs second record is discarded.
So same job is yielding different results in different runs.

Sainath.Srinivasan wrote:What are your keys for remove duplicates ?

Column1 and Column2 i.e, In example:
Record Column1 Column2 Date1 Date2 Flag
1 100 200 15-May-2009 24-May-2009 Y
2 100 200 15-May-2009 20-May-2009 Y

The values 100 and 200 combination makes the key value.

Did you check whether this is the impact of 'First' in remove duplicates? Since the Sort stage is "Hash" Partitioned on Column1, Column2 but "Sort" on Column1,Column2, ColumnX, ColumnY and ColumnZ (All in ascending order). And in the Next Remove Duplicates Stage Partiton is &quo...

It is picking the wrong record but not both the records. For example: Record Column1 Column2 Date1 Date2 Flag 1 100 200 15-May-2009 24-May-2009 Y 2 100 200 15-May-2009 20-May-2009 Y In the above e.g, Record 2 should be fetched as Date2 is more nearer to Date1. but some times it is fetching Record 1 ...

In all the environments we are running the jobs using 4 nodes configuration files. For more clarity on the issue, please find below some examples of source data, Record Column1 Column2 Date1 Date2 Flag 1 100 200 15-May-2009 24-May-2009 Y 2 100 200 15-May-2009 20-May-2009 N In the above e.g, Record 1...

Hi, I am having a requirement to select a record based on following criteria: Record layout Column1,Column 2, Date1, Date2,Flag Flag can consists of either 'Y' or 'N'. Based on Column 1, Column 2 if we get more than one record in the source then always take the record whose Flag is 'Y'. If the two r...

Hi, We are running our jobs in prod environment. We were able to run the same job twice successfully but the third time we faced the below error with the same amount of data - main_program: Internal Error: (d->nPartitions()== nPartitions_): datamgr/segment.C: 276 Traceback: Could not obtain stack tr...

Hi,

I am developing a parallel job but after the job i am calling a subroutine(routine written in Basic). Can i insert/update a record in the table through after job subroutine. If so please let me know, how can we acheive that?

As we are maintaining 7 years of data, I feel Partitioning on this column is not a good idea and deletion of data by creating a nonunique index on this column is a good approach (hoping this wont occur too frequently)

The Database is Db2udb 8.x. Is there any flash back option in Db2? If we go for the option of having a date column or Etl_id column (suggested by Nick bond) to identify the records for a particular run and delete them if the job aborts in the middle, Is it a good idea to perform a delete operation o...

Upsert will be much slower, if your target table keeps growing. You can always load the data in certain order (sort) based on some key. So that each time you load, you can find the max of available key value in the target and load from that point from the source. But i dont have a Surrogate key in ...

Hi, In my job i m reading from a dataset and loading it into a table. Source consists of 2 Million records and I have only inserts as the data is transactional data, inorder to take care for the restartability(Say if the job abends in the middle after loading 1 million records) which of the followin...

Hi, One of my job ran successfully so many times. But now when i tried to run the same job, It aborted with the following message. buffer(4),0: Fatal Error: waitForWriteSignal(): Premature EOF on node XXXX No such file or directory and with this it also shows the player terminated unexpectedly. But ...

Yes Kumar, I am getting .000000 but not the exact value say for eg .000012 or so.

iamnagus wrote:I have a parallel job in which have the stage variables and constraints. I just want to know the order of the execution in transformer stage?
Can any one give idea? Thanks in advance.

First stage variables will execute and then constraints