Transformer output rows reduced!
Moderators: chulett, rschirm, roy
-
- Participant
- Posts: 21
- Joined: Wed Oct 01, 2003 11:53 am
Transformer output rows reduced!
Hi,
I have a transfomer stage which has 'x' # of rows in input stream.
There is no constraint on the transformer.
Even then, the output is 'y' # of rows where y < x.
There are no corresponding warnings/messages in the log file.
Any pointers to this?
Thanks in advance.
Regards,
Nitin
I have a transfomer stage which has 'x' # of rows in input stream.
There is no constraint on the transformer.
Even then, the output is 'y' # of rows where y < x.
There are no corresponding warnings/messages in the log file.
Any pointers to this?
Thanks in advance.
Regards,
Nitin
Nitin
If you are talking about in DataStage monitor then I have never seen that before. If you are talking about doing counts on the tables before and after a job runs then you could have 2 records with the same key and the update the same record in the target table.
If the first option then what OS and what version of DS and what database?
Kim.
Kim Duke
DwNav - ETL Navigator
www.Duke-Consulting.com
If you are talking about in DataStage monitor then I have never seen that before. If you are talking about doing counts on the tables before and after a job runs then you could have 2 records with the same key and the update the same record in the target table.
If the first option then what OS and what version of DS and what database?
Kim.
Kim Duke
DwNav - ETL Navigator
www.Duke-Consulting.com
May be we should check the Update strategy in the target stage. This can happen if that had a Insert/Update or Update/Insert strategy.
Badhri ...
Badhrinath Krishnamoorthy
www.cognizant.com
Badhri ...
Badhrinath Krishnamoorthy
www.cognizant.com
-
- Participant
- Posts: 21
- Joined: Wed Oct 01, 2003 11:53 am
Okay further details are as below:
Source Stage: Dataset (or an output stream from a previous lookup stage)
Target stage: DataSet
I am viewing the number of records flowing through each link using either of the following:
-'Show performance statistics' Option
-(job) Monitor
Both essentially give the same data.
I suspect if this issue of dropping of records is related to NULL in some field values?
Please suggest.
Regards,
Nitin
Source Stage: Dataset (or an output stream from a previous lookup stage)
Target stage: DataSet
I am viewing the number of records flowing through each link using either of the following:
-'Show performance statistics' Option
-(job) Monitor
Both essentially give the same data.
I suspect if this issue of dropping of records is related to NULL in some field values?
Please suggest.
Regards,
Nitin
Nitin
If that was true then you would get a warning in the log. Also what type is your source and target like ODBC, OCI or whatever.
Kim.
Kim Duke
DwNav - ETL Navigator
www.Duke-Consulting.com
If that was true then you would get a warning in the log. Also what type is your source and target like ODBC, OCI or whatever.
Kim.
Kim Duke
DwNav - ETL Navigator
www.Duke-Consulting.com
-
- Participant
- Posts: 21
- Joined: Wed Oct 01, 2003 11:53 am
Nitin, you're going to have to be VERY explicit with your posts. There's over 2000 DataStage installations out there of DataStage Server, and only tens of PX, so everyone assumes Server based questions unless PX is stated.
What is your partitioning scheme? Have you specified unique? Did you switch node pools in between?
Kenneth Bland
What is your partitioning scheme? Have you specified unique? Did you switch node pools in between?
Kenneth Bland
-
- Participant
- Posts: 21
- Joined: Wed Oct 01, 2003 11:53 am
I understand Kenneth.
I will try to be more explicit now..
-partitioning scheme: It is set to 'Auto' in all prior stages in that job. Data sets being used are set with Preserve partitioning = 'Default (Propagate)'.
-Job Nodes Pools: 4 nodes i.e. 4x4
No, I am not switiching node pools in between. I am keeping 4x4 nodes for all jobs.
Btw I did not understand your following question:
Have you specified unique?
Regards,
Nitin
I will try to be more explicit now..
-partitioning scheme: It is set to 'Auto' in all prior stages in that job. Data sets being used are set with Preserve partitioning = 'Default (Propagate)'.
-Job Nodes Pools: 4 nodes i.e. 4x4
No, I am not switiching node pools in between. I am keeping 4x4 nodes for all jobs.
Btw I did not understand your following question:
Have you specified unique?
Regards,
Nitin
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
-
- Participant
- Posts: 21
- Joined: Wed Oct 01, 2003 11:53 am
Based on the ongoing discussion, I have certain related questions:
-When we talk of transformer, there is no key like thing isnt it.
So where does the question of two records being same come in here.
Is it something related to the dataset?
I even tried having the sequential file as output from transformer. And even that has same probelm( receving lesser # of records than in input with no constraint)
What is the solution? I mean my requirement is to have all the records in input stream (unique or non unique) sent to the output stream of the transformer.
Could you please clarify the concept if I am missing out on something.
Regards,
Nitin
-When we talk of transformer, there is no key like thing isnt it.
So where does the question of two records being same come in here.
Is it something related to the dataset?
I even tried having the sequential file as output from transformer. And even that has same probelm( receving lesser # of records than in input with no constraint)
What is the solution? I mean my requirement is to have all the records in input stream (unique or non unique) sent to the output stream of the transformer.
Could you please clarify the concept if I am missing out on something.
Regards,
Nitin
I would refer you to www.datastagexchange.com, where there's a PX specific forum moderated by bigpoppa. You'll probably get the best answers there.
Kenneth Bland
Kenneth Bland
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
It's really difficult to diagnose this without an explicit explanation of the job design. For example, you did not mention whether there is any constraint on the Transformer stage's output link. If there were, it would be expected to limit the number of rows output.
Were I consulting to solve this, I would need to look at the job in detail, either on site or by having had an export of the job plus sample data mailed to me.
Were I consulting to solve this, I would need to look at the job in detail, either on site or by having had an export of the job plus sample data mailed to me.