Hi,
I am reading data from seq.file and next i am using Sort Stage and next using Transformer for avoid duplicates and next lookup i am getting this warning for these 3 stages like this
where ever i am using seq.file as target in this job ia m getting warning like " Trgt_Rej_data_FF: When checking operator: A sequential operator cannot preserve the partitioning
of the parallel data set on input port 0." any can help me
thanks in advance
How to avoid this warning
Moderators: chulett, rschirm, roy
-
- Participant
- Posts: 111
- Joined: Thu Jun 01, 2006 5:12 am
- Location: Detroit
How to avoid this warning
kOTI,
1586-738-7785
1586-738-7785
I didn't quite get the clear picture of what you are doing in your job. The order of stages are confusing.But what you will need to do get rid of the warning is set your PreservePartioning flag to "Clear" in one of the stages downstream of the Sort stage because sequential file writes records only sequentially. How exactly are you removing duplicates from Transformer. Using Constraints? Wouldn't Remove Duplicates stage help you in this case? Just wondering.
P.S. Please delete the other duplicate post you have in place before anyone starts replying to it.
P.S. Please delete the other duplicate post you have in place before anyone starts replying to it.
Kris
Where's the "Any" key?-Homer Simpson
Where's the "Any" key?-Homer Simpson
-
- Participant
- Posts: 3593
- Joined: Thu Jan 23, 2003 5:25 pm
- Location: Australia, Melbourne
- Contact:
Assuming you are sending reject rows from the transformer (via Trgt_Rej_data_FF) to a sequential file the job is warning you that you are switching from partitioned data to sequential data. Each transformer across each node will have to send rejects to a link that repartitions (or unpartitions?) the data into a sequential stream to write it to a single file.
This is not a big deal since you would hope to get a very low number of rejects.
You could switch this message across to an information message via the message handler. Would be interested to hear from other programmers as to whether there is an easy way to avoid the warning altogether. Maybe if you switch the partitioning output link from SAME to sequential.
This is not a big deal since you would hope to get a very low number of rejects.
You could switch this message across to an information message via the message handler. Would be interested to hear from other programmers as to whether there is an easy way to avoid the warning altogether. Maybe if you switch the partitioning output link from SAME to sequential.
Certus Solutions
Blog: Tooling Around in the InfoSphere
Twitter: @vmcburney
LinkedIn:Vincent McBurney LinkedIn
Blog: Tooling Around in the InfoSphere
Twitter: @vmcburney
LinkedIn:Vincent McBurney LinkedIn
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact: