Page 1 of 1

Merge Stage using Two Sequential Files

Posted: Tue Mar 02, 2010 10:54 pm
by sarathi
Trying to Merge two Sequential files with Primary key (one file coming from insert source and other update source).
Used Merge stage and combined two Sources but 'Output' Properties are disabled and due to this i'm unable to write to FINAL Sequential file.

I have given output file location of two CSV files and its not accepting if not given. How can i merge this type of files.

Sarathi

Posted: Tue Mar 02, 2010 11:01 pm
by ray.wurlod
Your description is not clear.

Is this a server job or a parallel job? Can you please draw a picture of your job design (or publish an image of it and link to that from here)?

Posted: Tue Mar 02, 2010 11:48 pm
by sarathi
I have published Job flow and send you link below. Let me know if you unable to access it.

http://picasaweb.google.com/sarathit/Da ... 4738266178

Posted: Wed Mar 03, 2010 12:15 am
by gssr
sarathi wrote:I have published Job flow and send you link below. Let me know if you unable to access it.

http://picasaweb.google.com/sarathit/Da ... 4738266178
Use Copy stage , Instead of writing the CSV file directly and get anither link from copy stage to Join. Now you can edit the Metadata in the links.

Posted: Wed Mar 03, 2010 1:01 am
by ReachKumar
Same sequential file can't be used as input and output.

Since you have used sequential files as outputs in the job.They can't be used as inputs in the same job.Only the reject link is allowed for the output file.

If you want to join the output files, then you can do that in another job.

Posted: Wed Mar 03, 2010 1:02 am
by ray.wurlod
OK you're feeding one of the Merge stage inputs with a Reject link from a Sequential File stage. A reject link has no metadata associated with it, so you won't be able to specify the key to use to effect the merge (I suspect you want a Join in any case). Insert a Column Import stage into the Reject link to re-parse the raw string that is received into it.

What are you trying to achieve?

Posted: Wed Mar 03, 2010 1:04 am
by ReachKumar
Adding one more point to the above post :

You can do as gssr told.

Posted: Wed Mar 03, 2010 6:57 am
by sarathi
GSSR - I'm little confused as what you are saying below -

Use Copy stage , Instead of writing the CSV file directly and get anither link from copy stage to Join. Now you can edit the Metadata in the links

Should i use TWO Copy stages for each CSV File and then Join them. And then output of Join goes to Final CSV file.

Sarathi

Posted: Wed Mar 03, 2010 7:08 am
by abhijain
it will be something like this...


Switch Stage-->Copy Stage (in place of Seq file)

And from copy stage, drag two links ...one for Sequential File and one for your join stage.

Re: Merge Stage using Two Sequential Files

Posted: Wed Mar 03, 2010 8:40 pm
by Kryt0n
sarathi wrote:Used Merge stage and combined two Sources but 'Output' Properties are disabled and due to this i'm unable to write to FINAL Sequential file.
Sarathi
[splitting hairs] That looks like a join stage to me... [/splitting hairs]

Are you trying to join the data or the rejects from each file?