Append to DataSet - expecting 3 segments, read 4 segments

pneumalin · Post by **pneumalin** » Wed Jun 21, 2006 8:41 am

Hi All,
Has anyone encountered similar warning message like:
opt/bi/datasets/bloomberg/bb_bond_price_FromFile.ds, expecting 2 segments, read 3 segments.

I have 6 instances appending to the same dataset concurrently, they finished ok with the above warnings. All the warning message states the same content, but the counting is different as following:
expecting 3 segments, read 4 segments
expecting 5 segments, read 6 segments...etc.

Any advises except Message Handler to avoid the warning is appreciated!

Thanks!

kumar_s · Post by **kumar_s** » Wed Jun 21, 2006 8:45 am

Updating a dataset would given you very different error. May I know what is the job design?

pneumalin · Post by **pneumalin** » Wed Jun 21, 2006 9:23 am

The Job design is very simple. It only contains 3 stages including SourceFile, Transformer and Target DataSet. The job simply uses "append" since it will be called by 6 instances using different source files and appending to the same dataset.

ArndW · Post by **ArndW** » Wed Jun 21, 2006 9:30 am

How many nodes do you have in your configuration file and do you repartition?

pneumalin · Post by **pneumalin** » Wed Jun 21, 2006 9:33 am

Arnold,
I know someone will ask this question, fotgot to mention that in previous update.
I used 1 node only!
Thanks!

kumar_s · Post by **kumar_s** » Thu Jun 22, 2006 2:54 am

Are all of the sources compatiable with the given layout?

ray.wurlod · Post by **ray.wurlod** » Thu Jun 22, 2006 3:33 am

Look in Tools > Data Set Management in Manager client. On how many partitions is each of the Data Sets actually defined?

pneumalin · Post by **pneumalin** » Thu Jun 22, 2006 8:13 am

Only one partition# 0 showed up and the NODE name is node1.
We just use the default.apt that is configured to one node only.

kumar_s · Post by **kumar_s** » Thu Jun 22, 2006 8:41 am

What if you run all the job serially? Will this error prompts again?

pneumalin · Post by **pneumalin** » Thu Jun 22, 2006 8:43 am

No, The warning message goes away if the jobs are run sequentially.

kumar_s · Post by **kumar_s** » Thu Jun 22, 2006 8:53 am

I am wondering, how it allowed you to write to a dataset in paralle. And not gave out all those errors that I got when I did the same.
Why cant you serialize the flow, unless requiered to write in parallel otherwise?

pneumalin · Post by **pneumalin** » Thu Jun 22, 2006 12:24 pm

Kumar,
We have to schedule the 6 instances to wait for the files arrival in a certain period, and we don't know when exatly the file will arrive. That's why we have to run them in parallel.
Thanks for your advise.