Who will use more memory Dataset or Seq. file

lokesh_chopade · Post by **lokesh_chopade** » Tue Aug 07, 2007 4:39 am

If the data inserted into sequential file and same data stored in dataset, which will require more memory? Dataset Or Sequentail file.

ArndW · Post by **ArndW** » Tue Aug 07, 2007 4:44 am

If by memory you mean disk space then both are going to use similar amounts of storage. If you mean memory usage while processing the dataset will most likely use more (but run faster) since it will have at least one reader process per node

ray.wurlod · Post by **ray.wurlod** » Tue Aug 07, 2007 5:57 am

Are you using unbounded VarChar data types (where no maximum length is specified)? Are you using bounded VarChar data types (where a maximum length is specified)?

lokesh_chopade · Post by **lokesh_chopade** » Tue Aug 07, 2007 6:06 am

what will be in both options? as such am using bounded data types.

ArndW · Post by **ArndW** » Tue Aug 07, 2007 4:04 pm

Ray - thanks for catching that; I had forgotten that DataSets will pad out VarChar strings and thus can use significantly more disk storage. We had a case here recently where a VarChar(800) column was used to store 15 characters of data - but for millions of rows. Just changing the data type significantly reduced the size and therefore the speed of the DataSet.

ray.wurlod · Post by **ray.wurlod** » Tue Aug 07, 2007 4:19 pm

On the other hand, Data Sets store numbers in binary format, which can be much more compact than storing them as text.

How long is a piece of string?

ArndW · Post by **ArndW** » Tue Aug 07, 2007 6:32 pm

A piece of string is ALWAYS exactly 42 long. Of course, the units used for measuring are always different.

Ed Purcell · Post by **Ed Purcell** » Tue Aug 07, 2007 7:00 pm

ArndW wrote:Ray - thanks for catching that; I had forgotten that DataSets will pad out VarChar strings and thus can use significantly more disk storage. We had a case here recently where a VarChar(800) column was used to store 15 characters of data - but for millions of rows. Just changing the data type significantly reduced the size and therefore the speed of the DataSet.

So, do I have this correct? Unbounded VarChar strings are greatly discouraged by the manufacturer. Bounded VarChars too have their pitfalls. A dataset will allocate the maximum declared length for a VarChar. If you specify a max length that is too big, then you waste lots of space and slow things down. Right?

ArndW · Post by **ArndW** » Tue Aug 07, 2007 7:12 pm

Right.

DSXchange

Who will use more memory Dataset or Seq. file

Who will use more memory Dataset or Seq. file

Further...