Job using dataset files is slower than sequential files

[ArndW]

...Both versions output the file to the same folder...

No, probably just your dataset descriptor and sequential file are in the same directory. You should try writing your sequential file to the directory pointed to by the resource disk setting in your APT_CONFIG file. That way you are comparing performance on the same disk partition and removing potentional I/O differences from the equation.

[patonp]

I've read in another post that datasets containing bounded-length varchar fields can grow to be be quite large as they allocate almost the full amount of space defined, even when only a few characters of the varchar field are actually used.

Is the total size of your datasets much larger than your sequential file? (i.e. could the I/O involved writing out a larger set of files be the cause of your performance discrepancy?)

[splayer]

Arndw, when I changed my resourcedisk path to the Datasets folder where datasets are created, the time taken for sequential file was the same as for data set files, 69secs. So depending on the folder, performance varies. Do you have any idea as to why it might be?

I would think that dataset version should at least be a few seconds faster.

[ArndW]

splayer - look at the filesystems and options used for your two partitions.

[splayer]

This is my job:
SeqFile --> Modify --> SurrogateKeyGenerator --> Transformer --> TargetFile(Dataset stage)

To answer balajisr's question, partitioning is set to Auto throughout.

[ray.wurlod]

Arnd means to look at the hardware. For example, is one directory on local disk and the other in a SAN?

[kumar_s]

That particular directory can be of different mount point, which might have network congestion.

[splayer]

Ray, pardon my ignorance about hardware but what does SAN stand for?

[DSguru2B]

Storage Area Network

[chulett]

Not to be confused with NAS.

[ray.wurlod]

Storage Area Network

More usually Storage ARRAY Network - an array of storage devices (disks) connected with intelligent controllers so that they can be managed as a single entity or partitioned as required.