Copy Vs Transformer..
Moderators: chulett, rschirm, roy
Copy Vs Transformer..
i have a .ds file with 2.6billion records.
i just want to have a dump of this file.
i can use a copy stage but just my curiosity can we use a tranformer in place of copy..??
i know that transfomer is robust and compiles in C++ then orch..
but is the same process time for copy and transformer..??
as per my knowledge the performance degradation of a job will effect if that job has... say 6+ transformers...(Datastage v 8.1)
so for a basic copy function can we use transformer..??
i just want to have a dump of this file.
i can use a copy stage but just my curiosity can we use a tranformer in place of copy..??
i know that transfomer is robust and compiles in C++ then orch..
but is the same process time for copy and transformer..??
as per my knowledge the performance degradation of a job will effect if that job has... say 6+ transformers...(Datastage v 8.1)
so for a basic copy function can we use transformer..??
....................Shanthi
I ve tried both the ways..
and had to abort them...reasons being...
both the cases...the processing started around 120-140k rows/sec and stable around 60k rows/sec.
at this speed loading 2.6billion...????
my observation was the diff in comipiling time...
processing time both took almost the same rows/sec.
So, wanted to know ...
also is there any other method to have a copy of .ds..???
and had to abort them...reasons being...
both the cases...the processing started around 120-140k rows/sec and stable around 60k rows/sec.
at this speed loading 2.6billion...????
my observation was the diff in comipiling time...
processing time both took almost the same rows/sec.
So, wanted to know ...
also is there any other method to have a copy of .ds..???
....................Shanthi
Compile time will be different, longer with the transformer initially because of the need to create the C++ source and then compile it, where with the copy stage it merely adds the stage to the OSH script to be executed at runtime.
The simplest job for copying a ds to another ds? DataSet-->DataSet. No need for the copy stage in the job design. You can use job monitor to see the progress instead of performance statistics, or better yet use the performance analysis feature.
At this point, you are probably limited by:
1) The maximum throughput of your hardware (storage, network, server)
2) Transport buffer sizes within the parallel engine (tunable)
Regards,
The simplest job for copying a ds to another ds? DataSet-->DataSet. No need for the copy stage in the job design. You can use job monitor to see the progress instead of performance statistics, or better yet use the performance analysis feature.
At this point, you are probably limited by:
1) The maximum throughput of your hardware (storage, network, server)
2) Transport buffer sizes within the parallel engine (tunable)
Regards,
- james wiles
All generalizations are false, including this one - Mark Twain.
All generalizations are false, including this one - Mark Twain.
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Re: Copy Vs Transformer..
No you don't. The .ds file contains no records whatsoever. It's a descriptor file that describes locations of physical files that do contain your data.bskumar4u wrote:i have a .ds file with 2.6billion records.
Another way to make a copy of a data set is to use the orchadmin cp command. This does not generate any rate statistics (e.g. rows/sec) but you can still measure its elapsed time.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.