DSXchange

...I checked with DS admin for I/O process during that time and it was quite minimal it seems and so the CPU usage. Something isn't correct here - either your CPU or your ... I'm sorry..I meant to say "I checked with DS admin for I/O process during that time and it was quite minimal it seems a...

This issue is probably more closely related to how you have defined the record layout and the output record. If you include the Group etc... then it will take a bunch of time to read the data especially with rows that wide. I just had this issue with an 18gb file and 1800 columns and I was able to ...

ArndW wrote:Do you also have "Read from multiple Nodes" turned on? ...

Yes.Its turned on.

ArndW wrote:What is your CPU usage during the job run? At the moment you don't know if I/O, I/O per process, or CPU Usage is the bottleneck. ...

I checked with DS admin for I/O process during that time and it was quite minimal it seems and so the CPU usage.

This issue is probably more closely related to how you have defined the record layout and the output record. If you include the Group etc... then it will take a bunch of time to read the data especially with rows that wide. I just had this issue with an 18gb file and 1800 columns and I was able to ...

How many CPUs does your system have and how many nodes are in your APT_CONFIG_FILE? Can you experiment on timing using multiple readers per node to see if you speed up reading your file (in order to test this, write to a PEEK stage instead of to the dataset) We have 4 nodes in config file. I've rai...

You have two problem areas, and I'm not sure which you consider: a) reading from flat file and writing to a dataset b) reading the dataset. Is your mainframe file fixed length? If so, you might ... My Mainframe file is fixed. Here it goes my design for better understanding. I have Mainframe fixed w...

Hi, All, I have Mainframe file as source and it is of 6GB size and have around 3500 fields. I tried to read this file with Sequential file stage using Schema File and RCP options and pass it to a dataset thru' transformer. It is taking around 25 minutes to read the data from dataset in another job (...

I have used this command also dscc in command prompt, this also takes the same time.

There are 3 transformers and 3 lookup stages.Input dataset has about 1145 columns and there are 11 output datasets, almost all of the input columns are required.

Only this job takes a long time to compile, also this job always takes about 4 hrs to compile.Even import of this job takes about 30 mins.

Job compilation takes about 4 hrs, is there any way to reduce the compilation time?
There are 1145 columns in the input file and all the columns are used for transformation.

Actually my requirement was to do develop a parallel job,how can we implement this using server job?Can you please help me on this?

No I have to use Parallel job only.

Ya I have checked the PXPivot stage,even this stage is not having RCP option. My requirement is like e.g. Key_A Col_A Col_B...................Col_N 100 fgh abc...................... xyz Then I should get the output as Key_A ColumnNames ColumnValues 100 Col_A fgh 100 Col_B abc ..... ..... ..... 100 C...