DSXchange

Posted: **Wed Jul 09, 2014 4:10 am**

Hi

We have recently installed Datastage on Windows and ran simple job which uses Row Generator as Source, Transformation as Placeholder and Dataset as Target with 4 Columns and 10m records.

I took around 2 mins to load the Configuration file as below log entries:

Can you some one advise whether there is any problem with configuration?

Occurred: 14:34:01 On date: 08/07/2014 Type: Control Event: Starting Job Test_Job_Dataset_File. (...)
Occurred: 14:34:02 On date: 08/07/2014 Type: Info Event: Environment variable settings: (...)
Occurred: 14:34:02 On date: 08/07/2014 Type: Info Event: Parallel job initiated
Occurred: 14:34:02 On date: 08/07/2014 Type: Info Event: Parallel job default NLS map ASCL_MS1252, default locale OFF
Occurred: 14:34:03 On date: 08/07/2014 Type: Info Event: main_program: IBM InfoSphere DataStage Enterprise Edition 9.1.0.6791 (...)
Occurred: 14:34:03 On date: 08/07/2014 Type: Info Event: main_program: conductor uname: -s=Windows Server 2008 R2; -r=Service Pack 1; -v=6.1; -n=USAWS09WVDBS002; -m=x86-Intel
Occurred: 14:34:03 On date: 08/07/2014 Type: Info Event: main_program: orchgeneral: loaded (...)
Occurred: 14:35:43 On date: 08/07/2014 Type: Info Event: main_program: APT configuration file: E:/IBM/InformationServer/Server/Configurations/default.apt (...)
Occurred: 14:35:43 On date: 08/07/2014 Type: Info Event: Sequential_File_18,0: Export complete; 10000000 records exported successfully, 0 rejected.
Occurred: 14:35:43 On date: 08/07/2014 Type: Info Event: main_program: Step execution finished with status = OK.
Occurred: 14:35:44 On date: 08/07/2014 Type: Info Event: main_program: Startup time, 0:00; production run time, 1:39.
Occurred: 14:35:49 On date: 08/07/2014 Type: Info Event: Parallel job reports successful completion
Occurred: 14:35:55 On date: 08/07/2014 Type: Control Event: Finished Job Test_Job_Dataset_File.
Occurred: 15:12:02 On date: 08/07/2014 Type: Control Event: Starting Job Test_Job_Dataset_File. (...)
Occurred: 15:12:04 On date: 08/07/2014 Type: Info Event: Environment variable settings: (...)
Occurred: 15:12:04 On date: 08/07/2014 Type: Info Event: Parallel job initiated
Occurred: 15:12:04 On date: 08/07/2014 Type: Info Event: Parallel job default NLS map ASCL_MS1252, default locale OFF
Occurred: 15:12:05 On date: 08/07/2014 Type: Info Event: main_program: IBM InfoSphere DataStage Enterprise Edition 9.1.0.6791 (...)
Occurred: 15:12:05 On date: 08/07/2014 Type: Info Event: main_program: conductor uname: -s=Windows Server 2008 R2; -r=Service Pack 1; -v=6.1; -n=USAWS09WVDBS002; -m=x86-Intel
Occurred: 15:12:05 On date: 08/07/2014 Type: Info Event: main_program: orchgeneral: loaded (...)
Occurred: 15:14:19 On date: 08/07/2014 Type: Info Event: main_program: APT configuration file: E:/IBM/InformationServer/Server/Configurations/default.apt (...)
Occurred: 15:14:19 On date: 08/07/2014 Type: Info Event: Sequential_File_18,0: Export complete; 10000000 records exported successfully, 0 rejected.
Occurred: 15:14:19 On date: 08/07/2014 Type: Info Event: main_program: Step execution finished with status = OK.
Occurred: 15:14:19 On date: 08/07/2014 Type: Info Event: main_program: Startup time, 0:00; production run time, 2:12.
Occurred: 15:14:19 On date: 08/07/2014 Type: Info Event: Parallel job reports successful completion
Occurred: 15:14:20 On date: 08/07/2014 Type: Control Event: Finished Job Test_Job_Dataset_File.

Posted: **Wed Jul 09, 2014 6:49 am**

Don't worry about it, your loads are working fine, the timestamp on the log entry for your default config file is a little misleading as the message is generated (flushed) at the same time as the job completes. Test it with 1000 rows and you will see what I mean. As a matter of interest can you post your config file setting? It looks like you are running on a single node.

Posted: **Wed Jul 09, 2014 7:46 am**

Code: Select all

{
	node "node1"
	{
		fastname "DWBI"
		pools ""
		resource disk "E:/IBM/InformationServer/Server/Datasets" {pools ""}
		resource scratchdisk "E:/IBM/InformationServer/Server/Scratch" {pools ""}
	}
	node "node2"
	{
		fastname "DWBI"
		pools ""
		resource disk "E:/IBM/InformationServer/Server/Datasets" {pools ""}
		resource scratchdisk "E:/IBM/InformationServer/Server/Scratch" {pools ""}
	}
}

Posted: **Wed Jul 09, 2014 9:29 am**

Every job will have a bottleneck. In this case, you are writing to a sequential target file, and that step needs to be sequential. You know (we don't) how many bytes are in each row and what the final file size it. With that data you can compute the Mb/S write speeds to disk and that will most likely be pretty close to the system's limits. If you want this job to process more rows per second with a 2-node configuration, just write to a dataset.

Posted: **Wed Jul 09, 2014 9:35 am**

Note that the question asked in the original post was answered by battaliou. Anything beyond that should really be in another post rather than continuing to fall down this (potentially) endless rabbit hole. This in my not so humble moderator opinion.

Posted: **Wed Jul 09, 2014 11:32 am**

I only responded because no matter how many nodes you declare in your configuration file, you won't be doing any parallel writing to a sequential file.

DSXchange

Log Information

Log Information