External Source Stage 'ls' Command Issue

dspradeep · Post by **dspradeep** » Mon Dec 28, 2009 7:11 am

We have developed all the jobs in Windows environment after that we have moved into AIX environment.
In AIX, we are facing one isue in ESS stage.
In windows the below path we have all 1000 xml files.

Code: Select all

C:\IBM\InformationServer\Server\Projects\MS1_POC\Ms1\JobPackage

In the ESS stage I have used the command like

Code: Select all

ls C:\IBM\InformationServer\Server\Projects\MS1_POC\Ms1\JobPackage\*.xml

The above command to execute the job successfully but now in
AIX Environment
We have changed the path like

Code: Select all

 ls /PBMS1/input/AT/jobpackage/*.xml

now i am not able to view the file name in the ESS stage. when i am trying to view then it's giving the below error

Code: Select all

 ##I IIS-DSEE-TFCN-00001 08:08:19(000) <main_program> 
 IBM WebSphere DataStage Enterprise Edition 8.1.0.5040 
 Copyright (c) 2001, 2005-2008 IBM Corporation. All rights reserved
  
 
 
 ##I IIS-DSEE-TFCN-00006 08:08:19(001) <main_program> conductor uname: -s=AIX; -r=3; -v=5; -n=dwdbd1; -m=00C8A9C24C00
 ##I IIS-DSEE-TOSH-00002 08:08:19(002) <main_program> orchgeneral: loaded
 ##I IIS-DSEE-TOSH-00002 08:08:19(003) <main_program> orchsort: loaded
 ##I IIS-DSEE-TOSH-00002 08:08:19(004) <main_program> orchstats: loaded
 ##W IIS-DSEE-TOSH-00049 08:08:19(005) <main_program> Parameter specified but not used in flow: _Oracle_MS1_TRW_SID
 ##W IIS-DSEE-TOSH-00049 08:08:19(006) <main_program> Parameter specified but not used in flow: _Oracle_MS1_TRW_Schema_name
 ##W IIS-DSEE-TOSH-00049 08:08:19(007) <main_program> Parameter specified but not used in flow: _Oracle_MS1_TRW_UsrName
 ##W IIS-DSEE-TOSH-00049 08:08:19(008) <main_program> Parameter specified but not used in flow: _Oracle_MS1_TRW_UsrPwd
 ##W IIS-DSEE-TOSH-00049 08:08:19(009) <main_program> Parameter specified but not used in flow: _PB_MS1_AT_Jobpackage_Path
 ##W IIS-DSEE-TOSH-00049 08:08:19(010) <main_program> Parameter specified but not used in flow: DSProjectMapName
 ##I IIS-DSEE-TFSC-00001 08:08:19(016) <main_program> APT configuration file: /datastage/IBM/InformationServer/Server/Configurations/default.apt
>##E IIS-DSEE-TOIX-00161 08:08:19(000) <External_Source_61,0> Filter status 589,833;
>filter process died from signal 9;
>import error at record 0.
>##E IIS-DSEE-TOIX-00179 08:08:19(001) <External_Source_61,0> Import error at record 0.
>##E IIS-DSEE-TFOR-00089 08:08:19(002) <External_Source_61,0> The runLocally() of the operator failed.
 ##I IIS-DSEE-TFOR-00094 08:08:19(003) <External_Source_61,0> Output 0 produced 0 records.
 ##I IIS-DSEE-USBP-00001 08:08:19(000) <APT_CombinedOperatorController,0> called postFinalRunLocally
>##E IIS-DSEE-TFOR-00089 08:08:19(004) <APT_CombinedOperatorController,0> The runLocally() of the operator failed.
>##E IIS-DSEE-TFPM-00040 08:08:19(005) <APT_CombinedOperatorController,0> Operator terminated abnormally: runLocally() did not return APT_StatusOk
>##E IIS-DSEE-TFSC-00011 08:08:25(000) <main_program> Step execution finished with status = FAILED.
 ##I IIS-DSEE-TCOS-00026 08:08:25(001) <main_program> Startup time, 0:05; production run time, 0:01.

If I run the job I am getting the below error

Code: Select all

 /usr/bin/ls: 0403-027 The parameter list is too long

[/code]

chulett · Post by **chulett** » Mon Dec 28, 2009 7:38 am

That is a very common UNIX problem that googling for "The parameter list is too long" would reveal the answers for.

dspradeep · Post by **dspradeep** » Mon Dec 28, 2009 8:04 am

Ok thanks Chulett, I will do the search in google also if any one knows what exactly I need to do for that please post here.

chulett · Post by **chulett** » Mon Dec 28, 2009 8:12 am

*I* know but there are several solutions for this and you need to learn how to find your own answers, especially when they are easily found with two minutes effort. Teach a man to fish and all that.

dspradeep · Post by **dspradeep** » Mon Dec 28, 2009 9:12 am

Please tell me since I am not able to find out the exact solution for that.

chulett · Post by **chulett** » Mon Dec 28, 2009 9:50 am

Then you're not really looking. Here's the first hit from a Google search, lays it all out:

http://www.tek-tips.com/viewthread.cfm? ... 140&page=1

dspradeep · Post by **dspradeep** » Tue Dec 29, 2009 4:10 am

Hi

I have checked what ever you given link and some other link also.

but i don't know how to use these in ESS stage since for loop and all i don't think i can able to use in ESS stage

chulett · Post by **chulett** » Tue Dec 29, 2009 6:35 am

You would have to script the 'for' loop and the run the script in the ESS. Did you not try the 'find' command?

dspradeep · Post by **dspradeep** » Tue Dec 29, 2009 10:46 am

I have tried the "find" command and it's working but I feel it's taking some extra time compare to ls command. so I am looking some other command.

chulett · Post by **chulett** » Tue Dec 29, 2009 10:51 am

How much "extra time"? Enough to make this search for another way worth your time? I sincerely doubt it.

ray.wurlod · Post by **ray.wurlod** » Tue Dec 29, 2009 3:41 pm

Perhaps dspradeep is being paid by the hour?

dspradeep · Post by **dspradeep** » Wed Dec 30, 2009 3:52 am

In windows we have used 'ls' command which was taken not even 1 min to complete 1000 xml but now in AIX environment there is some limitation for ls command so we could not use this and we have tried different command but nothing work out for me.
Finally

Code: Select all

 find <path> -name "*.xml"

command working fine but it's taking 27 min to complete 300 xml. The volume of data also very less compare to windows which we are processed earlier.

Please any one help me in this.

chulett · Post by **chulett** » Wed Dec 30, 2009 7:01 am

There's no way the find command is taking 27 minutes. Seek your bottleneck elsewhere.