DSXchange

Hi,
I have a requirement like below and iam not able to proceed any further. can anybody let me know if there is a way to do the below task.

I have 5 Input files each of different metadata. I want to get the count of each file and write into a new file. I have created a script using wc -l and tried executing in DS sequencer (by calling through script) but it says invalid command. The same command is getting executed when i use the wc -l in execute command explicitly in Sequencer, but the output is not as expected.

Example:

Input:

Input file 1: Has 1 records
Input file 2: Has 2 records
Input file 3: Has 3 records
Input file 4: Has 4 records
Input file 5: Has 5 records

Output:

NewFile: Outputfilename.txt (should contains the value as below)
File1=1
File2=2
File3=3
File4=4
File5=5

I want this to be handled in script itself and i need to trigger the script in sequencer.
Thanks in advance.

How did you call the script in DataStage. Are you able to run your script and get the results outside of DataStage.

vinothkumar wrote:How did you call the script in DataStage. Are you able to run your script and get the results outside of DataStage.

Yes i can execute my script outside datastage. I am giving my script name in my user variblae in sequencer and using execute stage iam passing the parameters. The script is executing with all other commands but wc -l is giving the error like "Invalid command" but if i hardcode the same command in the execute command stage then its working.

Are you giving full path to the script in the sequencer job?

Try giving wc alone and see what happens.

anbu wrote:Are you giving full path to the script in the sequencer job?

Yes Iam giving full path.

vinothkumar wrote:Try giving wc alone and see what happens.

Getting the same error even with wc

Give the full path to 'wc' in the script. If you still have problems, post the script and the exact, unedited error message(s) from the log.

chulett wrote:Give the full path to 'wc' in the script. If you still have problems, post the script and the exact, unedited error message(s) from the log.

Input:

wc -l TargetFilePath/FileName > Countfilepath/CountFileName

Error:
User logged in.
Type set to A.
?Invalid command
You have transferred 0 bytes in 0 files.

And getting aborted

Alternative Option tried:

Gave the wc -l command inside the sequencer using execute stage

Input:

wc -l in command line
Countfilepath/CountFileName in the parameter

Output:

Executed: wc -l ; Countfilepath/CountFileName
Reply=126
Output from command ====>
0
SH: /opt/datastage/staging/rdsunit/control/NEWTEST.txt: 0403-006 Execute permission denied.

But when i hardcode the path, getting the output like below,

" count countpath/filename.txt"

where as getting permission denied if i parameterised it.

The script is created with my ID and it has 777 option. The same script is getting executed if i removed wc command and have some other command in the script.

I am using v7.5, let me know if there is some other alternative.
Thanks a lot.

**** This first error is from an FTP session. FTP does not include a "wc
" option.

Newtest.txt requires a "chmod +x" to execute


Does your Datastage job attempt to do a rsh to a secondary system where the script resides. The second test is a basic error in permissions. The file could not have permissions of 777

daignault wrote:**** This first error is from an FTP session. FTP does not include a "wc
" option.

Newtest.txt requires a "chmod +x" to execute


Does your Datastage job attempt to do a rsh to a secondary system where the script resides. The second test is a basic error in permissions. The file could not have permissions of 777

Can you please explain more, i dont understnad this point FTP does not include a "wc
" option.

AND

Does your Datastage job attempt to do a rsh to a secondary system where the script resides.

The scrpit has 777 option, the same script is getting executed when i run it with different command.

Post. The. Script.

Not one line from it or what you are putting in the stage but the entire contents of this "script" you are allegedly running. You have several problems here. Ray mentioned "ftp" because of this:

Error:
User logged in.
Type set to A.
?Invalid command
You have transferred 0 bytes in 0 files.

Those messages come from an ftp session, and you cannot execute "wc" within one without shelling out from it to the command line - hence the "Invalid command" message.

His next comment comes from the fact that, instead of redirection, for some reason you have a semicolon between your two commands:

Executed: wc -l ; Countfilepath/CountFileName

So first it does the "wc -l" without any other arguments and then tries to execute "Countfilepath/CountFileName" leading to the "chmod" comment.

Help us help you and post your complete script and explain what exactly it is it should be doing. Or if it is not in fact a script, make that clear. Help us understand what exactly it is you are doing as right now it is not clear at all.

Guessing from the flow of your logic, it appears you are ftp-ing a file from some server and checking the record count in the transmitted data.

So a successful ftp is essential for your flow.

Also chmod of your existing file will not be of much use as you may overwrite with the incoming file.

First fix the ftp issue.

Next check whether you are having a semi-colon after 'wc -l' and remove it (if present).

Hi Gopi,

If script is causing a problem, then try with the Aggregator stage for getting the row count and funnel all the results into a single Otuput file/dataset.

rohith - it sounds like he's trying to get a count of the file AFTER it is ftp'd, probably to verify it transmitted correctly. Getting the count out of the aggregator wouldn't help in that regards.