DSXchange

Hi Chulett It's not parameters that I want to select. It's the user-variables, and it seems they not in the External Parameter Helper list. This is what I want to do: I created a User-Variable, say CURRTIME, and popuplated it with the current date/time. It is then passed onto other jobs/scripts to b...

Hi We have this standard Exception Handler process like this: ExceptionHandler_Stage --> Email_Stage --> Terminator_Stage In the Email_Stage we can references parameters in the subject field, like this: "MyJob Contained the following exception on #$DS_ENV#: #Exception_Handler.$ErrMessage#"...

Hi I added the "nohup ... &" and it returns immediately. I have a new issue though: I needed it to wait for all these process to completed before the next step. Solution 1: I added the sequencer2 after these ExeCmds, like this (ignore the dots. They are there to align these words): ......

I didn't think about the background option for the unix command.


Thanks for all your helps

Hi I have a single Sequence job. Within it are five independent Execute Command stages, each calling a shell script, which then calls a Hive (Hadoop) command. At the beginning of this Shell Script I immediately log the system-time; at the end I log the time also. This way I would know when the shell...

I would like to understand whether the "Row Column Number" option in the BDFS file input step is impacted by partitioning or other parallelism options, or if it will always produce row numbers matching the source file.

When a job is optimized to use JAQL, the output files are broken up into a bunch of pieces labeled like part00001. Is there a way to have JAQL output a single file?

I am using the option 'File Name Column = ?' in the BDFS stage, and I would like to optimize the job as JAQL, but this option does not seem to come across in the optimized code. Is there a way, in JAQL, to manage this scenario: Many files in a folder on Hadoop Read in all the rows from all the files...

It looks to me like the Expand stage can only unpack a file that has been created in DataStage's own compressed format, is that correct? It cannot unzip an ordinary gzip file? I've tried using Sequential File stage and DataSet stage to read the zipped file and send to Expand, and it doesn't seem to ...

Had not tried the External Source stage. That seems to be working, with command: hadoop dfs -cat /DL/INCOMING/GOOGLE_DFA/temp/aaa.gz | gunzip -c Was wondering why the Expand stage does not seem to work, though. Zipped file is simply put onto hadoop for storage, we don't want to unzip them with the a...

I would like to be able to read a compressed (probably gzip, maybe other) file directly into Datastage from Hadoop. What is the correct way to do this? I have tried the Expand stage, but have not been able to make it work. (getting error The decode operator is being used on a non-encoded data set) W...

I have looked around on google and this forum, as well as the documentation, and experimenting for myself, and not found a solution. If there is no DataStage native solution, I would like confirmation that that is the case. I do not want to see a command-line scripting solution to this problem. It s...

I am on version 9.1.0.0 according to the Help page.

The TimestampToString() function is not effective, I see the same problem:

TimestampToString(lnk_src.estimate_start_date,"%m/%d/%yyyy %h:%nn")

"03/09/2013 00:00"

I have a file coming in which may have different column sets on different days. It has a header row, so it is possible to determine which columns exist on a given day, but I have not been able to find a way of getting the incoming file stage (HDFS stage) to understand the header and map the input co...

I am not getting the default, I am getting a non-default string that matches the string I expected except for the padding. The default string would be '%yyyy-%mm-%dd %hh:%nn:%ss'. The output format is specified in the output Sequential File stage, so it should output the value correctly.