DSXchange

Hi All,

How can we implement dynamism in DataStage ? My requirement is there is a metadata file which contains the column name , length and thecolumn order and there is another input file which contains the data corresponding to the metadata file . The metadata file is dynamic in nature i.e. the no. of columns can vary in the file. Please suggest me a way to implement the above functionality.

Thanx in advance.

What is the format of your metadata file? Can you give an example?

I_Server_Whale wrote:What is the format of your metadata file? Can you give an example?

The metadata file is a csv file. It contains three columns :

ColumnName ColumnWidth ColumnOrder
Name 10 1
EmpId 10 3
Salary 10 2

The Metadata file is more or less like this , but the rows may increase or decrease and that is the dynamism involved. The rows needs to be sorted according to the column order and mapped with the input data file and finally load in a another file.

I am unaware of any way to achieve this in server jobs.

Doubtless some kind of wizard could be written to read the file and create the table definition in the DataStage Repository - indeed, I've done that, but what becomes of the metadata after that? There's no extra information in the file about what to do with it. And, for text files, whence is the information about the format to be obtained?

ray.wurlod wrote:I am unaware of any way to achieve this in server jobs.

Doubtless some kind of wizard could be written to read the file and create the table definition in the DataStage Repository - indeed, I've done that, but what becomes of the metadata after that? There's no extra information in the file about what to do with it. And, for text files, whence is the information about the format to be obtained?

Can you go ahead with the wizard part, may be that can be of some help.

Too busy right now (at least for a week).

Invoking a Wizard is generally considered risky... best not to anger them with unreasonable demands.

Even if you do get this 'some help' I don't see how it's really going to help. The functionality you seek isn't available if you are expecting to be able to handle this kind of information 'on the fly' inside one job.

Something could be built to automagically build jobs from a template if you have the skillz, one for each unique combination of file values. By the time you do that however, you could have created a true 'template' job and spun off the jobs yourself by hand I would think.

chulett wrote:Invoking a Wizard is generally considered risky... best not to anger them with unreasonable demands.

Even if you do get this 'some help' I don't see how it's really going to help. The functionality you seek isn't available if you are expecting to be able to handle this kind of information 'on the fly' inside one job.

Something could be built to automagically build jobs from a template if you have the skillz, one for each unique combination of file values. By the time you do that however, you could have created a true 'template' job and spun off the jobs yourself by hand I would think.

What about RCP property ?

chulett wrote:Invoking a Wizard is generally considered risky... best not to anger them with unreasonable demands.

Even if you do get this 'some help' I don't see how it's really going to help. The functionality you seek isn't available if you are expecting to be able to handle this kind of information 'on the fly' inside one job.

Something could be built to automagically build jobs from a template if you have the skillz, one for each unique combination of file values. By the time you do that however, you could have created a true 'template' job and spun off the jobs yourself by hand I would think.

What about RCP property ?

What about it? Are we moving now from Server to PX jobs?

Don't see how it would help as it can't make metadata up out of whole cloth.

RCP is not present in server jobs.

DSguru2B wrote:RCP is not present in server jobs.

If this requirement can be met in Parallel jobs , then no issues.

RCP is a good option that px jobs have but thats just for propogating the records. For any explicit transformations, the columns need to be defined.
And Yes, your requirement can be met in px jobs as long as you understand about the limitations of rcp.

DSguru2B wrote:RCP is a good option that px jobs have but thats just for propogating the records. For any explicit transformations, the columns need to be defined.
And Yes, your requirement can be met in px jobs as long as you understand about the limitations of rcp.

I have tried the RCP option and created the schema file that contains the metadata information , but the job is getting aborted with the following error :
Delimiter for field "Store" not found; input: <empty>, at offset: 8

The file that I am currently reading has comma as the filed delimeter and same has been defined in the job also, but its not working. Please provide a solution.