DSXchange

Hi
I have created schema file below format and i tried to run the job with
4 columns with data and it is working fine.
i tried to run the job with 2 columns only but it is not running and it is throwing error for 3rd column delimeter is missing.

can anybody suggest me how can i do .

record {final_delim=';', delim=',', quote=none,padchar='#'}
(
name: string[max=255];
addres: nullable string[max=255];
street: nullable string[max=220];
road: nullable string[max=200];
)

A schema file gives DS a list of columns and their definitions to a stage. By only passing 2 columns when its expecting 4 - then you are sending incomplete data - which will cause an error.

If you want to get around it, then you would have to pass different schema files.

Hi

from source we are not sure how many columns we receive and key columns should come definately and remaining not necessarily...
please suggest me how to do...

Are you mentioning from the source the number of columns is 4 but the columns can have null values. Or not sure on number of columns? Anyway you can try as:

If from source they cant provide 4 delimiters, then read the data as single column and split the data in to columns in transformer using field function

So you're saying that your target is static but your input (from a sequential file) is variable?

Edit: What Prasanna said - provided that they can always provided the columns in the same position each time. ie if there is a column X in the full schema in position 3, then column X will always arrive in position 3.

Hi

From soure i have 4 columns and runtime i need to pick 1 key column and any 2 of those 3 columns.
file is comma separated only,Please suggest me how to do this.

There are lot of option to get it done. But
Understand how the source team is generating data.

runtime i need to pick 1 key column and any 2 of those 3 columns

If there are values for only one non-key column will they send only one delimiter and to which column we will be mapping this?

Hi

From source we will receive 1 key column and out of 3 remaining columns only 2 columns ,that we are not sure which 2 columns out of 3 columns we receive.
example if 3rd colums is not comming then we dont need to send those column to target..

thats way i mentioned all possible columns in schema file.
please suggest me how to do this

raju4u wrote:From soure i have 4 columns and runtime i need to pick 1 key column and any 2 of those 3 columns.
file is comma separated only,Please suggest me how to do this.

When you say Any 2 of those 3 columns, are you suggesting that the input file could have a mix of columns?
eg
Row 1 of the file could be Key,Col1,Col2
Row 2 of the file could be Key,Col1,Col3
Row 3 of the file could be Key,Col2,Col3
Row 4 of the file could be Key,Col1,Col3

etc

Or would their be separate input files in that example?

No , it is in this format
Row 1 of the file could be Key,Col2,Col3
Row 2 of the file could be Key,Col2,Col3
Row 3 of the file could be Key,Col2,Col3

or in this format
Row 1 of the file could be Key,Col1,Col2
Row 2 of the file could be Key,Col1,Col2


or in this format

Row 1 of the file could be Key,Col1,Col3

Do you have static column headers provided with the file?

record {final_delim=';', delim=',', quote=none,padchar='#'}
(
name: string[max=255];
addres: nullable string[max=255];
street: nullable string[max=220];
road: nullable string[max=200];
)

Sorry I wasn't clear on that.

For each input file that you receive, will it have column headers?

So there will be always two comma delimiters only. Change the schema as

record {final_delim=';', delim=',', quote=none,padchar='#'}
(
name: string[max=255];
col1: nullable string[max=255];
col2: nullable string[max=255];
)

In the transformer, based on the business rule map the columns to target

I got it ,thank you very much prasanna for your input.