Regarding schema file

[raju4u]

Hi
I have created schema file below format and i tried to run the job with
4 columns with data and it is working fine.
i tried to run the job with 2 columns only but it is not running and it is throwing error for 3rd column delimeter is missing.

can anybody suggest me how can i do .

record {final_delim=';', delim=',', quote=none,padchar='#'}
(
name: string[max=255];
addres: nullable string[max=255];
street: nullable string[max=220];
road: nullable string[max=200];
)

[ShaneMuir]

A schema file gives DS a list of columns and their definitions to a stage. By only passing 2 columns when its expecting 4 - then you are sending incomplete data - which will cause an error.

If you want to get around it, then you would have to pass different schema files.

[raju4u]

Hi

from source we are not sure how many columns we receive and key columns should come definately and remaining not necessarily...
please suggest me how to do...

[prasannakumarkk]

Are you mentioning from the source the number of columns is 4 but the columns can have null values. Or not sure on number of columns? Anyway you can try as:

If from source they cant provide 4 delimiters, then read the data as single column and split the data in to columns in transformer using field function

[ShaneMuir]

So you're saying that your target is static but your input (from a sequential file) is variable?

Edit: What Prasanna said - provided that they can always provided the columns in the same position each time. ie if there is a column X in the full schema in position 3, then column X will always arrive in position 3.

[raju4u]

Hi

From soure i have 4 columns and runtime i need to pick 1 key column and any 2 of those 3 columns.
file is comma separated only,Please suggest me how to do this.

[prasannakumarkk]

There are lot of option to get it done. But
Understand how the source team is generating data.

runtime i need to pick 1 key column and any 2 of those 3 columns

If there are values for only one non-key column will they send only one delimiter and to which column we will be mapping this?

[raju4u]

Hi

From source we will receive 1 key column and out of 3 remaining columns only 2 columns ,that we are not sure which 2 columns out of 3 columns we receive.
example if 3rd colums is not comming then we dont need to send those column to target..

thats way i mentioned all possible columns in schema file.
please suggest me how to do this

[ShaneMuir]

From soure i have 4 columns and runtime i need to pick 1 key column and any 2 of those 3 columns.
file is comma separated only,Please suggest me how to do this.

When you say Any 2 of those 3 columns, are you suggesting that the input file could have a mix of columns?
eg
Row 1 of the file could be Key,Col1,Col2
Row 2 of the file could be Key,Col1,Col3
Row 3 of the file could be Key,Col2,Col3
Row 4 of the file could be Key,Col1,Col3

etc

Or would their be separate input files in that example?

[raju4u]

No , it is in this format
Row 1 of the file could be Key,Col2,Col3
Row 2 of the file could be Key,Col2,Col3
Row 3 of the file could be Key,Col2,Col3

or in this format
Row 1 of the file could be Key,Col1,Col2
Row 2 of the file could be Key,Col1,Col2


or in this format

Row 1 of the file could be Key,Col1,Col3

[ShaneMuir]

Do you have static column headers provided with the file?

[raju4u]

record {final_delim=';', delim=',', quote=none,padchar='#'}
(
name: string[max=255];
addres: nullable string[max=255];
street: nullable string[max=220];
road: nullable string[max=200];
)

[ShaneMuir]

Sorry I wasn't clear on that.

For each input file that you receive, will it have column headers?

[prasannakumarkk]

So there will be always two comma delimiters only. Change the schema as

record {final_delim=';', delim=',', quote=none,padchar='#'}
(
name: string[max=255];
col1: nullable string[max=255];
col2: nullable string[max=255];
)

In the transformer, based on the business rule map the columns to target

[raju4u]

I got it ,thank you very much prasanna for your input.