dynamic creation on metadata

pattemk · Post by **pattemk** » Tue Jul 14, 2009 9:26 am

Hi,

I have a task where my input text file could be in any format each time.

I number of coulmns wil be different each time, i was wondering if it is possible to create the number of coulmns dynamically if i know how many coulmns do i need.

Please advice

chulett · Post by **chulett** » Tue Jul 14, 2009 10:37 am

Okay... who in the heck is supplying you data in a file that "could be in any format each time"? Sheesh. Now we need a RTM stage.

Can you expalin what "any format" actually means? What might change from file to file?

RTM = Read Their Mind

pattemk · Post by **pattemk** » Tue Jul 14, 2009 10:49 am

[quote="chulett"]Okay... who in the heck is supplying you data in a file that "could be in any format each time"? Sheesh. Now we need a RTM stage.

Can you expalin what "any format" actually means? What might change from file to file?

[size=7]RTM = Read [i]Their[/i] Mind[/size][/quote]

Thanks for replying,
Only the number of fields can differ each time.it could be 10 one time and caould be 30 one.

I was wondering if we can do this without making use os schema files.

Please suggest

chulett · Post by **chulett** » Tue Jul 14, 2009 11:25 am

More details, please. Do you always want just the first 10 no matter how many come or do you always need to process all columns?

pattemk · Post by **pattemk** » Tue Jul 14, 2009 12:19 pm

[quote="chulett"]More details, please. Do you always want just the first 10 no matter how many come or do you always need to process all columns? ...[/quote]

No, i might need to process all 10 columns or all 15 or 20.
i believe i can do it by dynamically creating schema files since i know the metadata before running the jobs. but wondering if i can do this without schema files.

Sainath.Srinivasan · Post by **Sainath.Srinivasan** » Tue Jul 14, 2009 2:53 pm

What do you mean by "process" ?

Will the transformation logic be different based on the layout of run instance ?

jcthornton · Post by **jcthornton** » Tue Jul 14, 2009 4:19 pm

The use of 10, 15, 20 and 30 to describe the number of fields implies that what is coming in is going to be in multiples of 5 (5 key fields perhaps and 5 repeating fields?). That sounds a lot like a pivot, which has been discussed many times in these forums. A search on 'pivot' will give you more information than you can shake a stick at.

If it isn't a pivot operation, the more details you provide, the better an answer that you will get.

ray.wurlod · Post by **ray.wurlod** » Tue Jul 14, 2009 4:23 pm

Schema File property?

Or don't they even inform you what the format is?

pattemk · Post by **pattemk** » Wed Jul 15, 2009 9:14 am

I got it resolved.

I making use of my current architecture, tables.

I am sure i can do it by making use of some tables that have been defined,i will know the format before running the jobs and each request can be in a different format.

I am parameterizing the whole logic that we put in the transformer derivation field.i tested it and its working.

Terala · Post by **Terala** » Wed Aug 12, 2009 10:10 pm

Hi Pattemk,

I have same kind of requirement, is it possible for you to share your dsx file reference?
Apprecaite