Sequential File path query

[chandra.shekhar@tcs.com]

Hi All,
I have a small job where the source is a Sequential File.
The path of the file will be

Code: Select all

/data/XXX/20130420/A.txt

The path will contain a date folder(without hyphens) as shown above.
The job will run daily and so the folder will also change according to the run date e.g 20130421, 20130422 etc.

I have a parameter set where one of the parameter is $BusinessDate and its value can only be a normal date e.g 2013-04-20, 2013-04-21 etc.
So my question is how to parametrize the date folder using this BusinessDate parameter ?

[BI-RMA]

Have a second parameter on the job like BusinessDate_YYYYMMDD
and pass Convert('-','',$BusinessDate) to it in the Sequence.

[chandra.shekhar@tcs.com]

@BI-RMA
Thanks for your input, but we have a constraint at our end that we dont use sequences and secondly I can't use another parameter.
Cant there be something done using only BusinessDate parameter which I have mentioned earlier ?

[BI-RMA]

@BI-RMA
... and secondly I can't use another parameter.

WHY?

[chandra.shekhar@tcs.com]

OK, even if I can use another parameter but I cant create a sequence here.
Actually in our project we do not create sequences and thats the way we work.
Even if I create a sequence, my TL/client wont approve it.
So now tell me what options do I have ?

[jerome_rajan]

1. Before Job Script: Create Temporary Folder

2. In Job: Write your file into the temp folder created

3. After job script:
Move file from temp folder to folder with name

Code: Select all

echo #$businessdt# | tr -d "-"

Then remove temp folder

[chandra.shekhar@tcs.com]

2. In Job: Write your file into the temp folder created

How ?
How can I write the files when I cant even read them ?
As told above, daily new date folders will be created so I need a parameter to read that folder.

[BI-RMA]

Even if you don't use sequences you do pass the parameters to your job in some way, likely by means of some scheduling-tool and a command-line call of dsjob -run.

I don't know which options your scheduler supports to manipulate the value passed to your job as parameter BusinessDate_YYYYMMDD, but I'm almost sure it does provide options to get rid of the hyphen-separators, maybe something like the echo-command jerome proposed...

[chandra.shekhar@tcs.com]

We use UNIX/Perl scripts for scheduling of jobs.
Those scripts pass the parameters to the jobs at run time.
These scripts are generic in nature i.e. I cant change the script to as to accommodate 1 more parameter.
This particular job run in Production but due to some client policy, the client is now creating one 1 more folder which is the date folder and we have to read the file from there.
Whatever I have to do, I have to do it in the datastage itself.
I know I have very limited options.

[chulett]

With the restrictions you've posted, I think you are basically... what's the technical term... screwed. I'll ponder this and I'm sure others are as well so perhaps someone can think of something to unscrew you.

The problem is your parameter needs to be used right away in the job for your source stage. You've got no time or mechanism to manipulate it before it is needed in the DataStage job that it is passed to. In any other shop, you would use a Sequence job to manipulate it to remove the dashes and then pass the transformed value to the job. Or the job control would do it before passing it. Being locked into generic job control is fine and not all that unusual but not being able to use Sequence jobs? That's just plain stupid. Out of curiosity, what's their justification for that?

[BI-RMA]

Which means: all your jobs have to share the same set of identical parameters, so that you are able to schedule them all through the same UNIX/Perl-Scripts?

Does not sound that generic to me...

All kinds of transformations are possible within pearl to generate calls of dsjob with different sets of parameters for individual jobs. It seems to me that your framework is too inflexible for things like what your customer has in mind.

[chulett]

I'm assuming 'generic' in this case means that the job control queries the job and gets a list of the parameters that it needs so it knows what to pass to each one. But that would be just a guess.

[chandra.shekhar@tcs.com]

With the restrictions you've posted, I think you are basically... what's the technical term... screwed

Well thats another way to describe my position
My client is a stubborn company , we dont have the flexibility to ask why, how, when etc.. We do it what we are asked to do.

@BI-RMA/Craig

Which means: all your jobs have to share the same set of identical parameters, so that you are able to schedule them all through the same UNIX/Perl-Scripts

Yes all of our jobs share same parameters.
We have 3-4 scripts to run all the jobs, jobs are divided into batches.
If I change one script to accommodate one more parameter for this particular job then it may affect other jobs' running too.
And changing script is not as simple as it sounds, it will involve raising defect, then client approval.

[BI-RMA]

That would be great, because then you should be able to add parameters to the jobs as needed. But according to chandra the scripts are unable to accomodate an additional parameter on any one job. Which sounds to me like the whole setup was pretty much hardwired...

[eph]

Hi,

Maybe you can use an external source stage using an unix command like

Code: Select all

cat /data/XXX/`date "+%Y%m%d"`/A.txt

You might also be able to call a script with parameters which could echo the file content as you like.

Eric