Space Occupied by datastage logs

[samyamkrishna]

Hi,

I have created a job to check how good the source file is.

For every import error there will be a warning.

We read the logs and identify the columns which have junk data in them.

Now if the source file has 6 million records and worst case all of them have junk data the director log is going to be massive.

wanted to do an impact analysis to see how this would impact datastage sever.

Regards,
Samyam

[PaulVL]

pardon me... yuk.


Don't inject a warning message in your job execution log which valdiates data quality of your input data. I would rather create a seperate file which contains the dirty records and one that contains valid records.
You could have a COUNT value echoed to your job execution log if you wish.

Always plan for worst case, and that would be that you have suddenly injected 6million rows of log file information into your UV database. You do not want to do that.

What do you plan on doing with the warning messages?
Did your site turn off the default 50 warnings and you abort setting?

[qt_ky]

You could also using Information Analyzer. If you don't have it, maybe ask for a trial.

[samyamkrishna]

Thanks for the info you guys gave....

We already have information analyser will do it using that.

But i would like to know how the logs take up space in UNIX or in XMETA.

[chulett]

Where are you logging to, the 'legacy repository' or to XMETA? The former will have an ~2GB maximum size per 'log' and if that is exceeded they will corrupt.

[samyamkrishna]

Datastage is logging into XMETA.

If the job creates a log with 6 million warning. is there a possibilty of over loading the database.

[chulett]

Of course and that would be true of any table. The obvious solution is to correct your job so it doesn't log extraneous message, especially millions of them.