core.* files

[durgaps]

What is the significance of core.* files? They are present in the Project directory. E.g. core.9990 , what does the number 9990 indicate?

[chulett]

They are the output of a core dump when a program aborts / abends. Sometimes a vendor will be able to examine one to figure out what went wrong, but typically they just are just messy little piles of information that need to be cleaned up. It is not atypical for a cron job to exist that runs periodically and removes any that it finds.

As to the number... [shrug]... the pid that dumped, if I had to guess. Wouldn't really worry about it too much.

[ray.wurlod]

A core file is (may be) dumped when something goes wrong in the software. Because there are many processes in a parallel job, the name core has the process ID of the process that had the problem. You need to be able to identify which process was involved - the easiest way to achieve this is to set APT_SHOW_PIDS to True before invoking the job. Ask your UNIX system administrator about core files. There exist analysis tools, mainly within support organizations, that can extract diagnostic information from core files. A core file contains some or all of the process's memory.

[durgaps]

Thanks for the reply. In case these files need to be removed what is the best way to delete them. Manually do a delete or schedule it?

Thanks,

[chulett]

Schedule it... unless you enjoy manual labor, that is.

[ray.wurlod]

Hopefully they don't appear. So scheduling their removal ought not to be necessary. If they are appearing with regularity, then you have processes that are failing regularly, and you really should be finding out why.

[chulett]

Not unusual on UNIX systems to have regularly scheduled processes sweep the system for core files. They do tend to be fairly common. Too bad they aren't doing that here, just did a quick check of the limited areas on our dev server I can reach with my userid and found 56 of them.

[trokosz]

In addition, if your deleting the Core* files then go to the Project and cd "&PH&" and delete those temp files.

[durgaps]

What does this &PH& contain? I noticed files like DSD.RUN_66475_14179 , DSD.StageRun_70976_14179, SH_46330_14172 etc.

[chulett]

That is the 'phantom' directory. Cat out the contents of any of the files, that should help answer your question.

[ray.wurlod]

Each phantom (= background) process in DataStage directs its output to a file in the &PH& directory. The file name is made up from the program name then the time started (in DataStage internal format) then the date started (in DataStage internal format).

• DSD.RUN is the program that runs server jobs and job sequences.
  
  DSD.StageRun executes server and BASIC Transformer stages.
  
  SH executes the UNIX shell, which is used to invoke osh to run parallel jobs.

When a job aborts and is subsequently reset, the "from previous run" information is transferred from that job's output file in &PH&.

[n_suresh_82]

In our production environment, core files are getting created for particular DS job for every run. We did some analysis and found that there is a date format error and fixed the DS job on Feb 4. After we fixed the Ds job, we cleaned-up all the exisyting core files. But still the core file creation continues. When we examined the new core files after the DS job fix, we found that the log entries which appear in the new core files are the history logs. Please note the data timestamp in the log lesser than 4th Feb.

[prd:dsadm:batch:uscrm:dwapp1w29m3] sudo strings core.13411 | grep "yyyy" | tail
3324\2010-01-30 03:48:55\3\\89\TFM_Check_For_Empty_Values,0: Data string ' ' does not match format '%yyyy-%mm-%dd %hh:%nn:%ss': an integer was expected to match tag %yyyy.
3361\2010-01-31 03:57:08\3\\90\TFM_Check_For_Empty_Values,1: Data string ' ' does not match format '%yyyy-%mm-%dd %hh:%nn:%ss': an integer was expected to match tag %yyyy.
3363\2010-01-31 03:57:08\3\\90\TFM_Check_For_Empty_Values,0: Data string ' ' does not match format '%yyyy-%mm-%dd %hh:%nn:%ss': an integer was expected to match tag %yyyy.
3399\2010-02-01 03:32:02\3\\91\TFM_Check_For_Empty_Values,1: Data string ' ' does not match format '%yyyy-%mm-%dd %hh:%nn:%ss': an integer was expected to match tag %yyyy.
3401\2010-02-01 03:32:02\3\\91\TFM_Check_For_Empty_Values,0: Data string ' ' does not match format '%yyyy-%mm-%dd %hh:%nn:%ss': an integer was expected to match tag %yyyy.
3437\2010-02-02 03:38:10\3\\92\TFM_Check_For_Empty_Values,1: Data string ' ' does not match format '%yyyy-%mm-%dd %hh:%nn:%ss': an integer was expected to match tag %yyyy.
3439\2010-02-02 03:38:10\3\\92\TFM_Check_For_Empty_Values,0: Data string ' ' does not match format '%yyyy-%mm-%dd %hh:%nn:%ss': an integer was expected to match tag %yyyy.
3475\2010-02-03 03:45:32\3\\93\TFM_Check_For_Empty_Values,1: Data string ' ' does not match format '%yyyy-%mm-%dd %hh:%nn:%ss': an integer was expected to match tag %yyyy.
3476\2010-02-03 03:45:32\3\\93\TFM_Check_For_Empty_Values,0: Data string ' ' does not match format '%yyyy-%mm-%dd %hh:%nn:%ss': an integer was expected to match tag %yyyy.
TFM_Check_For_Empty_Values,1: Data string ' ' does not match format '%yyyy-Successm-4095d %:_date": Exporting nullable field without null handling properties

So how to stop the core file creation?

[ray.wurlod]

Prevent the error by using date strings that match the format strings supplied.

It's no surprise that the log contents are in the core file - after all they are in memory having been transferred through memory to the conductor process which looks after all logging.