Posted: Mon Feb 09, 2004 10:11 am
Woaw, what a furious excitation around this matter. Please, no, no, it's too much, don't involve yourselves that way :)
... or :( , i don't know.
... or :( , i don't know.
We also found a memory leak in the OLEDB stage that caused some a failue when the memory usage hit the 2 GB window. We have 16 GB in our server so I am not sure if it would be the same in you configuration.essaion wrote:Thanks for your answers !
(... Well, not to Craig, who laughes at those using DataStage on Windows)
The facts are :
- OS takes about 100-150 MB of RAM (seen at the server startup)
- SQL server can take up to 1 GB of RAM and 100% CPU ressources (this is, by the way, really annoying when it occurs). Let's say it takes an average 500 MB of RAM / swap.
- i never saw the ever-dying process take more than 700 MB (it howles an "ARGGGHHH" before)... The 1 GB mentionned is seen on the "Performances" tab ; the "Processes" one shows a 280-750 MB max used memory (by uvsh.exe).
The OS never complains about a low memory issue...
Again, thanks for your answers !
So the only way i see to resolve this issue is... adding RAM onto the server, at this time (shrinking source files is not possible in this case).essaion wrote:If you see an other-and-less-memory-blaster way to do this using DataStage (not a command)
Code: Select all
FUNCTION DTM(PathName)
* History (most recent first)
* Date Programmer Version Details of Modification
* ---------- ---------------- ------- ----------------------------------
* 04/11/2004 Ray Wurlod 2.0.0 Initial coding
*
$INCLUDE UNIVERSE.INCLUDE FILEINFO.H
DEFFUN OpenTextFile(Filename, OpenMode, AppendMode, LoggingFlag) Calling "DSU.OpenTextFile"
Ans = @NULL
hFile = OpenTextFile(PathName, "R", "O", "N")
If FileInfo(hFile, FINFO$IS.FILEVAR)
Then
Status FileStuff From hFile
Then
TimeModified = FileStuff<15>
DateModified = FileStuff<16>
Ans = Oconv(DateModified, "D-YMD[4,2,2]") : " " : Oconv(TimeModified, "MTS:")
End
End
RETURN(Ans)
I just ran into an error on the folder stage as well. The job is:ray.wurlod wrote:It's either a bug in the Folder stage, or that one of the ulimit settings is too small. Will investigate further Monday and post results.
Code: Select all
FOLDER -> TFM -> SEQ
Code: Select all
DataStage Job 355 Phantom 17015
Program "JOB.2091227139.DT.1340829553.TRANS1": Line 37,
Available memory exceeded. Unable to continue processing record.
DataStage Phantom Finished
Code: Select all
* Tokens were replaced below as follows:
* JobParam%%1 <= DATAFILEPATH
* Pin%%V0S57P1.Column%%1 <= LN_DATAFILEPATH.COL_1
* GET.Pin%%V0S57P1 <= GET.LN_DATAFILEPATH
* Pin%%V0S57P1.REJECTEDCODE <= LN_DATAFILEPATH.REJECTEDCODE
* Pin%%V0S57P2.Column%%1 <= LN_TFM_01.COL_1
* PUT.Pin%%V0S57P2 <= PUT.LN_TFM_01
* Pin%%V0S57P2.REJECTED <= LN_TFM_01.REJECTED
* Pin%%V0S57P2.REJECTEDCODE <= LN_TFM_01.REJECTEDCODE
*
* Subroutine for active stage SIIPSTransMerge01DailyFilesListed.TFM_01 generated at 15:45:30 17 NOV 2004
*
SUBROUTINE DSTransformerStage(HANDLES,ERROR)
$INCLUDE DSINCLUDE DSD_RTCONFIG.H
$INCLUDE DSINCLUDE DSD_STAGE.H
$INCLUDE DSINCLUDE DSD_BCI.H
$DEFINE JobParam%%1 STAGECOM.JOB.STATUS<7,1>
$INCLUDE DSINCLUDE JOBCONTROL.H
DEFFUN DSRLoadString(Num,Text,Args) CALLING '*DataStage*DSR_LOADSTRING'
$DEFINE Pin%%V0S57P1.Column%%1 STAGECOM.ARR(1)
$DEFINE GET.Pin%%V0S57P1 CALL $DS.UVGETNEXT(1,Pin%%V0S57P1.REJECTEDCODE)
IF STAGECOM.TRACE.STATS THEN CALL $PERF.NAME(-2,'LN_TFM_01.Derivation')
$DEFINE Pin%%V0S57P2.Column%%1 STAGECOM.ARR(2)
$DEFINE PUT.Pin%%V0S57P2 CALL $DS.SEQPUT(2, Pin%%V0S57P2.REJECTEDCODE)
UPDATE.COUNT = STAGECOM.RATE
LOOP
REJECTED = @TRUE
* Get next row from primary input pin LN_DATAFILEPATH
STAGECOM.PINNO = 1
GET.Pin%%V0S57P1
ERROR = Pin%%V0S57P1.REJECTEDCODE
WHILE NOT(ERROR)
STAGECOM.PINNO = 2
IF STAGECOM.TRACE.STATS THEN CALL $PERF.BEGIN(-2)
IF @TRUE THEN
* Column derivation code for pin LN_TFM_01
Pin%%V0S57P2.Column%%1 = (JobParam%%1 : Pin%%V0S57P1.Column%%1)
Pin%%V0S57P2.REJECTED = @FALSE
IF STAGECOM.TRACE.STATS THEN CALL $PERF.END(-2)
PUT.Pin%%V0S57P2
IF NOT(Pin%%V0S57P2.REJECTEDCODE) THEN
REJECTED = @FALSE
END ELSE
Pin%%V0S57P2.REJECTED = @TRUE
END
END
ELSE
Pin%%V0S57P2.REJECTED = @TRUE
Pin%%V0S57P2.REJECTEDCODE = 0
END
UPDATE.COUNT -= 1
IF UPDATE.COUNT LE 0 THEN CALL DSD.Update(HANDLES);UPDATE.COUNT = STAGECOM.RATE
REPEAT
RETURN
END
Code: Select all
/udd003/Development/DataFiles/SEQ_SIIPS_MERGE_20041001.SEC
/udd003/Development/DataFiles/SEQ_SIIPS_MERGE_20041002.SEC
/udd003/Development/DataFiles/SEQ_SIIPS_MERGE_20041004.SEC
....
.....
/udd003/Development/DataFiles/SEQ_SIIPS_MERGE_20041110.SEC
/udd003/Development/DataFiles/SEQ_SIIPS_MERGE_20041111.SEC
/udd003/Development/DataFiles/SEQ_SIIPS_MERGE_20041112.SEC
/udd003/Development/DataFiles/SEQ_SIIPS_MERGE_20041113.SEC
/udd003/Development/DataFiles/SEQ_SIIPS_MERGE_20041115.SEC
/udd003/Development/DataFiles/SEQ_SIIPS_MERGE_20041116.SEC
Code: Select all
/udd003/Development/DataFiles/SEQ_SIIPS_MERGE_OVERFLOW.SEC
Yes! There is definitely a limit on the file sizes the Folder stage can handle. Unfortunately, I don't believe there is a hard-and-fast number published by Ascential... it seems to vary by hardware and version.denzilsyb wrote:Are file sizes an issue?
I remember that the official answer to this question in the Microsoft exam for Windows NT 4.0 Administrator was that the operating system requires 16MB.OS takes about 100-150 MB of RAM (seen at the server startup)