Hi,
I have a job that loads data into a sequential file. The jobs takes close to 30 minutes to finish.
If I try to view the data in the sequential file from DataStage designer when the job is running I was able to see the records in the file until a few days ago. But I have noticed that from past few days if I try to view the sequential file data when the job processing records into it it says "data source is empty" even if the performance statistics for the link shows a few number of records went into the file. Another interesting thing is if I wait for some more records to be processed into the file and then try to view the data the data shows up.
I think the data does not physically go into the sequential file until the job is either finished or has processed atleast a certain number of records.
Is there any commit size (like transaction size for RDBMS) for sequential files also? If yes then where do we define it?
Thanks and Regards,
-Sumit
Commit in Sequential file
Moderators: chulett, rschirm, roy
-
- Participant
- Posts: 197
- Joined: Mon Feb 17, 2003 11:20 pm
- Location: India
No, afraid there's no such thing as a 'transaction size' or a commit for sequential files from DataStage that you could be seeing the result of.
What you may be seeing is a product of your disk subsystem, especially with 'enterprise storage' like EMC and the like. Depending on the settings and the amount of cache involved (which can be substantial), you may be right in thinking that it is not being flushed to disk until a certain number of records are involved... or a certain buffer size is reached or the system simply needs the space for something esle. When that happens, your cached / dirty information is flushed to disk and starts to show up when you 'View Data'. That's my take on it, anyways.
Person to talk to would be your SA, whomever is in charge of the physical disk. They should be able to shed more light on the issue.
What you may be seeing is a product of your disk subsystem, especially with 'enterprise storage' like EMC and the like. Depending on the settings and the amount of cache involved (which can be substantial), you may be right in thinking that it is not being flushed to disk until a certain number of records are involved... or a certain buffer size is reached or the system simply needs the space for something esle. When that happens, your cached / dirty information is flushed to disk and starts to show up when you 'View Data'. That's my take on it, anyways.
Person to talk to would be your SA, whomever is in charge of the physical disk. They should be able to shed more light on the issue.
-craig
"You can never have too many knives" -- Logan Nine Fingers
"You can never have too many knives" -- Logan Nine Fingers
There is no hope, this is an OS issue with buffered writes. Remember your C days and flushing? Your attempting to view data can have errant results, because data is writing in blocks, not "rows". Rows are a database concept, files are blocks of bytes. Odds are that the last row will be partial because it falls into the block issue. You'll just have to live with it.
Kenneth Bland
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
There are a lot of clever tricks built in to the Sequential File stage, including read-ahead and write-ahead buffers.
This means that there will be some time before you will see any rows actually appear. It seems to be 1000 rows before any rows are sync'ed.
These buffers can not be accessed or configured.
This means that there will be some time before you will see any rows actually appear. It seems to be 1000 rows before any rows are sync'ed.
These buffers can not be accessed or configured.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
-
- Participant
- Posts: 3337
- Joined: Mon Jan 17, 2005 4:49 am
- Location: United Kingdom
Hi,
(A shot in the dark)
on top of all this check your file's modified time, if it is not consistant with the job's log time of write to it it might have been truncated since, hence empty when you look at it.
please, post your solution eithere way,
(A shot in the dark)
on top of all this check your file's modified time, if it is not consistant with the job's log time of write to it it might have been truncated since, hence empty when you look at it.
please, post your solution eithere way,
Roy R.
Time is money but when you don't have money time is all you can afford.
Search before posting:)
Join the DataStagers team effort at:
http://www.worldcommunitygrid.org
Time is money but when you don't have money time is all you can afford.
Search before posting:)
Join the DataStagers team effort at:
http://www.worldcommunitygrid.org
-
- Participant
- Posts: 197
- Joined: Mon Feb 17, 2003 11:20 pm
- Location: India