Lookup stage Windows vs Linux

pavankvk · Post by **pavankvk** » Wed Sep 09, 2015 3:04 pm

Hi,

We have many jobs that use lookup stages and we are currently porting them from Linux(ds 8.1) to Windows(11.3.1).

We ran into the classic lookup problem

Could not map table file, not enough space.

While these jobs are running fine with much larger volumes in Production environment(8.1, Linux), they are failing instantly in 11.3(Windows 2012) in Dev with much lower volumes. Both OS are 64bit.

I came across this in IBM doc

Each physical process can address only up to 2 GB of memory because it is a 32-bit application. The Windows version of the InfoSphere DataStage Parallel Engine is available only with 32-bit pointers.

Is Datastage 32 bit irrespective of the platform or its 32 bit only on windows platform? Is that the reason, these jobs are running fine against Linux,but failing in Windows.

are there any settings(OS level and datastage level) that can be done to avoid a job change. we are talking about few hundred complex jobs and we can't start modifying them to replace with a join.

chulett · Post by **chulett** » Wed Sep 09, 2015 3:28 pm

DataStage comes in 32bit and 64bit flavors depending on platform, so it's not "Windows v. Linux" but rather "32bit v. 64bit" as you noted. And I'm kind of surprised there isn't a 64bit Windows version but support shows "64 - Tolerate" meaning still 32bit for part of all of the product.

I'm not aware of any 'settings' that can solve this problem for you, you'll need to redesign the jobs with the 32bit limitation in mind. Do you really need to load more than 2GB into the lookup? Wondering if it can be cut back to 'just what you need' rather than 'everything you could possibly need'. Making some assumptions here, obviously.

PaulVL · Post by **PaulVL** » Wed Sep 09, 2015 4:09 pm

also, what is the size of the mount for your TMPDIR defined at your project level?

pavankvk · Post by **pavankvk** » Wed Sep 09, 2015 4:12 pm

chulett wrote:DataStage comes in 32bit and 64bit flavors depending on platform, so it's not "Windows v. Linux" but rather "32bit v. 64bit" as you noted. And I'm kind of surprised there isn't a 64bit Windows version ...

That is what i thought, it should come in 32 and 64 bit. infact the download we have from IBM is IS_11.3.1_WIN_64_ML. but the doc surprised me when it said " Each physical process can address only up to 2 GB of memory because it is a 32-bit application"

this document is not tied to windows, though the next line says windows. it is a general doc for all versions of OS.

pavankvk · Post by **pavankvk** » Wed Sep 09, 2015 4:35 pm

PaulVL wrote:also, what is the size of the mount for your TMPDIR defined at your project level?

its about 500gb, pointed to same location as other datasets.

Mike · Post by **Mike** » Wed Sep 09, 2015 4:54 pm

How much physical memory did you have for Linux vs. what you have now on Windows?

And is the concurrent workload about the same on the 2 servers?

The same article that you found indicates that the memory allocated to the lookup needs to be contiguous. Perhaps Windows is unable to deliver as much contiguous memory as linux was able to provide.

You can try throwing more memory at the issue on your Windows server.

Mike

pavankvk · Post by **pavankvk** » Wed Sep 09, 2015 6:12 pm

on linux, this is our ulimit setting.Does windows even have an option to change similar settings? i see SFU has been discontunued. ours is a windows 2012 server, i dont see services for unix in the registryu

ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 270336
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 11263
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 270336
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

chulett · Post by **chulett** » Wed Sep 09, 2015 8:21 pm

It's 32bit software that "tolerates" 64bit Windows, hence your issue. Bit of a downgrade for you, I'm afraid. There's no setting that can change that fact.

pavankvk · Post by **pavankvk** » Wed Sep 09, 2015 8:31 pm

You mean, going to windows from unix is a bad choice in first place?

chulett · Post by **chulett** » Wed Sep 09, 2015 9:53 pm

It's called an opinion and I has one... I personally don't think it was a good decision. [shrug] Now you get to deal with the limits of 32bit memory addressing, unfortunately.

qt_ky · Post by **qt_ky** » Thu Sep 10, 2015 8:15 am

Every place I have seen has either started on Linux or UNIX and stayed there, or they started on Windows then moved to Linux or UNIX.

In any case, have you tried toggling the APT_LUTCREATE_NO_MMAP setting?

chulett · Post by **chulett** » Thu Sep 10, 2015 9:54 am

A little due diligence to confirm the availability of 64bit software in that environment would have helped reverse that decision, I would think. And I haven't been anywhere that went that direction, as noted it is usually the other way - something that actually happened here, moved their environment from Windows to Solaris when they finally got serious with the project.

Interesting... APT_LUTCREATE_NO_MMAP says it is more about performance but will be curious if it helps get around your issue.

pavankvk · Post by **pavankvk** » Thu Sep 10, 2015 10:25 am

We have the 64 bit version of the Infopshere suite. the download itself has 64 in the name as i indicated earlier. Not sure if osh/parallel engine is still 32 bit. Not sure how IBM markets it as 64 BIT.

pavankvk · Post by **pavankvk** » Thu Sep 10, 2015 10:32 am

IS_11.3.1_WIN_64_ML is the download we used

pavankvk · Post by **pavankvk** » Thu Sep 10, 2015 10:36 am

qt_ky wrote:Every place I have seen has either started on Linux or UNIX and stayed there, or they started on Windows then moved to Linux or UNIX.

In any case, have you tried toggling the APT_LUTCREATE_NO_MMAP setting?

I agree. just tried this setting. didn't help.

Lookup_0,0: Could not alloc memory for table file E:/IBM/InformationServer/Server/Dev/Datasets/lookuptable.6812.20150910163235815000.zlmgm5d (1228978640 bytes). Ulimit settings may need to be adjusted, or memory may be exhausted

we have 32 gig ram available on the server.