Maximum export size for a DSX

clarcombe · Post by **clarcombe** » Mon Mar 06, 2006 3:37 am

Customer has a 75mb project dsx file which they are trying to import from integration to production.

Normally we get around the import timeouts by using an import selected and importing in 4 steps however now we cannot even get DS to show us the jobs to import!

We have looked at splitting the dsx but this would mean that we would have to incorporate common routines in different DSXs thus meaning doubling the maintenance work each time a common routine was modified

1) Does DS have a maximum size for a DSX ? Is there a recommended maximum size
2) If we leave the executable off, would this reduce the DSX file size by much
3) Any suggestions as to other work arounds ?

Many Thanks

Colin

ArndW · Post by **ArndW** » Mon Mar 06, 2006 3:56 am

I've done some exports of projects in that size and hadn't gotten issues - but it was on fast and mostly idle hardware. By excluding the executables from your DSX files you will save quite a bit of space; the binaries are UUENCODED {remember that old technology from old e-mail systems?} which increases their already large sizes. The DSX and XML formats are, unfortunately for you, anything but compact.

I've found the selective import takes a long time and would have expected timeouts there, since it has to process the whole text file in one pass to get the list of valid objects.

What is keeping you from splitting your DSX file; either manually and re-inserting headers or by exporting the routines/functions and jobs separately?

clarcombe · Post by **clarcombe** » Mon Mar 06, 2006 5:45 am

Thanks for the information

The issue is that a lot of the jobs are interdependant. Whilst we can separate the routines , transforms etc relatively simply, the jobs are more difficult.

I am trying to avoid manual cut and paste of a 75mb file !!

I think I will have to get the developer to really analyse his code and work out where he can break it and then import in order so that the interdependencies are taken into account

ArndW · Post by **ArndW** » Mon Mar 06, 2006 6:41 am

What problems are you getting importing interdependant jobs? If you just import a job sequence which calls a server job that hasn't been imported yet you won't get a problem (at least not until you compile it).

chulett · Post by **chulett** » Mon Mar 06, 2006 7:13 am

I've imported from 200 or 300MB exports without anything timing out. Depending on the speed of your machine and how much RAM you have, it can take quite some time, however. An xml export will take even longer as it is first converted to a dsx 'under the covers' and then imported. Patience is the key here.

If the timeouts are from DataStage, use the Administrator to change your timeout value to 'Do not timeout' and see if that helps. Windows will definitely get to a state where it says it is 'not responding' but that can be ignored.

If you know you want everything in the export, then you really don't want to be doing a 'selective' import as you are at least doubling the amount of time it will take end to end. Also suggest you select the 'Overwrite without query' option unless you actually want it to stop if an object already exists in the repository. Nothing more frustrating than finding that 'Are you sure?' query waiting for you, buried behind other windows when it should have been done hours ago dagnabbit.

ray.wurlod · Post by **ray.wurlod** » Mon Mar 06, 2006 7:29 am

Get more memory on your client machine!

clarcombe · Post by **clarcombe** » Mon Mar 06, 2006 7:41 am

DS version 6
Server : HP 6 cpu PA 8900 16gb memory
Client : Pentium 4 2.8 ghz 512mb memory running XP

Is the 512mb not enough. Are there any ways of working out what size the client memory should be ?

Thanks

chulett · Post by **chulett** » Mon Mar 06, 2006 7:44 am

512MB is barely enough to run XP.

Is there a workstation there with more? I have 2GB but even 1GB would help tremendously.

kcbland · Post by **kcbland** » Mon Mar 06, 2006 8:18 am

Instead of import selected search the forum for "dsx cutter". Steve Boyce posted a perl script (parsedsx) for exploding a .dsx into individual files for each job and routine in a large dsx file. He also posted a concatenator (catdsx) script for combining all files in a directory.

My point is, explode your fat dsx file using parsedsx and then pick the items for import using Windoze Explorer. Move them into another folder, run catdsx on them. Use the resulting file for import. Even better, catdsx lets you specify how many equal sized import files you want, so that you can run multiple imports simultaneously to more quickly setup a project.

clarcombe · Post by **clarcombe** » Mon Mar 06, 2006 8:26 am

I don't see how that would work. Surely if I explode a DSX I have to ensure the import order follows a bottom up order i.e. routines first then jobs that use those routines and then jobs that use other jobs ?

Unless that is, the dsx cutter is intelligent and it knows which routines to associate with which jobs.

ArndW · Post by **ArndW** » Mon Mar 06, 2006 8:30 am

Colin,

as I stated before the order isn't important within the dsx file.

kumar_s · Post by **kumar_s** » Mon Mar 06, 2006 8:30 am

Multiple import simulteneously may agian eat up the memory.
Command line import can be tryed to import all the dsx from a folder after scheduling.

kcbland · Post by **kcbland** » Mon Mar 06, 2006 8:33 am

There's no import order in a dsx file. Routines do not have to precede jobs. Trust me, I've used this product for 8 years.

Steve and I wrote those perl scripts to facilitate interfacing DS with 3rd party version control tools.

Since you're doing an import selected, you're manually choosing which jobs to import. The same thing can be done without waiting for the import selected to load the job/routine list into the selection window (which requires twice parsing the large dsx file, time consuming). By exploding the large dsx file into a file per job and routine, you can just use Explorer to tag the jobs for import and then merge them together.

kcbland · Post by **kcbland** » Mon Mar 06, 2006 8:35 am

kumar_s wrote:Multiple import simulteneously may agian eat up the memory.

The problem is the "import selected". On large dsx files this is a fundamentally BAD METHOD. You basically wait while it "imports" the file into a selection list, then wait again while it rescans the dsx file and imports your chosen few. This is really slow and takes a long time, and also a lot of memory.

clarcombe · Post by **clarcombe** » Mon Mar 06, 2006 9:34 am

Trust me, I've used this product for 8 years. Steve and I wrote those perl scripts to facilitate interfacing DS with 3rd party version control tools.

I would love to believe you are right but when I worked with the developer we did an export selected to 3 separate DSXs and then imported them to a new project and this failed with a whole bunch of constraint errors ( I don't have the exact error codes returned).

Going back to the memory issue, I thought this was a good route until I was told it would be easier to transport the Eiffel Tower to Egypt than to get a memory upgrade on a standard homogenised firewalled locked company PC

At lunchtime, we exported the DSX hierarchically and I am awaiting the results of that import.

Thanks all for your assistance

DSXchange

Maximum export size for a DSX

Maximum export size for a DSX

Specification of client and server