DSXchange

Hi all
we have been using datastage for a few month now, we are still looking for a proper development process (check-in, check-out process, deployment process, version control process... etc, etc).

currently I have created different sandboxes for each developer, they can import the job into the sandbox and update the version number/comment in general tab them import the job back to shared project.

I just want to know what's the development processes you guys use at work, hopefully I can learn some from you guys

thanks
abyss

Hi abyss,

We've created a custom menu item in the DataStage Designer client which invokes a checkin-process.

The process is execute by Atlassian Bamboo (a build automation tool similar to Jenkins or Hudson) which extracts the currently open DataStage job from XMETA, invokes a set of unit tests, then checks the job into Stash (Atlassian's wrapped version of Git). The unit tests involve loading the DataStage job's OSH into a graph database and testing for anti-patterns, such as adjacent transformers, or anything else we want to identify. Some lead to warnings, some lead to the check-in being rejected. Bamboo can additionally update the job's long description with version/deployment information.

Jobs are then automatically deployed to an integration test environment (again using Bamboo) where they are tested against restricted data volumes with functional test cases. When all tests are passed the job becomes a candidate for promotion to production, or other test environments as necessary.

This process means all developers work in the same project, and we have good control over, and transparency of, our code releases.

HTH,
J.

jhmckeever - would you mind doing a write up on the code validation steps that you have built ?

From a unit test perspective, are you testing unit tests against data or did you mean the code validation steps were your unit tests.


We are currently investigating how we can build jenkins pipelines, deploy continuous integration databases, load sample data, run automated tests, and if all successful, release the code to QA environment.

My current location puts assigned developers into a common dev project. When they're satisfied with unit testing, the lead exports objects into a dsx file, and we promote that to a separate test project. When user signs off, we promote to production. We don't bother with all the fancy, "proper" versioning numbers, check in, and check out processes. It's been working quite well that way for 10+ years. I have also seen much more formal, tedious, time-consuming processes in previous jobs. I guess it depends on the work culture (and possibly on which industry you're in) and how uptight or relaxed the organization is and how much effort they can afford throw into formalities...

My previous project used the check-in/check-out capabilities exposed in Information Server Manager to interact with SubVersion. Developers had to be trained to use this mechanism universally, and there was some management overhead monitoring compliance.

patpicos wrote:jhmckeever - would you mind doing a write up on the code validation steps that you have built? From a unit test perspective, are you testing unit tests against data or did you mean the code validation steps were your unit tests.

I'll send you a private message and we can discuss it.

Those following this thread might be interested to know that my organisation's solution for implementing DevOps approaches with DataStage development (called 'MettleCI') has now been announced on the Atlassian Marketplace:

https://marketplace.atlassian.com/plugi ... .mettleci/

You can find out more here: https://www.mettleci.com