DSXchange

We have contacted IBM and we were advised to set the environment variable $APT_IMPORT_REJECT_STRING_FIELD_OVERRUNS to TRUE. This causes records with character strings not fitting the schema definition to be sent to the reject link.

Also is there any documentation that would explain well what can be defined in the schema file, specifically record header and field options.
I search through online documentation, Red Books and documentation that comes with Datastage, but could not find any.

My load job is a generic job that uses Column Import stage, a schema file (passed as a parameter) and RCP to load the data from a source file into the target table. The test schema file contains the following definition: record { final_delim=end, delim='|', quote=none, null_field='' } ( abc:nullable...

The Head stage also has a limit per partition - Number of Rows (Per Partition)

My jobs may reject records in various stages, but I would like to log only a specific number (project wide parameter X) of rejected records and disregard the rest. I have a shared container that I send all the reject to. The container logs the rejects into the database tables. I'm only interested in...

I'm trying to solve the problem described in http://www.dsxchange.com/viewtopic.php?t=142966 The number of reject records should act as a filter for "good" records. If there are no rejects the job finished successfullyt and inserts/updates data in the target table, but if there are some re...

I was wondering whether there is a parallel stage in Datastage that acts as a "tap"? It's like it to take two inputs, one is my nomal data stream, and another uses one value from abother input that acts as a switch to turn on and off data flowing from the first input. I am using join for t...

Hmm, according to this thread my job may be correctly commiting the records into the target database. I was assuming (probably incorrectly) that both - writing to the target and the lookup stage after the aggregator, that causes the job to fail - belong to one transaction. But it might not be the ca...

Or send it to a Copy stage.

I too wonder what is the rationale. Can you tell us a bit more?

I have tried that, but it still somehow manages to insert some rows (9 out of 23) into the database, before it rejects. Actually it seems to be the common problem. I found many posts on this forum where the target commit is only required when a job finished successfully but the records are still in...

Thans for your reply. Splitting the job is definitely good practise and very much advised in the Redbook "Infosphere DataStage Parallel Framework Standard Practises". I agree, however this should rather be done rather when the job is quite large or for reusability reasons. The redbook ment...

My job is performing a lookup (logically, it can be done via lookup, job, or merge stage, no constraints here). If the lookup fails it is considered a fatal error and the batch, that this job is part of, should stop. Howevere, at the same time I would like to log the offending records (i.e. for whic...

I ended up with writing wrapper stored procedures for all the existing SPS that take or return DATE parameter. They take two VARCHAR2 parameters, date string and date format, and convert they into DATE and call the existing SPs with this date. It works fine now.

I created 3 different jobs to check the performance.

May I know the data volumes you did this testing with?