DSXchange

Hi Nikhil, It was around 10 million records of 80 bytes row size. However it is worth to add that our SAN disk system is pretty much powerfull and well sized. That's why timings you see are achievable on single node config. The other thing is that using more than 1 node with seq file approach won't ...

Hi again, Answering your question - yes, we use Vertica ODBC driver provided by HP / Vertica. I did run mentioned test (Connector vs Seq File + COPY) and indeed second approach seems to be slightly faster: ODBC Connector / 1 node Elapsed time: 1:02 Command used by Vertica: COPY public.test01 ( col1,...

Hi Nikhil, That's interesting, I will give it a try tomorrow. I knew about that technique however I discarded it due to the fact that my ODBC Connectors's INSERT statement is actually executed by Vertica as COPY command (it uses pipe instead of file). This is Vertica ODBC driver feature to upgrade b...

Ok, we have a workaround for the issue. Very "brute force" workaround but it works and satisfies us :lol: It turned out the main evil comes from the Vertica in-built SQLStatistics ODBC method implementation. The main purpose of it is to return some object statistics and it's implemented as...

Hi David, We did similar exercise last year - 7.5 to 8.5 migration (yes - it's not an upgrade as you have to install 8.5 on side of 7.5). No big surprises on the way, just a few minor issues occurred: 1. O/S commands porting from Unix to Linux may be challenging - subtle differences in shell command...

Hi experts, I've been recently playing a lot with Vertica 6.0 connectivity under DataStage. Setting up the connection has its "gotchas" and isn't described well but what bothers me more is the ODBC Connector performance when it's used in "write" mode. It basically takes at minimu...

Hi Juliesimo, I got hit by the same issue today - under Information Server v8.5 with FP1 installated on AIX all the mappings generated via FastTrack are having source-to-target columns mixed. Moreover, it's mixed-up non-deterministally - every time I generate job I got different results :o The solut...

If the number of records to be updated in the target table is less then 5-10% of the total table - just create an index on update keys. If it's higher - create some temporary table (preferably in bulk mode) and then issue MERGE statement (as "after" Stage action). If you can't predict what...

I asked: When playing with new features of Information Server 8.5 FP1 I met interesting new stage available in "Processing" tab - BloomFilter stage. Bloom filter concept is an exciting feature but unfortunately I was unsuccessful in using it - DataStage was firing lots of unclear internal ...

Please keep in mind that when using middle tables you will need CDC licence (database-to-database) while usage of CDC Plugin Stage or file-driven approach that I mentioned might be implemented via CDD license (database-to-datastage). If you posses adequate amount of CDC database licenses then your w...

It's worth to add that InfoSphere Change Data Capture for DataStage can also work without CDC plugin stage - there is an option to generate file-driven DataStage jobs directly from CDC Management Console. In this approach CDC creates files on the DataStage server side which are then picked up by pre...

I have just checked v8.1 FP1 vs v8.5 FP1 behaviour - it works exactly the same under my environments (Windows), so: -Nullable source -> Not-nullable target drops null records (and you see adequate warning message in logs) -Nullable source -> Nullable target drops null record only when you run data t...

I don't have the 8.1 installation to check but under 8.5 with legacy null processing turned on records are dropped only if I do something with data (like running arithmetic operation against input column)...

How do you drop the records in Transformer? Is there specific condition defined in constraints section?

My initial understanding was - you do not do anything, it's Transformer who automatically drops nulls...

Regards
Buzz

One more thing - you can control legacy null handling on the project/job level - see APT_TRANSFORM_COMPILE_OLD_NULL_HANDLING environment variable.