Split a record into multiple basing on a condition

[bmnaidu]

I have a raw data file, which contains records. I need to process that file and export to DB.

The process involves adding a corresponding for each record and for this we have to compare a field (say F1) value. If the value of F1 is 1 insert one more record in DB with almost same values as that first record with some values varies and even these values are a part the record in the raw data. So for a record containing F1 value as 1 contains 2 corresponding records (assume A and B) in target DB with minor changes.

Also there are another fields F2 and F3. If F2 is not empty and F3 = 1 then add one record similar to A and another similar to B with only only difference in one field each.

How to solve this.

[kumar_s]

If it is fixed to field F1, F2 and F3 then you can use transformer to acheive this.
Add two links for each field check. Add constraints to each link.
For first link check If F1=1, Then do the required transformation in the derivation part. In the second link check for same F1=1 and do the other set of transformation in the derivation part.
Similarly proceed for F2 and F3.
Atlast FUNNEL it up.

[DSguru2B]

Just to clarify, by funnel kumar means to collect it into one file that will be loaded into the DB.
Didnt want the OP to get confused

[ray.wurlod]

There may be no need to bring the separate streams together - database servers can support multiple connections. If they are all inserts, for example, with different keys, it doesn't really matter how many "clients" are doing the inserts - up to a certain limit, of course.

This is one way to effect partition parallelism within a server job.

Code: Select all

                       +-------->  Transformer  --->  Database
                       |
SeqFile --->  Transformer  ----->  Transformer  --->  Database
                       |
                       +-------->  Transformer  --->  Database