DSXchange

Are your column names static or variable? IF static then you just need to hard code some values and then (as cdp suggests) pivot the data.

If your column names and the number of columns is variable then you need to get creative and read the metadata for the table first to get the column name values.

You are using version 11 so this is actually a lot easier than some earlier versions. Join your Full String to every word in your list (ie add a column which contains the offending word). You can then just sort and group your data into a transformer, set a stage variable to hold the updated output v...

Probably stating the obvious here, but 16114730 milliseconds does not equate to 16:11:4730 (as hh:mm:nnnn)

I would be going back to whomever provided you the spec and get clarification.

You could also try using a combination of a peek stage and a lookup then outputting the values to an external target stage. Eg: Set a peek to return all records, all columns, all partitions, and the output mode to Output (Rather than log), Show column names should be true. This will output the data ...

What is your ultimate target schema? It might be that you do not need to use a schema file, but rather you could just split them data into the relevant target columns, using a simple collection of loops and stage variables.

What are you using to open you csv file? I don't think the issue with with you writing your output, but rather with the program reading the file. If you want to combine the data over multiple lines into a single column, then you need to find a way to do that prior to your output. If you require thos...

You would either have to write a custom routine or call the jobs from the command line.

It depends entirely on what sort of job monitoring you want to do.

Not sure about most efficient, but I can only think of 2 ways: 1. Using some sort of regular expression at the unix command line (eg sed) 2. If using DS, the some combination of stage and loop variables that would determine the number of commas (DCount), hold the current, previous and/or next delimi...

You could have a file which is the same structure as your input file that has your default message in it. You could then use the same input stage with multiple file names. Make sure you sort the data ensuring that your default data comes last. You can then filter the line out if the number of input ...

This question is almost impossible to answer without knowing a lot more about your jobs and your environment set up.

But a starting place would be to ensure that your job designs are avoiding unnecessary re-partitioning and sorting of data.

Looks like its time to debug.

Set your $OSH_PRINT_SCHEMA to print the schemas when the job executes. You can then have a look at what the job is interpreting as your metadata.

You could also add a peek stage at some points in your job to see where the data is changing/outputting incorrectly.

How much RCP are you using? Are your columns surfaced anywhere? Or are the columns in question not specified throughout the entire job? Do you land the data to any temporary files? Are you using schemas to apply any metadata? If they are not specified anywhere, then it is likely that any "unexp...

HINT:
DaysSinceFromDate AND DateFromDaysSince

So if I follow you correctly - the job is not finishing - but rather hanging?

What is your stage after the lookup?

The obvious answer is that your key values do not match on the inputs. Have you proved that the input values are matching?

It could also be that your partitioning is incorrect so the values cannot find each other within the given partitions.