DSXchange

We write a lot of SQL statements to do data profiling. The results are compiled into an Excel sheet into tabs for each profiling rule. Now if the query returns a lot of data the analyst has to wait a long time for the data to show up and then copy it all into the excel sheet. I am asked to improve t...

Hi,
Use the Sort option in look up stage for the reference data and check the box for unique sort. So the duplicates will be removed. This will work fine for small number of reference records, but reduces the performance as the number increases.

Use a Sort Stage for the reference data, sort on the key fields and change "Allow duplicates" property to "False" or use a Remove Duplicates stage.

Hope this helps,

Try the following code in a transformer stage. In place of X give your coloumn name.
StringToDate(X[1,4] :"-" : X[5,2] : "-" : X[7,2], "%yyyy-%mm-%dd").
Also check the date format in the input to see if it is "YYYYMMDD". change the code accordingly.

Import the meta data again. Open the sequential file and see how the columns are seperated, the end delimiter and see if the data is in double quotes. see if all these properties are given correctly in the properties tab in sequential file stage.