Image data migration and transformation

parag.s.27 · Post by **parag.s.27** » Tue Mar 21, 2006 8:17 am

Hi,

I am having a source file in which i am getting images. Can i migrate the image data from source to target.

i need to transform it to reduce its size.

so i wanted to know. is it possible,

if yes then what data type shall i use for it in the meta data, because i never found any oracle like BLOB data type. and how to reduce its size in transformer

kumar_s · Post by **kumar_s** » Tue Mar 21, 2006 8:22 am

But Binary/Varbinary/LongVarBinary could be used.

parag.s.27 · Post by **parag.s.27** » Tue Mar 21, 2006 8:27 am

kumar_s wrote:But Binary/Varbinary/LongVarBinary could be used.

But i don't know whether Transformer stage can simply read image data and transform it or not.

So if i use these Binary/Varbinary/LongVarBinary data types then can i transform the image to reduce its size to a certain limit. and what impact wii there be on the image.

the image data is actually a digital signature images of customers of the company.

kumar_s · Post by **kumar_s** » Tue Mar 21, 2006 8:30 am

Any stages used in datastage doesnt reduces any used in it.
But you can use compress stage initially to compress the files if required.

parag.s.27 · Post by **parag.s.27** » Tue Mar 21, 2006 8:35 am

kumar_s wrote:Any stages used in datastage doesnt reduces any used in it.
But you can use compress stage initially to compress the files if required.

Thanks for quick help Kumar,

But i am not able to find compress stage in my version. the version is 7.5.1.A, can you please help me again.

thanks

kcbland · Post by **kcbland** » Tue Mar 21, 2006 8:36 am

Don't pass binary image, audio, or video data thru any ETL tool. Moving large amounts of binary data on each row thru each active stage is a staggering misuse of memory. Moving text and numbers is no big deal, the tools are written to do that. Rows are measured in bytes, or maybe just a few kilobytes, but when you bring megabytes into play the performance sucks.

One of your next posts will be: My jobs run at <1 row/s, how can I make them faster?

You will need to NOT move BLOBS using DS if they're over some threshold, maybe 10K. Consider using the database tools to extract the primary key and the BLOB column to files named after the primary key row. Use the target database tools to load the data. You'll find that the database makers are better at that than ETL tools.

parag.s.27 · Post by **parag.s.27** » Tue Mar 21, 2006 8:39 am

kcbland wrote:Don't pass binary image, audio, or video data thru any ETL tool. Moving large amounts of binary data on each row thru each active stage is a staggering misuse of memory. Moving text and numbers is n ...

Hi,

Actually the images are not that big and secondly the data is of Customer Masters so not having even 1000 records for this particular kind of data.
and since its client's requirement that it should be atleast tried before dropping from the agenda.

thanks

ArndW · Post by **ArndW** » Tue Mar 21, 2006 8:40 am

I am fairly certain that the transform stage will not be able to transform the image data. How can it? You haven't specified what sort of modifications you want to do. DS can read and write binary data without changing it - but why do you want to do this?

kcbland · Post by **kcbland** » Tue Mar 21, 2006 8:42 am

Explain the data, what kind of information you expect to change in it. ETL tools are about text and number manipulation. Binary data manipulation requires you to understand the binary nature of the attribute and how to manipulate it.

parag.s.27 · Post by **parag.s.27** » Tue Mar 21, 2006 8:43 am

ArndW wrote:I am fairly certain that the transform stage will not be able to transform the image data. How can it? You haven't specified what sort of modifications you want to do. DS can read and write binary da ...

Thanks again,

Then can i compress it earlier before feeding it to the transformer stage, as mentioned by Kumar_s, where can i find compresser stage.

parag.s.27 · Post by **parag.s.27** » Tue Mar 21, 2006 8:53 am

kcbland wrote:Explain the data, what kind of information you expect to change in it. ETL tools are about text and number manipulation. Binary data manipulation requires you to understand the binary nature of the attribute and how to manipulate it.

Hi,

The data is actually in some binary code with each of atleast 50 to 78 KBs.

ArndW · Post by **ArndW** » Tue Mar 21, 2006 8:55 am

Ok, but what kind of transformation do you expect DS to be able to do with this image data? If you are only passing it through unchanged then DS will have no problem with that amount of data as long as the databases have the correct data types defined.

parag.s.27 · Post by **parag.s.27** » Tue Mar 21, 2006 8:59 am

ArndW wrote:Ok, but what kind of transformation do you expect DS to be able to do with this image data? If you are only passing it through unchanged then DS will have no problem with that amount of data as long a ...

Hi,

Actually i was looking for some compression kind of stage as mentioned by Kumar_S. If its available then i'll just have to compress it to atleast its 20 to 30 % and then pass it directly through the transformer unchanged.

thanks

ArndW · Post by **ArndW** » Tue Mar 21, 2006 9:03 am

Binary image data is usually already very well compressed. A normal compression algorithm will never achieve 20-30% on image data (assuming a common format such as .bmp, .gif, .jpeg). You might be able to use an external .JPG converter that does lossy compression, but your picture will not be recognizeable if you compress 30% out of it.

parag.s.27 · Post by **parag.s.27** » Tue Mar 21, 2006 9:04 am

ArndW wrote:Binary image data is usually already very well compressed. A normal compression algorithm will never achieve 20-30% on image data (assuming a common format such as .bmp, .gif, .jpeg). You might be abl ...

Hi Arnd,

Sorry but i am not able to see your reply.