DSXchange

Hi,
I have Tar files which i need to Untar and read them without landing them in any directory.How can i read those files using parallel job?

Why you want to accomplish in a px job rather than unix command?
Check Compress and Expand stages, whether they satiate your requirement.

I checked Expand stage but it works only with Datasets.

Untar files in DS? I guess you need to rely on some UNIX commands to achieve this.

I am not sure whether there is a new feature in 8.7!!

I came across some posts which had shown that it could be done using Named Pipe concept in server jobs.Is there any way to do that in parallel job?

What if you were to use tar as the filter command in a Sequential File stage?

Hi Ray,
Each tar file contains multiple folders and each folder inturn contain few files.Filter command is not working in this case as it cannot be read like file.

You neglected to mention that little fact.

I don't believe there will be a solution that does not involve saving the content of the archive to disk.

tar is a unix command and there is not a stage you can use out of the box that will read or write tar files without actually calling the tar command.

Could you share the links you found about using tar with named pipes?

Why not considering embedded perl/python script in prerun step?

Why not land the data? Is it to help performance or a lack of disk space?

Have you asked your UNIX admin about the named pipe method?

Hi,
As we have huge files,we cannot store it on disk.The issue is resolved.We are using External source stage to read the tar file to standard output and using them further .
tar -xOvf path of file.

It costs less to store huge files on disk than in memory. I'm just saying, that's what disk is for...

I don't see any tar -O option on my flavor of UNIX... What does -O do?

qt_ky wrote:I don't see any tar -O option on my flavor of UNIX... What does -O do?

-O is for standard output