Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.
Moderators: chulett , rschirm , roy
deepa.y
Participant
Posts: 56 Joined: Mon Nov 28, 2011 10:47 pm
Location: Bangalore
Post
by deepa.y » Wed Feb 08, 2012 4:21 am
Hi,
I have Tar files which i need to Untar and read them without landing them in any directory.How can i read those files using parallel job?
pandeesh
Premium Member
Posts: 1399 Joined: Sun Oct 24, 2010 5:15 am
Location: CHENNAI, TAMIL NADU
Post
by pandeesh » Wed Feb 08, 2012 4:30 am
Why you want to accomplish in a px job rather than unix command?
Check Compress and Expand stages, whether they satiate your requirement.
pandeeswaran
deepa.y
Participant
Posts: 56 Joined: Mon Nov 28, 2011 10:47 pm
Location: Bangalore
Post
by deepa.y » Wed Feb 08, 2012 4:34 am
I checked Expand stage but it works only with Datasets.
kandyshandy
Participant
Posts: 597 Joined: Fri Apr 29, 2005 6:19 am
Location: Singapore
Post
by kandyshandy » Wed Feb 08, 2012 4:39 am
Untar files in DS?
I guess you need to rely on some UNIX commands to achieve this.
I am not sure whether there is a new feature in 8.7!!
Kandy
_________________
Try and Try again…You will succeed atlast!!
deepa.y
Participant
Posts: 56 Joined: Mon Nov 28, 2011 10:47 pm
Location: Bangalore
Post
by deepa.y » Wed Feb 08, 2012 4:41 am
I came across some posts which had shown that it could be done using Named Pipe concept in server jobs.Is there any way to do that in parallel job?
ray.wurlod
Participant
Posts: 54607 Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:
Post
by ray.wurlod » Wed Feb 08, 2012 6:33 am
What if you were to use tar as the filter command in a Sequential File stage?
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
deepa.y
Participant
Posts: 56 Joined: Mon Nov 28, 2011 10:47 pm
Location: Bangalore
Post
by deepa.y » Wed Feb 08, 2012 7:13 am
Hi Ray,
Each tar file contains multiple folders and each folder inturn contain few files.Filter command is not working in this case as it cannot be read like file.
ray.wurlod
Participant
Posts: 54607 Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:
Post
by ray.wurlod » Wed Feb 08, 2012 6:33 pm
You neglected to mention that little fact.
I don't believe there will be a solution that does not involve saving the content of the archive to disk.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
qt_ky
Premium Member
Posts: 2895 Joined: Wed Aug 03, 2011 6:16 am
Location: USA
Post
by qt_ky » Fri Feb 10, 2012 2:15 pm
tar is a unix command and there is not a stage you can use out of the box that will read or write tar files without actually calling the tar command.
Could you share the links you found about using tar with named pipes?
Choose a job you love, and you will never have to work a day in your life. - Confucius
anriliu
Participant
Posts: 2 Joined: Thu Feb 16, 2012 3:02 am
Location: China
Post
by anriliu » Thu Feb 16, 2012 8:47 pm
Why not considering embedded perl/python script in prerun step?
qt_ky
Premium Member
Posts: 2895 Joined: Wed Aug 03, 2011 6:16 am
Location: USA
Post
by qt_ky » Thu Feb 16, 2012 9:51 pm
Why not land the data? Is it to help performance or a lack of disk space?
Have you asked your UNIX admin about the named pipe method?
Choose a job you love, and you will never have to work a day in your life. - Confucius
deepa.y
Participant
Posts: 56 Joined: Mon Nov 28, 2011 10:47 pm
Location: Bangalore
Post
by deepa.y » Fri Feb 17, 2012 12:32 am
Hi,
As we have huge files,we cannot store it on disk.The issue is resolved.We are using External source stage to read the tar file to standard output and using them further .
tar -xOvf path of file.
qt_ky
Premium Member
Posts: 2895 Joined: Wed Aug 03, 2011 6:16 am
Location: USA
Post
by qt_ky » Fri Feb 17, 2012 7:08 am
It costs less to store huge files on disk than in memory. I'm just saying, that's what disk is for...
I don't see any tar -O option on my flavor of UNIX... What does -O do?
Choose a job you love, and you will never have to work a day in your life. - Confucius
deepa.y
Participant
Posts: 56 Joined: Mon Nov 28, 2011 10:47 pm
Location: Bangalore
Post
by deepa.y » Fri Feb 17, 2012 7:18 am
qt_ky wrote: I don't see any tar -O option on my flavor of UNIX... What does -O do?
-O is for standard output