Hello All,
There is an existing sequencer which process some N number of files and produce N number of outputs daily. Number of Source files are dynamic. It is passing through loop one by one and producing one file at a time.Need help on the following two concerns
1)This process is consuming a lot of time.This has to be optimized.
2)Whenever this sequence is getting aborted in between,the loop is again starting from the first file till Nth file.The restartability feature has to be included in the loop.
Appreciate your help on this.
Processing Multiple files
Moderators: chulett, rschirm, roy
-
- Participant
- Posts: 47
- Joined: Thu Oct 04, 2007 3:37 am
- Location: chennai
Processing Multiple files
Cheers,
prem
prem
-
- Participant
- Posts: 47
- Joined: Thu Oct 04, 2007 3:37 am
- Location: chennai
-
- Premium Member
- Posts: 536
- Joined: Thu Oct 11, 2007 1:48 am
- Location: Bangalore
If you are on MPP or grid system and source file is fixed width, you can improve performance by setting 'read from multiple node' it will enable file to read from more then one node, otherwise you can increase "no of reader per node" setting, it will increase the instence of read operator but overall file will read on single node and on single cpu.
Thanks
Prasoon
ETL Consultant
LinkedIn :- http://www.linkedin.com/profile/view?id ... ab_pro_top
Blog:- http://dsshar.blogspot.com/
Prasoon
ETL Consultant
LinkedIn :- http://www.linkedin.com/profile/view?id ... ab_pro_top
Blog:- http://dsshar.blogspot.com/
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
-
- Participant
- Posts: 47
- Joined: Thu Oct 04, 2007 3:37 am
- Location: chennai
Ray,
As the number of source files are dynamic,sometimes 1 file and sometimes 60 files how can i decide the additional loop design to process.
Also,the existing process includes fetching data from Teradata table(as reference link).If am using multiple loops,i.e.,same select query will be executed multiple times against the table at the same instance.
Does this lock the table from reading it.
As the number of source files are dynamic,sometimes 1 file and sometimes 60 files how can i decide the additional loop design to process.
Also,the existing process includes fetching data from Teradata table(as reference link).If am using multiple loops,i.e.,same select query will be executed multiple times against the table at the same instance.
Does this lock the table from reading it.
Cheers,
prem
prem
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact: