Configuration file issue
Moderators: chulett, rschirm, roy
-
- Participant
- Posts: 10
- Joined: Thu Nov 16, 2006 5:55 am
- Location: Pune
- Contact:
Configuration file issue
Hi,
We have two configurations file:
1) with one node (default.apt)
2) with three node (threenode.apt)
I have designed the simple job, which take input from the Sequential file; filter the data and writes to FileSet. This job has Job Parameter as APT_CONFIG_FILE.
Under Fileset we have set Overwrite to "File set update policy" property.
Now,
1st Run,
The Job parameter APT_CONFIG_FILE is assigned as default.apt.
The job ran successfully creating one node fileset
2nd Run,
The Job parameter APT_CONFIG_FILE is assigned as threenode.apt.
The job ran successfully creating threenode fileset
3rd Run,
The Job parameter APT_CONFIG_FILE is assigned as default.apt.
The job aborted with the below error:
Error:
1) Invalid fileset /export/home/dstrain/VishalP_FS: problem with partitioning.
2) Error when checking operator: Node name "node2" not in config file
3) Error when checking operator: Node name "node3" not in config file
Is it something like, once the file is created, number of nodes can be increased and can't be decreased?
If so, we have option Overwrite selected and this does not make sense.
Please advise.
We have two configurations file:
1) with one node (default.apt)
2) with three node (threenode.apt)
I have designed the simple job, which take input from the Sequential file; filter the data and writes to FileSet. This job has Job Parameter as APT_CONFIG_FILE.
Under Fileset we have set Overwrite to "File set update policy" property.
Now,
1st Run,
The Job parameter APT_CONFIG_FILE is assigned as default.apt.
The job ran successfully creating one node fileset
2nd Run,
The Job parameter APT_CONFIG_FILE is assigned as threenode.apt.
The job ran successfully creating threenode fileset
3rd Run,
The Job parameter APT_CONFIG_FILE is assigned as default.apt.
The job aborted with the below error:
Error:
1) Invalid fileset /export/home/dstrain/VishalP_FS: problem with partitioning.
2) Error when checking operator: Node name "node2" not in config file
3) Error when checking operator: Node name "node3" not in config file
Is it something like, once the file is created, number of nodes can be increased and can't be decreased?
If so, we have option Overwrite selected and this does not make sense.
Please advise.
-
- Premium Member
- Posts: 783
- Joined: Mon Jan 16, 2006 10:17 pm
- Location: Sydney, Australia
-
- Participant
- Posts: 10
- Joined: Thu Nov 16, 2006 5:55 am
- Location: Pune
- Contact:
Hi,
I suppose my question still remain the same:
Error:
1) Invalid fileset /export/home/dstrain/VishalP_FS: problem with partitioning.
2) Error when checking operator: Node name "node2" not in config file
3) Error when checking operator: Node name "node3" not in config file
Question:
Is it something like, after the fileset ot dataset is created, number of nodes can be increased and can't be decreased to the descriptor file?
If so, we have option Overwrite selected and this does not make sense.
Please confirm the above understanding and please support it with explanation.
Regards,
Vishal P
I suppose my question still remain the same:
Error:
1) Invalid fileset /export/home/dstrain/VishalP_FS: problem with partitioning.
2) Error when checking operator: Node name "node2" not in config file
3) Error when checking operator: Node name "node3" not in config file
Question:
Is it something like, after the fileset ot dataset is created, number of nodes can be increased and can't be decreased to the descriptor file?
If so, we have option Overwrite selected and this does not make sense.
Please confirm the above understanding and please support it with explanation.
Regards,
Vishal P
-
- Premium Member
- Posts: 783
- Joined: Mon Jan 16, 2006 10:17 pm
- Location: Sydney, Australia
for processing stages it can be increased or decreased.
but to read the dataset, all the nodes using which the dataset was created must be present in the configuration file(may be as read only pool)
The default.apt can be set to something like(assuming two partition on each node):
{
node "Conductor"
{
fastname "server1"
pools "conductor"
resource disk "/pr1/data/dstage/ds01" {pools ""}
resource disk "/pr1/data/dstage/ds02" {pools ""}
resource disk "/pr1/data/dstage/ds03" {pools ""}
resource disk "/pr1/data/dstage/ds04" {pools ""}
resource disk "/pr1/data/dstage/ds05" {pools ""}
resource scratchdisk "/var/Scratch" {pools ""}
}
node "node1_1"
{
fastname "server2"
pools ""
resource disk "/pr1/data/dstage/ds01" {pools ""}
resource disk "/pr1/data/dstage/ds02" {pools ""}
resource disk "/pr1/data/dstage/ds03" {pools ""}
resource disk "/pr1/data/dstage/ds04" {pools ""}
resource disk "/pr1/data/dstage/ds05" {pools ""}
resource scratchdisk "/var/Scratch" {pools ""}
}
node "node1_2"
{
fastname "server2"
pools ""
resource disk "/pr1/data/dstage/ds02" {pools ""}
resource disk "/pr1/data/dstage/ds03" {pools ""}
resource disk "/pr1/data/dstage/ds04" {pools ""}
resource disk "/pr1/data/dstage/ds05" {pools ""}
resource disk "/pr1/data/dstage/ds01" {pools ""}
resource scratchdisk "/var/Scratch" {pools ""}
}
node "node2_1"
{
fastname "server3"
pools "read_only"
resource disk "/pr1/data/dstage/ds01" {pools ""}
resource disk "/pr1/data/dstage/ds02" {pools ""}
resource disk "/pr1/data/dstage/ds03" {pools ""}
resource disk "/pr1/data/dstage/ds04" {pools ""}
resource disk "/pr1/data/dstage/ds05" {pools ""}
resource scratchdisk "/var/Scratch" {pools ""}
}
node "node2_2"
{
fastname "server3"
pools "read_only"
resource disk "/pr1/data/dstage/ds02" {pools ""}
resource disk "/pr1/data/dstage/ds03" {pools ""}
resource disk "/pr1/data/dstage/ds04" {pools ""}
resource disk "/pr1/data/dstage/ds05" {pools ""}
resource disk "/pr1/data/dstage/ds01" {pools ""}
resource scratchdisk "/var/Scratch" {pools ""}
}
node "node3_1"
{
fastname "server4"
pools "read_only"
resource disk "/pr1/data/dstage/ds01" {pools ""}
resource disk "/pr1/data/dstage/ds02" {pools ""}
resource disk "/pr1/data/dstage/ds03" {pools ""}
resource disk "/pr1/data/dstage/ds04" {pools ""}
resource disk "/pr1/data/dstage/ds05" {pools ""}
resource scratchdisk "/var/Scratch" {pools ""}
}
node "node3_2"
{
fastname "server4"
pools "read_only"
resource disk "/pr1/data/dstage/ds02" {pools ""}
resource disk "/pr1/data/dstage/ds03" {pools ""}
resource disk "/pr1/data/dstage/ds04" {pools ""}
resource disk "/pr1/data/dstage/ds05" {pools ""}
resource disk "/pr1/data/dstage/ds01" {pools ""}
resource scratchdisk "/var/Scratch" {pools ""}
}
}
but to read the dataset, all the nodes using which the dataset was created must be present in the configuration file(may be as read only pool)
The default.apt can be set to something like(assuming two partition on each node):
{
node "Conductor"
{
fastname "server1"
pools "conductor"
resource disk "/pr1/data/dstage/ds01" {pools ""}
resource disk "/pr1/data/dstage/ds02" {pools ""}
resource disk "/pr1/data/dstage/ds03" {pools ""}
resource disk "/pr1/data/dstage/ds04" {pools ""}
resource disk "/pr1/data/dstage/ds05" {pools ""}
resource scratchdisk "/var/Scratch" {pools ""}
}
node "node1_1"
{
fastname "server2"
pools ""
resource disk "/pr1/data/dstage/ds01" {pools ""}
resource disk "/pr1/data/dstage/ds02" {pools ""}
resource disk "/pr1/data/dstage/ds03" {pools ""}
resource disk "/pr1/data/dstage/ds04" {pools ""}
resource disk "/pr1/data/dstage/ds05" {pools ""}
resource scratchdisk "/var/Scratch" {pools ""}
}
node "node1_2"
{
fastname "server2"
pools ""
resource disk "/pr1/data/dstage/ds02" {pools ""}
resource disk "/pr1/data/dstage/ds03" {pools ""}
resource disk "/pr1/data/dstage/ds04" {pools ""}
resource disk "/pr1/data/dstage/ds05" {pools ""}
resource disk "/pr1/data/dstage/ds01" {pools ""}
resource scratchdisk "/var/Scratch" {pools ""}
}
node "node2_1"
{
fastname "server3"
pools "read_only"
resource disk "/pr1/data/dstage/ds01" {pools ""}
resource disk "/pr1/data/dstage/ds02" {pools ""}
resource disk "/pr1/data/dstage/ds03" {pools ""}
resource disk "/pr1/data/dstage/ds04" {pools ""}
resource disk "/pr1/data/dstage/ds05" {pools ""}
resource scratchdisk "/var/Scratch" {pools ""}
}
node "node2_2"
{
fastname "server3"
pools "read_only"
resource disk "/pr1/data/dstage/ds02" {pools ""}
resource disk "/pr1/data/dstage/ds03" {pools ""}
resource disk "/pr1/data/dstage/ds04" {pools ""}
resource disk "/pr1/data/dstage/ds05" {pools ""}
resource disk "/pr1/data/dstage/ds01" {pools ""}
resource scratchdisk "/var/Scratch" {pools ""}
}
node "node3_1"
{
fastname "server4"
pools "read_only"
resource disk "/pr1/data/dstage/ds01" {pools ""}
resource disk "/pr1/data/dstage/ds02" {pools ""}
resource disk "/pr1/data/dstage/ds03" {pools ""}
resource disk "/pr1/data/dstage/ds04" {pools ""}
resource disk "/pr1/data/dstage/ds05" {pools ""}
resource scratchdisk "/var/Scratch" {pools ""}
}
node "node3_2"
{
fastname "server4"
pools "read_only"
resource disk "/pr1/data/dstage/ds02" {pools ""}
resource disk "/pr1/data/dstage/ds03" {pools ""}
resource disk "/pr1/data/dstage/ds04" {pools ""}
resource disk "/pr1/data/dstage/ds05" {pools ""}
resource disk "/pr1/data/dstage/ds01" {pools ""}
resource scratchdisk "/var/Scratch" {pools ""}
}
}
-
- Participant
- Posts: 10
- Joined: Thu Nov 16, 2006 5:55 am
- Location: Pune
- Contact:
Hi All,
Sorry to say but basically your repliles didn't help. I know the solution.
What I want to know basically is the reason/why the given error in thrown though the property is set to "Overwrite".
Error:
1) Invalid fileset /export/home/dstrain/VishalP_FS: problem with partitioning.
2) Error when checking operator: Node name "node2" not in config file
3) Error when checking operator: Node name "node3" not in config file
Please help with reason of abort.
Sorry to say but basically your repliles didn't help. I know the solution.
What I want to know basically is the reason/why the given error in thrown though the property is set to "Overwrite".
Error:
1) Invalid fileset /export/home/dstrain/VishalP_FS: problem with partitioning.
2) Error when checking operator: Node name "node2" not in config file
3) Error when checking operator: Node name "node3" not in config file
Please help with reason of abort.
At Version 8 the problem no longer exists. At Version 7 you can use the command "orchadmin rm {path to dataset}" as a UNIX shell command in the before-job routine call for your job. That will remove the file (using the configuration stored in the file itself) and will use the current APT_CONFIG settings to create the file.
-
- Premium Member
- Posts: 8
- Joined: Tue Apr 15, 2008 1:16 pm
- Location: Toronto, Ca
Vishal, you need to read a little more carefully...
To read the dataset, all of the nodes which were used to create the dataset must be present in the configuration file [used at runtime].
[/quote]
Ok, so the grammar is a little off, but basically I parsed this as:... to read the dataset, all the nodes using which the dataset was created must be present in the configuration file...
To read the dataset, all of the nodes which were used to create the dataset must be present in the configuration file [used at runtime].
[/quote]