DSXchange

Hi,

We have two configurations file:
1) with one node (default.apt)
2) with three node (threenode.apt)

I have designed the simple job, which take input from the Sequential file; filter the data and writes to FileSet. This job has Job Parameter as APT_CONFIG_FILE.

Under Fileset we have set Overwrite to "File set update policy" property.

Now,
1st Run,
The Job parameter APT_CONFIG_FILE is assigned as default.apt.
The job ran successfully creating one node fileset

2nd Run,
The Job parameter APT_CONFIG_FILE is assigned as threenode.apt.
The job ran successfully creating threenode fileset

3rd Run,
The Job parameter APT_CONFIG_FILE is assigned as default.apt.
The job aborted with the below error:

Error:
1) Invalid fileset /export/home/dstrain/VishalP_FS: problem with partitioning.
2) Error when checking operator: Node name "node2" not in config file
3) Error when checking operator: Node name "node3" not in config file

Is it something like, once the file is created, number of nodes can be increased and can't be decreased?
If so, we have option Overwrite selected and this does not make sense.

Please advise.

you need to add all other nodes in threenode.apt to default.apt as readonly nodes.

keshav0307 - could you post an example of how to do that please?

At Version 8 this works. I assume that when the DataSet is being deleted at V7.x it is taking the currently active configuration file instead of using the one stored within the DataSet for the delete operation.

Hi,

I suppose my question still remain the same:

Error:
1) Invalid fileset /export/home/dstrain/VishalP_FS: problem with partitioning.
2) Error when checking operator: Node name "node2" not in config file
3) Error when checking operator: Node name "node3" not in config file

Question:
Is it something like, after the fileset ot dataset is created, number of nodes can be increased and can't be decreased to the descriptor file?
If so, we have option Overwrite selected and this does not make sense.

Please confirm the above understanding and please support it with explanation.

Regards,
Vishal P

for processing stages it can be increased or decreased.
but to read the dataset, all the nodes using which the dataset was created must be present in the configuration file(may be as read only pool)

The default.apt can be set to something like(assuming two partition on each node):


{
node "Conductor"
{
fastname "server1"
pools "conductor"
resource disk "/pr1/data/dstage/ds01" {pools ""}
resource disk "/pr1/data/dstage/ds02" {pools ""}
resource disk "/pr1/data/dstage/ds03" {pools ""}
resource disk "/pr1/data/dstage/ds04" {pools ""}
resource disk "/pr1/data/dstage/ds05" {pools ""}
resource scratchdisk "/var/Scratch" {pools ""}
}
node "node1_1"
{
fastname "server2"
pools ""
resource disk "/pr1/data/dstage/ds01" {pools ""}
resource disk "/pr1/data/dstage/ds02" {pools ""}
resource disk "/pr1/data/dstage/ds03" {pools ""}
resource disk "/pr1/data/dstage/ds04" {pools ""}
resource disk "/pr1/data/dstage/ds05" {pools ""}
resource scratchdisk "/var/Scratch" {pools ""}
}
node "node1_2"
{
fastname "server2"
pools ""
resource disk "/pr1/data/dstage/ds02" {pools ""}
resource disk "/pr1/data/dstage/ds03" {pools ""}
resource disk "/pr1/data/dstage/ds04" {pools ""}
resource disk "/pr1/data/dstage/ds05" {pools ""}
resource disk "/pr1/data/dstage/ds01" {pools ""}
resource scratchdisk "/var/Scratch" {pools ""}
}
node "node2_1"
{
fastname "server3"
pools "read_only"
resource disk "/pr1/data/dstage/ds01" {pools ""}
resource disk "/pr1/data/dstage/ds02" {pools ""}
resource disk "/pr1/data/dstage/ds03" {pools ""}
resource disk "/pr1/data/dstage/ds04" {pools ""}
resource disk "/pr1/data/dstage/ds05" {pools ""}
resource scratchdisk "/var/Scratch" {pools ""}
}
node "node2_2"
{
fastname "server3"
pools "read_only"
resource disk "/pr1/data/dstage/ds02" {pools ""}
resource disk "/pr1/data/dstage/ds03" {pools ""}
resource disk "/pr1/data/dstage/ds04" {pools ""}
resource disk "/pr1/data/dstage/ds05" {pools ""}
resource disk "/pr1/data/dstage/ds01" {pools ""}
resource scratchdisk "/var/Scratch" {pools ""}
}
node "node3_1"
{
fastname "server4"
pools "read_only"
resource disk "/pr1/data/dstage/ds01" {pools ""}
resource disk "/pr1/data/dstage/ds02" {pools ""}
resource disk "/pr1/data/dstage/ds03" {pools ""}
resource disk "/pr1/data/dstage/ds04" {pools ""}
resource disk "/pr1/data/dstage/ds05" {pools ""}
resource scratchdisk "/var/Scratch" {pools ""}
}
node "node3_2"
{
fastname "server4"
pools "read_only"
resource disk "/pr1/data/dstage/ds02" {pools ""}
resource disk "/pr1/data/dstage/ds03" {pools ""}
resource disk "/pr1/data/dstage/ds04" {pools ""}
resource disk "/pr1/data/dstage/ds05" {pools ""}
resource disk "/pr1/data/dstage/ds01" {pools ""}
resource scratchdisk "/var/Scratch" {pools ""}
}
}

Hi All,

Sorry to say but basically your repliles didn't help. I know the solution.

What I want to know basically is the reason/why the given error in thrown though the property is set to "Overwrite".

Error:
1) Invalid fileset /export/home/dstrain/VishalP_FS: problem with partitioning.
2) Error when checking operator: Node name "node2" not in config file
3) Error when checking operator: Node name "node3" not in config file

Please help with reason of abort.

At Version 8 the problem no longer exists. At Version 7 you can use the command "orchadmin rm {path to dataset}" as a UNIX shell command in the before-job routine call for your job. That will remove the file (using the configuration stored in the file itself) and will use the current APT_CONFIG settings to create the file.

Vishal, you need to read a little more carefully...

... to read the dataset, all the nodes using which the dataset was created must be present in the configuration file...

Ok, so the grammar is a little off, but basically I parsed this as:

To read the dataset, all of the nodes which were used to create the dataset must be present in the configuration file [used at runtime].


[/quote]