Need Help: Persistent Selection Mask Script

[mindlesspuppet]

I'd like to have hotkeys set up that change the selection masks. The problem I'm having is that when you use the F-keys (e.g. SelectEdgeMask) to select the mask, you lose your current selection. If you select using the right-click marking menu, the selection remains, this is how I would like it to work with the hotkeys.

I found this script on CGTalk forums (I believe).

source "dagMenuProc.mel";
$sels=ls -dag -sl -objectsOnly -shapes;
$selObj=listRelatives -p $sels[0];
string $type = objectType $sels[0];

switch ($type) {
case "nurbsSurface":
doMenuNURBComponentSelection($selObj[0], "controlVertex");break;
case "subdiv":
doMenuComponentSelection($selObj[0], "subdivMeshPoint");break;
case "mesh":
doMenuComponentSelection($selObj[0], "vertex");break;
}

It works, but there is an annoyance I can't seem to work out. I set up a command for each component type, and bound them to keys 2-5. If I select and object and use the script to select verts and then decide to switch to edges (without selecting any verts or going back to object mode) it errors with '// Error: No object matches name: // '. If you make any sort of selection it works just fine switching between components. I assume the problem is that when you're in a selection mode with nothing selected there's no object for the script to work on, but it seems like there should be a way to make it use 'SelectEdgeMask' or such if no object is found that way you would still be able to switch modes without a selection. Though, I really have no clue how to get it to function like this and any help would be greatly appreciated.

[joojaa]

So you dont use the keys 2 3 4 and 5 then? I find them most usefull. But yes any code that relies on:

source "dagMenuProc.mel";

are at minimum a bit fishy. Not that i cant find good use for source but doing a dependency on a code that maya or may not be present and forcing maya to reload it is a BIT insane for a switch you will use more then 20 times a day. Not only that but it don't make your code exactly easier to maintain. Because the code undernath may change at any new version without notice. Anyway the code represents minimum effort, maximum problem.

Fortunately you came to the right place and ask, but first:

QUOTE
The problem I'm having is that when you use the F-keys (e.g. SelectEdgeMask) to select the mask, you lose your current selection

And so you do with the code above too, to set the record straight. When you go into individual components selection mode your selection converts to highlight. This is also whats causing your problem. Since your thinking your code somehow avoids the same pitfall. It doesn't unfortunately. Hence your problem. It does however NOT disallow other component selection true.

so first you need to do:

$sels2=ls -dag -hl -objectsOnly -shapes;

because your in component mode being object isnt selected its highlighetd (append $sels2 to $sels). However let us consider your strategy. Its a bit lacking since it only operates on one type at a time. It would be probably much better for you to do:

CODE
hilite ls -sl;
selectType -ocm -alc false;
selectType -ocm
           -controlVertex 1
           -vertex 1
           -subdivMeshPoint 1;

Not only is this secured, shorter, and teh way you should use it, updates better, it isnt reliant on externalities. You could have realized this by reading dagMenuProc.mel by the way

[rgkovach]

i'm embarassed to post this code, Jooja will probably scold me for being so inefficient and doing things way mroe complicated than they need to be, but here is a script i put together for one of our artists who is used to the way Max works...

it remembers the components selected for each mode, so when you switch from vertex to edge to face, etc, it remembers what you had last selected

CODE
proc string rkComponentMode() {
if ( selectMode -q -o == 1 )
    return "Object";
if ( selectType -q -v == 1 )
    return "Vertex";
if ( selectType -q -eg == 1 )
    return "Edge";    
if ( selectType -q -fc == 1 )
    return "Face";
if ( selectType -q -puv == 1 )
    return "UV";
return "Unknown"; }    

// declare global variables to store each component mode's selection
global string $vertSel[];
global string $edgeSel[];
global string $polySel[];
global string $uvSel[];

global proc rkSetSelMask( int $mode ) {

global string $vertSel[];
global string $edgeSel[];
global string $polySel[];
global string $uvSel[];

string $compMode = rkComponentMode;
switch ( $mode ) {
    case 0:
        if ( $compMode == "Edge" )
            $edgeSel = ls -sl;        
        else if ( $compMode == "Face" )
            $polySel = ls -sl;
        else if ( $compMode == "UV" )
            $uvSel = ls -sl;            
        else
            print "I got here\n";
        selectMode -component;
        selectType -v 1;
        select -r $vertSel;
        break;
    case 1:
        if ( $compMode == "Vertex" )
            $vertSel = ls -sl;        
        else if ( $compMode == "Face" )
            $polySel = ls -sl;
        else if ( $compMode == "UV" )
            $uvSel = ls -sl;            
        else
            print "I got here\n";
        selectMode -component;
        selectType -eg 1;
        select -r $edgeSel;
        break;
    case 2:
        if ( $compMode == "Vertex" )
            $vertSel = ls -sl;        
        else if ( $compMode == "Edge" )
            $edgeSel = ls -sl;
        else if ( $compMode == "UV" )
            $uvSel = ls -sl;            
        else
            print "I got here\n";
        selectMode -component;
        selectType -fc 1;
        select -r $polySel;
        break;
    case 3:
        if ( $compMode == "Vertex" )
            $vertSel = ls -sl;        
        else if ( $compMode == "Face" )
            $polySel = ls -sl;
        else if ( $compMode == "Edge" )
            $edgeSel = ls -sl;            
        else
            print "I got here\n";
        selectMode -component;
        selectType -puv 1;
        select -r $uvSel;
        break; } }

[combustible]

hehe same script I've been working on

The problem I have with selectType is that the default behavior isn't always going to turn off the unwanted masks, so there will be times when you're selecting edges and the vert selectType is still turned on. Have you not experienced this problem?

for taking in the different component selections you may want to use filterExpand
Right now you're taking in the whole list of the selection with ls -sl but with filterExpand you'll restrict it to only the specified component types.

[joojaa]

QUOTE
The problem I have with selectType is that the default behavior isn't always going to turn off the unwanted masks, so there will be times when you're selecting edges and the vert selectType is still turned on. Have you not experienced this problem?

You cant have problem with select type because if you do you cant change selection type.Well i just do what the asker wanted (he wanted the same behavior as rmb button i gave it to him). Anyway ive accounted for that its why i call:

selectType -ocm -alc false;

this clears the selection mask, this way i never have this problem. But alas theres a problem the modes in the status bar always superceeds what you've set (theres 2 modes global and object). It would even apply to the original code. This si very deliberate as thats the behavior he asked for. however were you to change the code to:
CODE
hilite ls -sl;
selectMode -component;
selectType -alc false;
selectType
           -controlVertex 1
           -vertex 1
           -subdivMeshPoint 1

without the ocm. it would change the global mask and keep the selection. Wich s closer to what combustible wants, and garnted mimics the f keys better without clearing mask. Then just unlioad the other selections if you want.

rgkovach: yes its a bit heavy. Actually max does store the selection per context. Anyway i do have a suggestion for you. instead of using

global string $vertSel[];

for selection keeping do keep the selection is a quick selection set instead thisway your less likely to clash later.

it would also simplify your code quite significantly as you could do off withe the if else and case closures. so you'd end up with:
CODE
sets  -clear $mode;  
sets -add $mode ls -sl;

Besides thsi would let you mine selections as accurately as you want even on per object basis.

PS building on this ist posssible to retrieve the selections that caused the hiostory objects to do something and rolling back history JUST like in max too.

PPS: even if the mel codes shipped have often a switch jsut to change one flag in mel you dont need its possible to pass a flag to a swicch withe the alternale form code so you can have:

$flag="-sl";
print (ls($flag));

so if you evere see something like:

CODE
switch( $mode ) {        case "hierarchical":
            selectionMaskRestoreOnce();
            selectMode -hierarchical;
            break;
        case "root":
            selectionMaskRestoreOnce();
            selectMode -root;
            break;
        case "leaf":
            selectionMaskRestoreOnce();
            selectMode -leaf;
            break;
        case "template":
            selectionMaskRestoreOnce();
            selectMode -template;
            break;

        case "object":
            selectionMaskRestoreOnce();
            selectMode -object;
            break;

        case "component":
            selectionMaskRestoreOnce();
            selectMode -component;
            break;

}

youd KNOW its not a mel coder who did it. but a c coder. as they could have as easily written:
CODE
            selectionMaskRestoreOnce();
            selectMode("-"+$mode);