Just make a conversion on the fly by dividing you rgb values by 255 to get a normalized value between 0 and 1 (note that I use 255.0 instead of 255 to obtain a float value instead of an int:
test:
print (171/255);
//result 0
print (171/255.0);
//result 0.670588
// ex. code for rgb color (171, 208, 188)
{
window -tlb 1 -t "Use RGB values";
rowLayout -adj 1;
button -l "rgb(171,208,188)" -bgc (171/255.0) (208/255.0) (188/255.0);
showWindow;
}
