Another way
At the top of the layer dialog is an 'X'
But, this only works if the layer is empty.
If the layer is not empty, select the layer then click on the little arrow at the top of the layer dialog. This will select all objects on that layer. Hit your delete key on the keyboard. The layer is now empty and you can now click on the 'X'