To remove an object from the map, click the button next to the object's icon, or right-click on the object's icon and in the context menu select Remove from map.

You can remove a map switch only by clicking the button.