Fix scheduled-swap crash

It turns out that, with the right timing, onItemSelected() can be
called while the activity is paused.