These only really happen when you use exclusive fullscreen and alt-tab out. Or, at least, it does on Windows with SDL2 in DirectX mode.