The only way this would not be ok, is if you were grabbing the outlet plug and unplugging it. The J1772 plug is designed for all this, it cuts power as soon as you press your thumb on the button. If you were to unplug from the wall, you'd be putting your hands very close to 1-3kW of flowing juice, which I don't recommend (the car would still be fine, it's just less safe for you).