You don’t have to explain yourself

If you need to say No, stop at No.

You aren’t obligated to explain why, let alone have an excuse that will appease the other party. If someone asks why, say No to that too.