Being decisive is a quality many people desire. It’s fine to say no. We might be letting the other person down, but we must make sure that our decisions are in our best interests too. We have all paid steep prices for saying yes.
Hesitation simply means that we are not fully committed and additional inquiry is necessary.
There will be instances when a quick choice is called for; this is where we learn to trust our intuition. It takes practice, trusting that what we decide is for the greater good. A higher context is always a good reference point.