After only a few years of being an academic I calculated that if I said yes to all the invitations to: review papers and grants, give talks, join committees, etc, then there would not be enough hours in the day to do them all, even if I didn't sleep. I had to start saying no.
Once I'd resolved to do this, I found it much easier than I'd feared. Most of my inviters were also very busy. They understood only too well that I might be too busy myself.
But still, most of us feel a bit guilty to decline. Academia works by a process of reciprocal altruism: I expect others to referee my papers, so I have a duty to referee theirs. To make you feel better about saying no, here is a list of alternative actions, some of which are much worse.
Your inviter will be very annoyed: it may be too late to recruit an alternative invitee. You will feel bad for not delivering - or, at least, you should do. If you do this often, you will get a bad reputation.
Your inviter will be disappointed in the quality, you will feel dissatisfied and your reputation will suffer.
If you are overloaded, most of what you do will inevitably be rushed and low quality.
If you are not competent to do something, then you run a high risk of messing it up. This may have an adverse and unfair effect on others, e.g., rejecting their excellent paper, accepting a rubbish one. Of course, you may be prepared to obtain the necessary qualifications, especially if they are just within your capabilities and you have the time to learn - so this is a fine judgement.
We should all strive to extend ourselves with tasks that challenge and improve us. Some routine tasks are an inevitable part of everyone's job, but don't increase them unnecessarily, especially if you are indulging in 'displacement activities' to avoid the real challenges.
You'll miss a chance to stretch yourself, learn new skills, gain new confidence and improve your reputation.
This list also suggests what excuses you can give an inviter when declining their invitation:
The danger here is that the inviter will call your bluff by offering to extend the deadline. So, only use this excuse if your overload problem is short term and you are genuinely willing and able to do the task given sufficient time - or, if you are sure that the deadline is immovable.
The danger in this one is that the inviter may try to persuade you that you are not only competent, but the ideal person for the job. So, only use this excuse if you are both confident in your incompetence and unwilling to make the investment to become competent, e.g., because you have better ways to spend your life.
This is probably the safest option, unless the inviter is willing to extend the deadline beyond the foreseeable future.
To sweeten the pill, when declining an invitation try to suggest an alternative invitee. Suggest one or more people who are not only more competent than yourself, but who would benefit from accepting the invitation. For instance, it might extend their expertise, enable them to exercise new kinds of responsibility or be a valued addition to their CV. If, in addition, you can offer to mentor them in the task, this would make the pill even sweeter.
Let's finish by switching roles: how to get someone to say yes.