When my husband and I host an event, we send invitations far in advance. One set of friends, however, always waits until the last minute to respond. The last time they did this, with our dinner just three days away, I cancelled their invitations and invited others in their place. They were furious and sent a sarcastic letter. I don't think I'm in the wrong. Do you?
When you say you "cancelled" their invitations, I hope you mean you sent them an actual invitation with an enormous red CANCELLED stamp on it.
Barring exceptional circumstances, dinner guests should respond to invitations within a day or two if at all possible – with a three-week lead time before a dinner, I'd expect an answer within a week. That said, why not include an "RSVP by" deadline on future invitations? There's no fuss and no guessing that way (although you also lessen the likelihood of receiving pouty "glad we could so easily be replaced" letters, which is sad, because by any measure, they're comic gold).
With an RSVP deadline, invitations effectively self-destruct, which is handy, really: If you time it right, you can cycle all the way down to your D-list, with plenty of time to spare.