All I ask is to pause video while people talk... Am I being rude?

I have a hard time hearing what someone is saying while the TV is on, so I ask for people to pause TiVo while talking to me. I also like to pause it when making a comment or discussing some interesting point rather than talking over it. I've asked for others to do this, but wife refuses outright (even if the controller is on her lap) and feels I should simply hear her over the TV, then rewind if I really want to see what I missed. Sometimes, when all else fails and I don't have the controller, I simply pretend not to hear rather than deal with two conflicting noises. What can I do?