Think about this: you have a colleague who has a muscular issue, and from the outside you can't tell but she can't climb stairs... are you going to be angry and say don't talk about elevators in this school, because I have been using the stairs this whole time?...
Just because you know this information and you've benefitted from this privilege, that doesn't mean you're a bad person. It means now I have information to better help my colleague"