It's fine, don't worry about it. There used to be a rule (and it was enforced) that you couldn't wear pro team jerseys. I haven't seen that rule enforced in a while although a lot of people still frown upon it. But as far as wearing logos from people you are not sponsored by (say a Twin Six jersey) that's not something you need to worry about.
You might be the most considerate bike racer ever. No one ever thinks to ask something like that.