Switching from 140mm to 160mm rear rotor: Big Deal?

Until recently, my Felt F65x's been sporting a set of stock BB5s, but a friend of mine decided to send me his old BB7s. I use this bike mostly for gravel grinding and urban commuting, so no racing yet.

Now the front brake, I had no problem swapping out: both were 160mm, and the front was always annoying me anyway because BB5s are a PITA to adjust. However, the rear brake is a 140mm, which means I'd have to get a new rotor if I wanted to go from a BB5 to a BB7.  This is the first bike with disc brakes I've ever owned: I have no idea what kind of impact this will have on my riding experience, or if such a change will even be good for my bike.

tl;dr. Rear wheel has a 140mm rotor with a so-so caliper. Have the option to swap to a 160mm rotor with a better caliper. Not sure if should.