I've heard from a few learners that they're worried they may use a neutral or impolite tone instead of being formal when necessary (not using gozaimasu for example), would you say that most native speakers won't care too much about this outside of a professional setting?
Yeah, we don't care about it we understand many foreigners can't speak Japanese not well.
If you made a mistake Japanese accent, usually we don't mention about that ( I'll mention that )
But when we didn't understand your Japanese, we might ask about what you saying.
we know Japanese language is so hard but if you try to speak Japanese we are happy.