Been living in SK for a LONG TIME. Since childhood. I lived in america for most of my child life but even living in Korea so long, my Korean is fucking awful. I practice and go to classes. Just so hard to learn as an adult. If someone in my position could not learn local language then how do people expect non natives and non residents of a country to learn? Especially with so little time to do so as you have said yourself.
These people saying "just learn japanese" have no clue how hard languages are. That's why anime fansubs have been popular since as far back as anime has been in the west. I like to imagine these people going to JP and telling them to "just learn English" when they want translations to western media and getting punched in the head.
I think when people say that it's harder to learn a language as an adult, they think so for the wrong reasons. In many cases, it's actually easier to learn as an adult. The main difference is simply exposure. When growing up, your mother tongue is all you hear and read. You are constantly surrounded by it, so you naturally pick it up, and as you don't know any other words for the things around you yet they are easier to remember because you cannot express yourself in any other way.
But think about how many years you still spend improving that language, the difference in how you express yourself as a kid and as an adult. You probably couldn't even read a solid novel until you were quite a few years into school. You can reach that point in Japanese a lot faster as an adult, as long as you expose yourself to the language enough, daily. That's the reason many people think it's harder to learn a language as an adult, insufficient exposure. Of course it's harder to get as much exposure from media as you get growing up in the language's country, where it's basically forced exposure 24/7, but you can still get pretty close to it.
I'm only a bit over half a year in and the start was definitely difficult and discouraging, but I also did not spend that much time with the language outside of doing daily Anki/Grammar. I'm definitely still lacking in a lot of places, but my understanding has gone up immensely ever since I started reading and listening more. That's also the point where it actually becomes fun to learn the language, because all you are doing is what you wanted to do anyway: spend time with Japanese media and learning new words that way. And having fun makes it easier to remember stuff. There's also just a lot of things you pick up subconsciously once you see them enough times through repeated exposure.
Of course, some people struggle more with languages than others. Now I don't know anything about you, but could it be that you mostly stick to English-speaking bubbles? That would obviously cause insufficent exposure. Of course, simply having little interest in the language will make it harder for you to learn it as well but again - I don't really know you so these are just guesses.
But yes, people who say shit like "just learn Japanese if it bothers you that much" should tell that to the "translators" instead, they are getting paid to bring the media to another language after all yet they can't seem to do it properly. I'm aware that most of it is done on purpose rather than due to insufficent knowledge of the language, but still.