I have a love-hate relationship with Japan. I still go every year, but I'm not under any illusions either. I think being a foreigner there long-term is very tough; a lot of people treat you with apprehension or bemusement, like you're a chimpanzee who managed to stand upright and put some clothes on. I dress well too, even by Japanese standards as it's part of what I do, so it's not like I'm rolling up there with a bad haircut, unshaven and with cheap clothes.
On top of that it's just socially broken. Foreigners often don't really make local friends except for their GF/BF. Also a lot of weirdo foreigners too, so it's not like you want to be friends with many of them. Interactions are almost always rote and formal unless you're dealing with locals who've lived abroad. Trust in relationships is low; everyone's cheating. Of course that works out to a man's benefit in a way.
It really is a lovely place to visit though; the best overall country in Asia, easily. Safe, first-class, incredible looking women (or men, depending), historical, best shopping in the world, good food, lots of beautiful nature, etc.
I just would need to be loaded to live there.