Disclaimer: I'm a cracker.
I had a weird experience back in the day. I used to be married to a Chinese woman. (That wasn't the weird part, but she was definitely quirky.) I went to China with her a few times, and at one point the teenaged (16 or so) daughter of her friend asked me, "What do you think of Chinese women?".
I answered, "I married one, didn't I?".
This kid beamed from ear to ear, grabbed the nearest person she could, and said, "Phineas thinks Chinese women are very beautiful!".
Fast forward a few years. I'm divorced and living on the other side of the country. Shortly after moving to California, I had a few very unfortunate incidents with Asian-American women, where saying something as simple as "I want to connect with Asian people, it's what I'm used to" got me labeled a creepy dude with yellow fever. So, I quickly adjusted.
In summary, Asian-Americans thought my comfort in a culture that I'd been immersed in for years was a creepy fetish-based thing designed solely to get in their pants. Asians from Asia thought "Chinese women are beautiful" was the highest compliment one could give.
So, what about the brown girls? I mean, South Asian is a thing. Oddly enough, all of the Indian women I've either dated or otherwise done foot stuff with had absolutely zero problem with me saying "Indian women are hot. Wanna?". The answer was nearly always "your place or mine?", unless they were trolling for a tomorrow-husband.
The most hypocritical one was my bestie, a Puerto Rican gal who solely dated black men but decried my "yellow fever" as being racist. It was different for her, you see, because reasons. (She's currently married to a hispanic guy so maybe she was right in the end.)
So yeah, I guess what I'm trying to say is there's no one-size-fits-all answer; how people feel about it is definitely rooted in (sub)cultural norms and individual preferences.