Recognizing barriers to managing sexual issues makes it more likely that effective ways to overcome them will be found. In Malaysia, where discussion of sexual issues is taboo, sociocultural factors may influence how physicians manage patients with these types of problems. This article focuses on the challenges encountered by 21 Malay family physicians when women experiencing sexual problems and female sexual dysfunction (FSD) attended their clinics, an uncommon occurrence in Malaysia, despite their high prevalence. This qualitative study employed a phenomenological framework and conducted face-to-face in-depth interviews. Three main barriers to managing women with sexual problems were identified that can hinder assessment and treatment: insufficient knowledge and training; unfavorable clinic environments; and personal embarrassment. Some barriers were associated with physician characteristics but many were systemic. These were further evaluated using social cognitive theory. Professional attitudes appear important as those physicians with an interest in managing women's health seemed to make greater effort to explore issues further and work to gain trust. Physicians who appeared indifferent to the impact of FSD showed greater reluctance to find solutions. Systemic issues included unfavorable clinical settings, lack of training, and lack of local evidence. Any strategy to address FSD needs to be underpinned by appropriate policies and resources.