Lymphocytic hypophysitis is a rare inflammatory disorder which is caused by autoimmune destruction of the pituitary gland. Almost all reported cases have been in women and the disease is often associated with pregnancy. We describe here the first male case of lymphocytic hypophysitis in Korea. The patient presented with headache, impotence, decreased libido, and deteriorated vision. Endocrinologic studies showed panhypopituitarism, and pituitary MRI imaging revealed a homogeneously enhanced pituitary mass with a thickened stalk. Treatment with prednisolone and thyroid hormone for five months was ineffective. Transsphenoidal resection of the pituitary mass was performed successfully with normalization of the visual field defect. Histologic examination revealed diffuse lymphocytic infiltration with dense collagenous fibrosis, consistent with lymphocytic hypophysitis. Lymphocytic hypophysitis should be considered in differential diagnosis even in men with hypopituitarism and an enlarged pituitary gland.