I think if a 23-month-old has yellow teeth with any kind of holes then he needs to see a dentist soon. Holes in teeth can be due to a lack of proper formation but are usually due to decay. Dental decay is related to bacteria but not those found on a pacifier. You state that you try to have him brush frequently but he refuses; I think this is the important part of the question. At 23 months, I recommend that parents do all the brushing at least twice a day even if the child refuses. Remember, you are the parent! I would also recommend you to get your son to a dentist very soon.
Our parenting advice is given as suggestions only. We recommend you also consult your healthcare provider, and urge you to contact them immediately if your question is urgent or about a medical condition.