Probably a demographic issue between your site and the users of the different SE's. Other possibility that I can see is the keyword(s) you're ranking with are not as relevant to what people are wanting as you (or yahoo) may think. Try looking at your keywords with fresh eyes (ask a friend/collegue to help).
On the bright side, you're ranked top 5 on yahoo. That's excellent. Now you need to do your best to stay there.