Использование case..when
оператора
COUNT(case when vote.vote='+eve' then 1 end)
в запросе
SELECT a.answer,a.id,a.user_id,
COUNT(case when v.vote='+eve' then 1 end)
-- SUM(v.vote='+eve') might be another alternative for the above COUNT
FROM user_answers AS a
LEFT JOIN user_answer_vote AS v
ON a.id =v.answer_id
WHERE a.question_id = $question_id
GROUP BY a.id
ORDER BY COUNT(case when vote.vote='+eve' then 1 end) DESC
-- SUM(v.vote='+eve') might be another alternative for the above COUNT
Редактировать (из-за вашего комментария):
SELECT a.answer,a.id,a.user_id,
SUM(case when v.vote='+eve' then 1 else -1 end )
FROM user_answers AS a
LEFT JOIN user_answer_vote AS v
ON a.id =v.answer_id
WHERE a.question_id = $question_id
GROUP BY a.id
ORDER BY SUM(case when v.vote='+eve' then 1 else -1 end ) DESC
Демонстрация для агрегаций выше