С этим оператором SELECT
SELECT MIN(t.ID),t.Score,t.ChallengeId,t.playerID,MIN( from_unixtime(date) )
FROM `times` t inner Join
(SELECT MIN(Score) score,challengeID,playerID
from `times`
group by challengeID,playerID
Order by Score ASC) s
ON t.challengeID = s.challengeID
and t.playerID = s.playerID
and t.Score = s.score
GROUP BY t.Score,t.ChallengeId,t.playerID
order by t.Score
Вы получите этот результат (Здесь отображаются только первые
ID Score ChallengeId playerID from_unixtime(date)
90488 10450 466 28 2018-03-20 02:16:29
92155 10450 466 8 2018-03-24 02:05:18
92448 10450 466 147 2018-03-24 13:37:02
92763 10450 466 97 2018-03-24 19:15:42
92787 10450 466 410 2018-03-24 19:23:24
85201 10460 466 255 2018-03-06 05:13:46
86256 10460 466 66 2018-03-09 10:48:16
92778 10460 466 1846 2018-03-24 19:21:07
84801 10470 466 47 2018-03-05 16:29:41
84804 10470 466 944 2018-03-05 16:30:34
85724 10470 466 599 2018-03-07 08:07:30
88139 10470 466 1651 2018-03-15 11:16:54
DBfiddle new sample https://dbfiddle.uk/?rdbms=mysql_5.6&fiddle=de581e5a5b6fa9a184cc2e235337b2d5