Вот быстрый способ. A
сортировать не нужно, числа могут быть любыми.
A = [1,1,1,1,2,2,2,2,3,3,3,4,4,5];
%# count the numbers in A (use unique so that the array
%# remains at a decent size even if the values are very different)
[uniqueEntries,~,idx] = unique(A);
counts = histc(idx,1:max(idx));
%# remove all the numbers whose count is less or equal than two
A(ismember(A,uniqueEntries(counts<=2))) = []