Вот одна из возможностей:
% A^3 - 4*A^2 + [1 0 2;-1 4 6;-1 1 1] = 0
% 1) Change base to diagonalize the constant term
M = [1 0 2;-1 4 6;-1 1 1];
[V, L] = eig(M);
% 2) Solve three equations "on the diagonal", i.e. find a root of
% x^4 - 4*x^3 + eigenvalue = 0 for each eigenvalue of M
% (in this example, for each eigenvalue I choose the 3rd root,
% which happens to be real)
roots1 = roots([1 -4 0 L(1,1)]); r1 = roots1(3);
roots2 = roots([1 -4 0 L(2,2)]); r2 = roots2(3);
roots3 = roots([1 -4 0 L(3,3)]); r3 = roots3(3);
% 3) Build matrix solution and transform with inverse change of base
SD = diag([r1, r2, r3]);
A = V*SD*inv(V) % This is your solution
% The error should be practically zero
error = A^3 - 4*A^2 + [1 0 2;-1 4 6;-1 1 1]
norm(error)
(Ошибка на самом деле порядка 10 ^ -14.)