Это простой перевод определения Википедии :
public static long hyper(long n, long a, long b) {
return
(n == 0) ? b + 1 :
(n == 1 && b == 0) ? a :
(n == 2 && b == 0) ? 0 :
(n >= 3 && b == 0) ? 1 :
hyper(n-1, a, hyper(n, a, b-1));
}
Пример вывода:
System.out.println(hyper(0,2,3)); // prints "4" increment
System.out.println(hyper(1,2,3)); // prints "5" addition
System.out.println(hyper(2,2,3)); // prints "6" multiplication
System.out.println(hyper(3,2,3)); // prints "8" exponentiation
System.out.println(hyper(4,2,3)); // prints "16" tetration
System.out.println(hyper(5,2,3)); // throws StackOverflowError; too deep