After studying Cantor's epsilon numbers more, I've come to realize that things aren't as simple as they seemed at first. On the surface, it seems that it should be easy to define transfinite tetration: after all, following the pattern of how a+w, a*w, and

are defined, it's just a matter of taking the union of all the elements in the sequence of exponential towers, right? And then the ordinals following that union are just a matter of taking that union, and then repeating the exponential tower operation on it.
However, after trying to rigorously define tetration via transfinite recursion, I ran into a seemingly impassable roadblock. The root of the problem is that exponentiation is non-associative. This in itself does not make things too hard when we're defining exponentiation, because exponentiation is based on iterating multiplication, and multiplication is associative. Because multiplication is associative, the "tail end" of the transfinite product w*w*w*... is still "accessible": we can define w^(w+1) to be (w*w*w*...)*w. So transfinite recursion works past limit ordinals because of this property.
Now, when defining tetration, we run into a problem: in an exponential tower w^(w^(w^x))), the "top" of the tower is inaccessible: once it is formed, we have no way of reaching into the x and stacking more exponentials onto it. The most we can do is to "push" the tower from below by raising the tower into the exponent of more w's. This is OK for towers of finite height, because in constructing w[4]w, we don't need to change anything at the top of the tower. However, once the height of the tower reaches w, we have a problem: it is now w^(w^(w^(w^...))), and adding more w's to the bottom of the tower does not make it any higher. What we want is to add more exponents to the
top of the tower. However, there is no way to access the top part of the tower to add more exponents to it! There is no straightforward way to define what w[4](w+1) means, because the non-associativity of exponentiation means that w[4](w+1) ≠ (w[4]w)^w. No matter how we try, there's no way to reach into the second argument of the tetration to make it bigger (no way to do this rigorously, anyway).
So we have a problem. Intuitively, we
know what w[4](w+1) is: it's just an exponential tower of height w+1. But I can see no way of defining this via transfinite recursion.
Because of these problems, it seems that there is no other way but to use Cantor's fixed point definition of the epsilon numbers. (So it seems that Cantor wasn't just ignorant of higher operations... there are technical problems with the naive construction of higher operations via transfinite recursion.) Here is one way I've found:
First, given that we intuitively understand w[4](w+1) to be a tower of height w+1, we may imagine laying it out each element in the tower in a sequence: w, w, w, ... w (where the last w lies past the end of a stretch of w occurrences of w). Now, if we add one more w to the base of the tower, this does not change the resulting sequence. In other words, this means that w^(w[4](w+1)) = w[4](w+1). So it must be an epsilon number! (Since the epsilon numbers are defined as ordinals x that satisfy x=w^x.) The question is then, which epsilon number is it?
To answer this question, we look at the behaviour of epsilon_0, which is w[4]w. Suppose we add 1 to it to get w[4]w+1. Obviously, this no longer satisfies the property x=w^x. Moreover, we can continue to add things to w[4]w, and all of them do not satisfy this property. After that, we may try multiplying w[4]w by 2, 3, 4, and so forth, and we see that all those ordinals don't satisfy the property either. We may even raise w[4]w to some ordinal exponent, such as (w[4]w)^w = w^(w*(w[4]w)), and so forth, but none of those satisfies the property either. In fact, we shall soon realize that
none of the ordinals less than w[4](w+1) satisfy this property!
Which means that w[4](w+1) is the next smallest ordinal satisfying x=w^x. This means that w[4](w+1) must be equal to epsilon_1. A similar line of argument will quickly show that w[4](w+2) must equal to epsilon_2, and so forth. In other words, w[4](w+x)=epsilon_x, for any ordinal x.
Hooray! So finally, we have found a way to define ordinal tetration. At least, we've defined it where the base of the tetration is w. I'm not sure if the fixed point trick will still work correctly for ordinals greater than w (this needs further research).
Now that we have a working definition, we can examine some of the properties of ordinal tetration. We have already seen that w[4](w+x) = epsilon_x. More specifically, w[4](w+w) = w[4](w*2) = epsilon_w, w[4](w+w+w) = epsilon_(w+w), w[4](w+w+w+w) = epsilon_(w+w+w), and so forth. Here, we notice something interesting: as we take the limit of this sequence, we see that even though the epsilon subscript has 1 less term than the argument to the tetration operation, when we reach epsilon_(w*w), the tetration argument also collapses: w[4](w*w). In other words, for x≥w*w, epsilon_x = w[4]x.
As we move to larger and larger x, we eventually reach epsilon_(w^w^w^...) = epsilon_epsilon_0 = w[4](w[4]w) = w[5]3. So now we see a new pattern: w[5]n = epsilon_epsilon_epsilon_..._epsilon_0, where there are n epsilon's. This lets us define w[5]w as the smallest ordinal x satisfying x=epsilon_x.
Now, the epsilons can no longer help us to go much farther; so pentation cannot in fact be defined in terms of the epsilon numbers. In order to do this, we will need to go full-scale into the Veblen hierarchy. Happily, using the method I described above, it is possible to rigorously define at least all the hyper operators of finite order with the help of the Veblen hierarchy. I'm a bit less certain about bigger things like the Ackermann function, but I suspect it would also work. The Veblen hierarchy is extremely powerful, and goes
way past most well-known large number functions.
I might post about the Veblen hierarchy sometime in another thread, it's a very interesting subject that has profound implications for how to construct
computable functions that make the Ackermann function look pathetic.