I’ll keep you in the loop. And yes, it’s a pity we only support 3.x for now (we were not expecting to have an usable engine so “early,” and were hoping for a faster uptake of the new version). In fact, that makes it very difficult for us to cut our teeth on real-world examples; I guess I’ll just bite the bullet and integrate with 2.6 too.

CrossTwine Linker does not currently include a tracing JIT, it’s more of a “classical” design. I’m not sure a tracing JIT buys you a lot on this particular front, though: the compiler still has to recognize the situation to be able to reuse frames (TANSTAAFL). On the other hand, you are right that it might be able to automatically inline—and possibly coalesce—frame creation, and to peephole the sequence of returns away.

Truth to be told, TCO (and inlining in general) are not “free” optimizations, especially in a dynamic language where a program can examine (and manipulate!) its call stack at any time. I’ll probably hack the Python compiler to do analysis on a couple large applications before I venture down that path…

P.-S. — As you mentioned, your “factorial” function is not eligible for TCO, which would require an accumulator to be used. I’m not aware of any (non-research) compiler which does that kind of transformation behind the scenes, though, which leads me to think it’s not worth the trouble. (But then, I’ve been wrong before.

For what it’s worth, these results were on python 2.6.

Crosstwine looks interesting, but unfortunately bytecodehacks isn’t available for python 3 (and 2to3 hasn’t managed an automatic conversion), so I haven’t been able to quickly test if this still offers a performance increase on crosstwine.

I don’t know what kind of JIT you are using, but I think that a tracing JIT could be modified automatically do most of the optimisation I’ve done – I started having a look at the tracing JIT in the newest version of pypy to see if it does, but I haven’t managed to get my head around the code well enough yet.

I’m afraid I don’t have any stats about how often tail-recursive calls are likely to be used, although there is also the wider variety of calls that could be converted to one.

e.g.
def f(x):
return x*f(x-1)

(which can easily be converted to the lisp-like version used above)

Even though I believe tail-recursive calls are pretty rare in real-world scenarios, it would be interesting to know which percentage of the calls are eligible for TCO in a typical Python program. Would you happen to have the numbers at hand?

Also, reusing the frame is not going to be possible when it is not wide enough to hold all the variables of the callee. I’ll probably toy a bit with the idea in our Python JIT when I get the opportunity; tell me if you would like to hear about the results.