One interesting corollary is that moving short strings in an implementation that does this could actually be ever so slightly (negligibly) slower than moving long ones (since byte copies are slower than word copies). But generally, this is a free lunch optimisation and can save you hundreds of megs of memory when writing programs dealing with millions of short strings.

[1] http://llvm.org/svn/llvm-project/libcxx/trunk/include/string - search for "union"

Also (pedantic):

   #define RSTRING_EMBED_LEN_MAX ((int)((sizeof(VALUE)*3)/sizeof(char)-1))

Conclusion: "Don’t worry! I don’t think you should refactor all your code to be sure you have strings of length 23 or less."

    struct RString {

      struct RBasic basic;

      union {
        struct {
          long len;
          char *ptr;
          union {
            long capa;
            VALUE shared;
          } aux;
        } heap;
    
        char ary[];
      } as;
    };

    /* apologies if I messed up the syntax here */
    #define RSTRING_EMBED_LEN_MAX (sizeof(((RString*)(0))->as) - 1)

http://ruby-hacking-guide.github.com

VALUE seems to be unsigned int defined via "typedef uintptr_t VALUE;" and "typedef unsigned __int64 uintptr_t;"

But why is it calculated like that I don't get. Anyone can explain?

When programmers don't know in advance how long name/email/input/whatever field is going to be - they just use the magic "power of two" length :)

So 32 (or 33) in this case would be more reasonable.

If this is intended to sit on the stack, which I find highly unlikely (especially given the timings that seem to be the delta between one malloc and two, and would be much more significant if it were a stack allocation versus a heap allocation. This is not comparable to small string optimizations for the stack in C++), maybe. But otherwise it seems like a poorly considered hack.

The string type could as easily have been dynamically allocated based upon the length of the string, where the ptr by default points inside that same allocated block. If the string is expanded it can then be realloced and the string alloced somewhere else. No waste, a single allocation, etc.

    allocation of the RString (which becomes larger and thus more difficult to malloc), and the 23 string bytes that will sit unused for longer strings.

The key point is that for short strings, you're not adding it to the cost of the long string, but instead overwriting the data that you would track the long string.

In other words, the memory layout you get is this:

     short:
       [Rbasic]['s', 't', 'r', 'i', 'n', 'g', '\0']
     long:
       [Rbasic][ length ][ dataptr ][capa or value]