Python Interning

In Python, a variable is just a pointer to a memory, where the value of that variable is stored. This means, when we do x = 5, then x contains an address to a memory location where the value 5 is stored.

a = 5
print(hex(id(x)))

This prints out 0x10108e7e0 which is the memory location of the value 5.

When we assign another variable to the same value then it will also point to the same memory location.

b = 5
print(hex(id(y)))

This again prints out 0x10108e7e0.

On assigning another value, say 500, we see the same behavior.

c = 500
print(hex(id(c)))

The output of this is 0x1013e3a50. However, now if we assign the value 500 to another variable, it gets assigned to another memory location.

d = 500
print(hex(id(d)))

This is stored in location 0x1013e3af0. When we compare c is d, the output is False and on comparing a is b, the output is True. This is because is compares memory location. However, on comparing with == operator, both these comparisons output True.

This is due to a Python feature called integer interning. Python caches a global list of integers in the range [-5, 256] on startup as these are used frequently. Internally, these are created using the singleton strategy.

Similar to integers, Python also implements string interning. This means that the strings which are instantiated at compile time are cached in memory. This idea holds good because since strings in Python are immutable objects and therefore only one instance of a particular string is needed at a time. One more advantage is that comparing memory addresses will take less time than comparing the individual strings with each other.

a = "test_string"
b = "test_string"
print(hex(id(a)), hex(id(a)))

This prints 0x10b2df030 0x10b2df030, which is the same memory locations. Both a == b and a is b output True. Once a string is interned, it behaves like a shared object and is available to all programs executed in the given Python session.

Python interns all the identifier strings — variable names, function and class names, dictionary keys, attribute and argument names. Apart from this, there are three rules which Python follows while interning.

• The string must be a compile-time constant.
• The string consists exclusively of ASCII letters, digits, or underscores.
• The string must not be subject to constant folding or no longer than 20 characters. Constant folding means calculation of expressions. For example, at compile-time, the two constants in the expression "test_" + "string", are folded and received as just a single string, "test_string". Also in general, strings longer than 20 characters are interned as long as they satisfy the above two rules. However, when the calculated string is longer than 20 characters, the string is not interned.

Empty strings are always interned.

Python provides functionality using which we can force the strings to be interned by using the sys.intern() method.