Can be seen from a distance but not played with: free variables in Python

Article directory

refer to
describe
Three attitudes of variables
- - global variable
  - free variable
  - - - cover phenomenon
  - local variable
UnboundLocalError
- - UnboundLocalError exception error
  - - func1()
    - func2()
  - Bytecode and dis.dis()
  - - bytecode
    - dis. dis()
    - - Try to disassemble func1()
      - Try to disassemble func2()
    - The reason why UnboundLocalError is thrown
global and nonlocal keywords
- - global keyword
  - nonlocal keyword
Is it not allowed to modify the value pointed to by a free variable or the reference it holds?
- - Mutable and immutable objects
  - - mutable object
    - immutable object
  - reveal the answer

Reference

Project	Description
Search Engine	Google, Bing
Python	official documentation
beep哩哔哩	[python] Can’t read the defined variables? Explain the use of global variables and free variables in detail!

Description

Project	Description
PyCharm	2023.1 (Professional Edition)
Python	3.10.6

Three postures of variables

Global variables

In Python, a global variable is a variable defined at the top level of the program (the top level refers to the outermost scope or namespace in the program, that is, the global scope), and can be anywhere in the entire program access. They are not restricted to a specific function or class, but are visible in the scope of the entire program.

Give me a chestnut

# Define variables in the global scope global_var
global_var = 'Hello World'


def func():
    # try to access global variable in local scope
    print(global_var)


func()

Execution effect

Hello World

Although global variables can be accessed everywhere in the program, excessive use of global variables can make the code difficult to maintain and debug. Therefore, when designing a program, reduce the use of global variables, and try to use function parameters and return values to pass and obtain data that needs to be shared.

Note:

In Python, inner scopes can access variables in outer scopes, but outer scopes cannot directly access variables in inner scopes.

Free variables

In Python, free variables are variables that are referenced but not defined inside a function, they are defined in a scope outside the function. Free variables can be global variables or local variables.

When a variable is referenced inside a function, Python first looks for the variable in the function’s local scope. If not found, it continues to look in the outer scope. This lookup continues until the variable is found or until the global scope is reached (more precisely, the built-in scope is reached).

Give me a chestnut

def external_func():
    free_var = 'Hello World'

    def internal_func():
        # The free_var variable is not defined, but in this scope
        # This variable is used, so free_var is a free variable
        print(free_var)

    # Call the internal_func() function
    internal_func()


external_func()

Execution effect

Hello World

Covering phenomenon

Name Shadowing occurs in scope hierarchies (nested relationships between scopes). When a variable with the same name as the outer scope is defined in the inner (as opposed to referencing free variable) scope, it will Overshadows a variable of the same name in an outer scope. This way, when the variable is used in the inner scope, it actually refers to the variable in the inner scope, not the variable in the outer scope. For this, please refer to the following example:

# outer scope
# (relative to the scope where internal_func resides)
free_var = 'Hello World'


def external_func():
    # inner scope
    # (relative to the scope where internal_func resides)
    free_var = 'Hello China'

    def internal_func():
        # If free_var is defined in the current scope
        # variable, the value of free_var in the current scope will be used
        print(free_var)

    internal_func()

Execution effect

Hello China

Local variables

In Python, local variables are variables that are visible and accessible within a particular scope. They are variables defined within a function, class, or a specific block of code that can only be used within the scope in which they exist.

When a variable is defined inside a function, it is considered a local variable of that function and can only be used inside that function. When the function completes or exits, the lifetime of the local variable also ends.

Give me a chestnut

def func():
    # define local variable local_var
    local_var = 'Hello World'
    # Access local variables in local scope
    print(local_var)
    

func()

Execution effect

Hello World

Note:

Local variables cannot be accessed outside the function. If you try to do this, Python will throw an exception error message.

UnboundLocalError

UnboundLocalError exception error

Consider the following code:

func1()

free_var = 'Hello World'


def func1():
    print(free_var)
    

func1()

During the execution of the above code, when the function is called func1(), the console will correctly output the result Hello World.

func2()

free_var = 'Hello World'


def func2():
    print(free_var)
    # Attempt to assign a value to the free_var variable
    free_var = 'Hello China'


func2()

Executing the above code will generate an abnormal error message, which contains the following content:

UnboundLocalError: local variable 'free_var' referenced before assignment

UnboundLocalError: local variable 'free_var' referenced before assignment is an exception error provided by Python, which indicates that a variable that has not been assigned before use is referenced in the local scope.

Bytecode and dis.dis()

bytecode

In Python, bytecode is an intermediate form of instruction set, which is executed by the Python interpreter. Python source code is compiled into bytecode before being run, and then the bytecode instructions are executed one by one by the interpreter.

dis. dis()

The dis.dis() function is a function in the Python standard library, which is used to disassemble the bytecode of a given function or method. It can be used to view the bytecode instructions of the function, help us understand the execution process of the code and optimize the code.

dis.dis(x=None, *, file=None, depth=None)

Note:

* in the parameter list means that the subsequent parameters of this symbol must be passed by keyword.

Where:

parameter	function
x	The function object or code object to disassemble. Can be a function, method, class, generator, source code string, etc. object.
file	Output the disassembly result to the file object provided by this parameter. If no file object is specified, output is defaulted to the standard output object ( `sys.stdin` ).
depth	Controls the depth of disassembly. If `None (default)`, all nested code objects are disassembled. If it is a `positive integer`, it means to limit the nesting depth of disassembly.

Attempt to disassemble func1()

from dis import dis


def func1():
    print(free_var)


dis(func1)

Execution effect

 5 0 LOAD_GLOBAL 0 (print)
              2 LOAD_GLOBAL 1 (free_var)
              4 CALL_FUNCTION 1
              6 POP_TOP
              8 LOAD_CONST 0 (None)
             10 RETURN_VALUE

Let’s explain what these bytecode instructions mean and do one by one:

LOAD_GLOBAL 0 (print)
This instruction pushes the print object in the global namespace as an operand into the stack (the stack is a storage structure with First In Last Out features).
LOAD_GLOBAL 1 (free_var)
This instruction loads the name free_var from the global namespace onto the stack. It pushes the free_var object in the global namespace as an operand onto the stack.
CALL_FUNCTION 1
This instruction calls the function object on top of the stack, passing the given number of arguments. Here, the function object is print, and the number of parameters is 1. So it passes the 1 argument on top of the stack, which is free_var, to the print function and calls it.
POP_TOP
This instruction pops an object from the top of the stack and discards it. Here, it will pop up the result returned after the print function call.
LOAD_CONST 0 (None)
This instruction loads the constant None onto the stack. It pushes a None object as an operand onto the stack.
RETURN_VALUE
This instruction takes the value at the top of the stack as the return value of the function and ends the execution of the function. Here, it uses the None object at the top of the stack as the return value of the function.

Attempt to disassemble func2()

from dis import dis


def func1():
    print(free_var)
    free_var = 'Hello China'


dis(func1)

Execution effect

 5 0 LOAD_GLOBAL 0 (print)
              2 LOAD_FAST 0 (free_var)
              4 CALL_FUNCTION 1
              6 POP_TOP

  6 8 LOAD_CONST 1 ('Hello China')
             10 STORE_FAST 0 (free_var)
             12 LOAD_CONST 0 (None)
             14 RETURN_VALUE

Let’s explain what these bytecode instructions mean and do one by one:

The bytecode corresponding to the fifth line of code in the source code is:

LOAD_GLOBAL 0 (print)
This instruction loads the print object from the global namespace onto the stack.
LOAD_FAST 0 (free_var)
This instruction loads the function’s local variable free_var onto the stack.
CALL_FUNCTION 1
This instruction calls the function object on top of the stack, passing the given number of arguments. Here, the function object is print, and the number of parameters is 1. So, Python passes the 1 parameters on the top of the stack (ie free_var) to the print function for calling.
POP_TOP
This instruction pops an object from the top of the stack and discards it. Here, it will pop up the result returned after the print function call.

The bytecode corresponding to the fifth line of code in the source code is:

LOAD_CONST 1 ('Hello China')
This instruction loads the constant 'Hello China' onto the stack.
STORE_FAST 0 (free_var)
This instruction stores the value at the top of the stack (i.e. 'Hello China') into the local variable free_var of the function.
LOAD_CONST 0 (None): This instruction loads the constant None onto the stack.
RETURN_VALUE
This instruction takes the value at the top of the stack as the return value of the function and ends the execution of the function. Here, it uses the None object at the top of the stack as the return value of the function.

Note: This is an explanation of the bytecode instructions, which shows how the code executes under the hood. for understanding

The reason why UnboundLocalError is thrown

Comparing the bytecodes of func1() and func2() functions, it can be found that func1 is used when loading free_var variables The bytecode instruction of LOAD_GLOBAL is LOAD_GLOBAL, and the byte instruction used when func2 loads the free_var variable is LOAD_FAST .
func1() tries to load the free_var variable from the global variable, while func2 tries to load the free_var from the local variable variable. Because func2 uses local variables before local variables are defined, an UnboundLocalError exception occurs.

The reason for the above phenomenon is: By default, Python only allows local scope to access free variables, but does not support modifying the value of free variables.

Python does not support the advantages of modifying free variables by default:

Benefits	Description
Avoid unexpected behavior	Prohibiting functions from directly modifying free variables can reduce non-deterministic and hard-to-debug errors in programs.
Clear scope boundaries	Restricting the modification of free variables helps to clarify the boundaries of the scope, making the code `easier to understand and maintain`. Modifications of variables only happen inside a specific scope and will not affect outer scopes.
Protecting the global state	Restricting the direct modification of the global variable by the function can protect the `consistency of the global state` and improve the code maintainability. Global variables are defined in the global scope and can be accessed and modified by any part of the program.
Function encapsulation and code reuse	Restricting the modification of free variables by functions helps to achieve `encapsulation` of functions and code `Reusability` promotes `modular` development.

If you need to modify the value of a free variable, you need to declare this to Python using global or nonlocal .

global and nonlocal keywords

global keyword

In Python, global is a keyword used to access and modify global variables inside a function.

When a variable defined in the global scope needs to be modified (or only accessed without modification) inside the function, the global keyword needs to be used to declare the variable as a global variable. This tells the Python interpreter not to create a local variable with the same name when using the variable inside a function, but to use the variable in the global scope directly. For this, please refer to the following example:

free_var = 'Hello World'


def func():
    global free_var
    # try to access the global variable free_var
    print(free_var)
    # try to modify the global variable free_var
    free_var = 'Hello China'


func()

# Changes to free_var inside the function will
# Respond to the global scope.
print(free_var)

Execution effect

Hello World
Hello China

Note:

The global keyword can be used in the global scope. But using the global keyword in the global scope is meaningless and will not cause a syntax error.

When using the global keyword to declare a variable as a global variable, the variable does not need to exist, but when trying to access or modify the variable, if the variable has not yet been created in the global scope, then Python will throw a NameError exception error message. For this, please refer to the following example:

def func():
    # Although the free_var global variable is not defined in the global scope at this time
    # , but free_var can still be declared as a global variable.
    # Subsequent access to the variable will look for the variable directly from the global scope.
    global free_var

    # If the relevant global variables are not defined in the global scope,
    # Attempts to access or modify variables will result in a NameError exception.
    # At this time, if the following statement is executed, it will cause a NameError exception error.
    # print(free_var)


func()

In other words, the role of the global keyword is only to tell the Python interpreter that it should look for a variable in the global scope without checking whether the variable is defined in the global scope.

If you declare a variable as global using the global keyword, you cannot attempt to access or modify the variable before the global keyword. Otherwise, Python will throw an exception error. For this, please refer to the following example:

free_var = 'Hello World'


def func():
    # Attempt to access or modify its
    # Declared global variables will cause SyntaxError.
    # When Python executes to the next line of code, a SyntaxError will be thrown.
    print(free_var)
    global free_var
    print(free_var)
    free_var = 'Hello China'


func()

print(free_var)

nonlocal keywords

When a nested function (a function inside a function is called a nested function) needs to access and modify a variable located in the scope of the outer function, if the variable is not global Variable, but a local variable or free variable of an external function, you need to use the nonlocal keyword. By using the nonlocal keyword, we can explicitly declare a variable as non-local, allowing the variable’s value to be modified within nested functions. For this, please refer to the following example:

def external_func():
    free_var = 'Hello World'

    def internal_func():
        nonlocal free_var
        # Attempt to access a free variable that is not a global variable
        print(free_var)
        # Attempt to modify free variables of non-global variables
        free_var = 'Hello China'

    # Execute nested functions internal_func()
    internal_func()

    # Modifications of nested functions to free variables of non-global variables will reflect
    # to the local scope where the variable is located.
    print(free_var)


external_func()

Execution effect

Hello World
Hello China

Note:

The nonlocal keyword has roughly the same restrictions as the global keyword, with the following differences:

The nonlocal keyword cannot be used in the global scope. Otherwise, Python will throw a SyntaxError exception.
Free variables for non-global variables declared by nonlocal must already exist during function definition. Otherwise, Python will throw a SyntaxError exception. For this, please refer to the following example:

def external_func():

    def internal_func():
        # A SyntaxError exception error will be generated when the next line of code is executed.
        # Python checks for the existence of
        Free variables for non-global variables declared by the # keyword.
        nonlocal free_var

    internal_func()


external_func()

Is it not allowed to modify the value pointed to by the free variable or the reference it holds?

Mutable and immutable objects

In Python, objects are divided into two types: Mutable Objects and Immutable Objects.

Variable objects

Mutable objects are objects whose internal state can be modified after creation. Common mutable objects include lists (list), dictionaries (dict) and collections (set), etc. When operating on a mutable object, its internal state can change, but the identity of the object itself (Identity, the identity of the object is the storage address of the object in memory) will not A change has occurred. For this, please refer to the following example:

arr = [1, 2, 3, 4, 5, 6]

# Attempt to output the identity of the mutable object arr
print(id(arr))

# Modify the mutable object
arr. pop()

# Attempt to output the identity of the mutable object arr
print(id(arr))

Execution effect

Since the identity of the object is a memory address. Therefore, the output results in different computers have different probabilities. But what is certain is that the value of the identity of the two outputs is the same.

2609552406912
2609552406912

Immutable objects

An immutable object is an object whose internal state cannot be modified after creation. Common immutable objects include integers (int), floating point numbers (float), Boolean values (bool), strings ( str) and tuple (tuple), etc. When operating on an immutable object, an object will be recreated with a different identity representation than the original object.

think = 'Hello World'

# Attempt to output the identity of the immutable object think
print(id(think))

think + = '\\
Hello China'

# Attempt to output the identity of the immutable object think
print(id(think))

Execution effect

Since modifying an immutable object will create a new object, the reference (that is, the memory address) associated with the identifier has also changed.

2629283135216
2629283524688

Reveal the answer

When modifying a free variable inside a function, Python does not allow to directly modify the reference (memory address) saved by the free variable, that is, bind the free variable to a new object . If you want to modify the reference saved by the free variable, you need to use the global or nonlocal keyword to declare the variable, and then modify it.

However, Python allows to modify the value pointed to by a free variable, that is, modifies the state inside the object without changing the reference to the object itself. This means that for mutable objects (such as lists, dictionaries, etc.), their internal values can be modified by reference without using the global or nonlocal keywords. But for immutable objects (such as integers, strings, etc.), their values cannot be modified, and each operation will return a new object.

free_var = list('Hello')


def func():
    # try without using global or nonlocal keywords
    # Modifies the mutable object.
    free_var.extend(list('World'))


func()
# use an empty string to concatenate all elements in the list into one string
print(''. join(free_var))

Execution effect

Hello World