ctypes of faster python calls so/dll

.so and .dll files

.dll (dynamic link library) and .so (shared object file) are two dynamic link library formats used on different operating systems.

.dll file:
- Dynamic link library file format used on Windows systems.
- Typically written in C/C++ and generated by a compiler.
- Can be shared by multiple programs and dynamically loaded into memory at runtime.
- Allows code and data to be shared between different programs, helping to save memory.
- In Python, you can use the ctypes library or other extension libraries to call functions in the .dll file.
.so file:
- Dynamic link library file format used on Unix-like systems (such as Linux).
- Typically written in C/C++ and generated by a compiler.
- It can also be shared by multiple programs and dynamically loaded into memory at runtime.
- Similar to .dll, allows sharing code and data between different programs.

Why can both Go and C files be compiled into .dll and .so files?

This is because compilers (such as GCC, Clang) and build tools (such as Go’s tool chain) support compiling source code into a variety of target formats, including executable files, static libraries, and dynamic link libraries.

The Go compiler supports compiling Go code into executable files, static libraries (.a files), and shared object files (.so files or .dll files), which allows Go to be used to build standalone applications as well as Build shared libraries for use by other programs.

The C compiler can also compile C code into executable files, static libraries (.a files) and shared object files (.so files or .dll files), which gives the C language similar flexibility.

In general, the .dll and .so files are to facilitate the sharing and reuse of code, so that multiple programs can share a set of functions or code libraries. This is useful for sharing code between different programs, especially when you want to avoid duplicating the same code in every program.

Compile go code into a dynamic link library

Prepare the xxx.go file (there must be a main package to compile)

package main

import "C"

//export Add
func Add(a, b int) int {<!-- -->
return a + b
}

func main() {<!-- -->}

Execute the command to compile:

go build -buildmode=c-shared -o tool.so tool.go

python call

import ctypes

mylibrary = ctypes.CDLL('./tool.so')

result = mylibrary.Add(3, 4)
print(result) # This will print out 7

Compile C code into a dynamic link library

Prepare xxx.c file

int add(int a, int b) {<!-- -->
return a + b;
}

Execute command to compile

gcc tool.c -fPIC -shared -o ctool.so
 
* -shared is the link library to let the compiler know that it is to compile a shared library
* -fPIC (Position Independent Code) compiles and generates code that is independent of position
* If you want to be able to debug, you can add parameters such as -g -Wall

python call

import ctypes

mylibrary = ctypes.CDLL('./ctool.so')

result = mylibrary.Add(3, 4)
print(result) # This will print out 7

ctypes library

Attachment: 3.7 documentation: https://docs.python.org/zh-cn/3.7/library/ctypes.html

ctypes is a module in the Python standard library. It provides an external function library interface compatible with the C language, allowing Python programs to call C functions in dynamic link libraries (DLL or .so files) . This allows Python to interact with libraries written in C or other external libraries.

The following are some main concepts and usage of the ctypes library:

Load shared libraries:

Use ctypes.CDLL() to load shared libraries. For example:
```
import ctypes

mylibrary = ctypes.CDLL('./mylibrary.so')
```
This will load the shared object file named mylibrary.so.
Call C function:

Once the shared library is loaded, you can use Python to call C functions within it. For example:
```
result = mylibrary.Add(3, 4)
```
This will call the C function named Add and pass it arguments 3 and 4.
Specify parameters and return types:

Before calling a C function, you should ensure that the parameter types and return type are correctly specified using ctypes to match the signature of the C function.
```
mylibrary.Add.argtypes = [ctypes.c_int, ctypes.c_int]
mylibrary.Add.restype = ctypes.c_int
```
In this example, we specify that the parameter type of the Add function is two integers, and the return type is also an integer.
Handling pointers and data types:

ctypes can handle basic data types in C and complex data structures such as pointers. You can map C data types using types from ctypes.
Error handling:

If you call a C function, an error code may be returned. You can handle the error by checking the return value.
Callback function:

You can use ctypes to define a Python callback function and pass it to a C function so that the C function calls the Python function at the appropriate time.
Structures and unions:

You can use ctypes to create and manipulate structures and unions in C.
Memory management:

ctypes provides tools to handle memory allocation and deallocation to ensure that problems such as memory leaks do not occur when interacting with C code.

Overall, ctypes is a powerful tool that allows Python to interact seamlessly with C code. It enables Python to take advantage of libraries written in C, while also providing a convenient way to test and debug C code. However, since ctypes is a dynamic Python library, in cases where performance is critical, you may want to consider using more advanced tools such as Cython or SWIG.

Basic data types

ctypes defines some basic data types compatible with C:

ctypes type	C type	Python types
`c_bool`	`_Bool`	bool (1)
`c_char`	`char`	Single character bytes object
`c_wchar`	`wchar_t`	single character string
`c_byte`	`char`	int
`c_ubyte`	`unsigned char`	int
`c_short`	`short`	int
`c_ushort`	`unsigned short`	int
`c_int`	`int`	int
`c_uint`	`unsigned int`	int
`c_long`	`long`	int
`c_ulong`	`unsigned long`	int
`c_longlong`	`__int64` or `long long`	int
`c_ulonglong`	`unsigned __int64` or `unsigned long long`	int
`c_size_t`	`size_t`	int
`c_ssize_t`	`ssize_t` or `Py_ssize_t`	int
`c_float`	`float`	float
`c_double`	`double`	float
`c_longdouble`	`long double`	float
`c_char_p`	`char*` (NUL terminated)	Bytestring object or `None`
`c_wchar_p`	`wchar_t*` (NUL terminated)	String or `None`
`c_void_p`	`void*`	int or `None`

How to use

Basic data types

# -*- coding: utf-8 -*-
from ctypes import *

# character, only accepts one character bytes, bytearray or integer
char_type = c_char(b"a")
# byte
byte_type = c_char(1)
# string
string_type = c_wchar_p("abc")
# Integer type
int_type = c_int(2)
# The object information is printed directly. To obtain the value, you need to use the value method.
print(char_type, byte_type, int_type)
print(char_type.value, byte_type.value, string_type.value, int_type.value)

Output:

c_char(b'a') c_char(b'\x01') c_int(2)
b'a' b'\x01' abc 2

Array type

The creation of an array is similar to that in C language. Just give the data type and length.

# Array
# define type
char_array = c_char * 3
# initialization
char_array_obj = char_array(b"a", b"b", 2)
# Printing can only print information about array objects
print(char_array_obj)
#Print value through value method
print(char_array_obj.value)

Output:

<main.c_char_Array_3 object at 0x7f2252e6dc20>
b'ab\x02'

You can also initialize it directly when creating it.

int_array = (c_int * 3)(1, 2, 3)
for i in int_array:
    print(i)

char_array_2 = (c_char * 3)(1, 2, 3)
print(char_array_2.value)

Output:

1
2
3
b'\x01\x02\x03'

It should be noted here that obtaining values through the value method only applies to character arrays. The use of other types such as print(int_array.value) will report an error:

AttributeError: 'c_int_Array_3' object has no attribute 'value'

Pointer type

ctypes provides two methods, pointer() and POINTER(), to create pointers. The difference is:

pointer() is used to convert objects into pointers, as follows:

# Pointer type
int_obj = c_int(3)
int_p = pointer(int_obj)
print(int_p)
# Use the contents method to access the pointer
print(int_p.contents)
# Get the value pointed to by the pointer
print(int_p[0])

Output:

<__main__.LP_c_int object at 0x7fddbcb1de60>
c_int(3)
3

POINTER() is used to define a pointer of a certain type, as follows:

# Pointer type
int_p = pointer(c_int)
# Instantiate
int_obj = c_int(4)
int_p_obj = int_p(int_obj)
print(int_p_obj)
print(int_p_obj.contents)
print(int_p_obj[0])

Output:

<__main__.LP_c_int object at 0x7f47df7f79e0>
c_int(4)
4

If you make a mistake in the initialization method, an error will be reported. POINTER() is as follows:

# Pointer type
int_p = pointer(c_int)
# Instantiate
int_obj = c_int(4)
int_p_obj = POINTER(int_obj)

Error reported:

TypeError: must be a ctypes type

pointer() is as follows:

# Pointer type
int_p = pointer(c_int)

Error reported:

TypeError: _type_ must have storage info

How to create a null pointer

null_ptr = POINTER(c_int)()
print(bool(null_ptr))

Output:

False

Pointer type conversion
ctypes provides the cast() method to convert a ctypes instance into a pointer to another ctypes data type. cast() accepts two parameters, one is a ctypes object, which is or can be converted into a pointer of a certain type, and the other is ctypes Pointer type. It returns an instance of the second argument that refers to the same block of memory as the first argument.

int_p = pointer(c_int(4))
print(int_p)

char_p_type = POINTER(c_char)
print(char_p_type)

cast_type = cast(int_p, char_p_type)
print(cast_type)

Output:

<__main__.LP_c_int object at 0x7f43e2fcc9e0>
<class 'ctypes.LP_c_char'>
<ctypes.LP_c_char object at 0x7f43e2fcc950>

Structure type

Implementations of struct types, structs and unions must derive from the struct and union base classes defined in the ctypes module. Each subclass must define a _fields_ attribute. _fields_ must be a list of tuples containing field names and field types. The _pack_ attribute determines the byte alignment of the structure. The default is 4-byte alignment. Use _pack_=1 when creating to specify 1-byte alignment. For example, the method of initializing student_t is as follows. What needs special attention is that the field name cannot have the same name as the python keyword.

# -*- coding: utf-8 -*-
from ctypes import *

# Student information is as follows
stu_info = [("class", "A"),
            ("grade", 90),
            ("array", [1, 2, 3]),
            ("point", 4)]

#Create structure class
class Student(Structure):
    _fields_ = [("class", c_char),
            ("grade", c_int),
            ("array", c_long * 3),
            ("point", POINTER(c_int))]

print("sizeof Student: ", sizeof(Student))

# Instantiate
long_array = c_long * 3
long_array_obj = long_array(1, 2, 3)
int_p = pointer(c_int(4))
stu_info_value = [c_char(b"A"), c_int(90), long_array_obj, int_p]

stu_obj = Student(*stu_info_value)
# This error is printed because the field name has the same name as the python keyword class. This is a point that requires special attention.
# print("stu info:", stu_obj.class, stu_obj.grade, stu_obj.array[0], stu_obj.point[0])
print("stu info:", stu_obj.grade, stu_obj.array[0], stu_obj.point[0])

Output:

sizeof Student: 40
stu info: 90 1 4

If _pack_ is changed to 1, the output is:

sizeof Student: 37
stu info: 90 1 4

Nested structure

The use of nested structures requires creating the type of the basic structure, and then using the type of the basic structure as a member of the nested structure. Note that the field type of the field to which the basic structure belongs is the class name of the basic structure, as follows:

# Create a type, the type of nest_stu field is the class name of the basic structure
class NestStudent(Structure):
    _fields_ = [("rank", c_char),
                ("nest_stu", Student)]

# Instantiate
nest_stu_info_list = [c_char(b"M"), stu_obj]
nest_stu_obj = NestStudent(*nest_stu_info_list)

print("nest stu info: ", nest_stu_obj.rank, "basic stu info: ", nest_stu_obj.nest_stu.grade)

Output:

nest stu info: b'M' basic stu info: 90

Structure array

The creation of a structure array is similar to that of an ordinary array. You need to create the type of the structure in advance, and then use the method of struct type * array_length to create the array.

# Structure array
#Create structure array type
stu_array = Student * 2
# Instantiate the structure array with an object of Student class
stu_array_obj = stu_array(stu_obj, stu_obj)

# Add structure array members
class NestStudent(Structure):
    _fields_ = [("rank", c_char),
                ("nest_stu", Student),
                ("strct_array", Student * 2)]

# Instantiate
nest_stu_info_list = [c_char(b"M"), stu_obj, stu_array_obj]
nest_stu_obj = NestStudent(*nest_stu_info_list)

#Print the information about the grade field of the second index of the structure array
print("stu struct array info: ", nest_stu_obj.strct_array[1].grade, nest_stu_obj.strct_array[1].array[0])

Output:

stu struct array info: 90 1

Structure pointer

First create the structure, and then use ctype’s pointer method to wrap it into a pointer.

# Structure pointer
# #Create structure array type
stu_array = Student * 2
# # Instantiate the structure array with an object of the Student class
stu_array_obj = stu_array(stu_obj, stu_obj)
# Once connected to the structure pointer member, note that the type initialized pointer is POINTER()
class NestStudent(Structure):
    _fields_ = [("rank", c_char),
                ("nest_stu", Student),
                ("strct_array", Student * 2),
                ("strct_point", POINTER(Student))]

# Instantiate, use pointer() to wrap the Student object as a pointer
nest_stu_info_list = [c_char(b"M"), stu_obj, stu_array_obj, pointer(stu_obj)]
nest_stu_obj = NestStudent(*nest_stu_info_list)

# The structure pointer points to the Student object
print("stu struct point info: ", nest_stu_obj.strct_point.contents)
#Access members of the Student object
print("stu struct point info: ", nest_stu_obj.strct_point.contents.grade)

Output:

stu struct point info: <__main__.Student object at 0x7f8d80e70200> # Object information pointed to by the structure pointer
stu struct point info: 90 # Student structure grade member

Array of structure pointers

The order of creating an array of structure pointers is to first create the structure, then wrap it into a pointer, and finally create the array, and use the structure pointer to instantiate the array.

# Array of structure pointers
#Create structure array type
stu_array = Student * 2
# # Instantiate the structure array with an object of the Student class
stu_array_obj = stu_array(stu_obj, stu_obj)
#Create an array of structure pointers
stu_p_array = POINTER(Student) * 2
# Initialize using pointer()
stu_p_array_obj = stu_p_array(pointer(stu_obj), pointer(stu_obj))
# Once connected to the structure pointer member, note that the type initialized pointer is POINTER()
class NestStudent(Structure):
    _fields_ = [("rank", c_char),
                ("nest_stu", Student),
                ("strct_array", Student * 2),
                ("strct_point", POINTER(Student)),
                ("strct_point_array", POINTER(Student) * 2)]

# Instantiate, use pointer() to wrap the Student object as a pointer
nest_stu_info_list = [c_char(b"M"), stu_obj, stu_array_obj, pointer(stu_obj), stu_p_array_obj]
nest_stu_obj = NestStudent(*nest_stu_info_list)

#The second index of the array is the structure pointer
print(nest_stu_obj.strct_point_array[1])
#The pointer points to the Student object
print(nest_stu_obj.strct_point_array[1].contents)
#grade field of Student object
print(nest_stu_obj.strct_point_array[1].contents.grade)

Output:

<__main__.LP_Student object at 0x7f3f9a8e6200>
<__main__.Student object at 0x7f3f9a8e6290>
90

How ctypes loads DLL

The ctypes library provides several ways to load dynamic link libraries (DLLs). The following are commonly used methods:

1. Use the `CDLL` class

Use the ctypes.CDLL class to load dynamic link libraries, which is the most common way.

import ctypes

#Load DLL
mylibrary = ctypes.CDLL('mylibrary.dll')

# Call functions in DLL
result = mylibrary.Add(3, 4)
print(result)

2. Use the `WinDLL` class (Windows specific)

On Windows systems, you can use the ctypes.WinDLL class to load DLLs. It is similar to ctypes.CDLL, but uses the stdcall calling convention.

import ctypes

#Load DLL
mylibrary = ctypes.WinDLL('mylibrary.dll')

# Call functions in DLL
result = mylibrary.Add(3, 4)
print(result)

3. Use the `cdll.LoadLibrary` method

You can use the cdll.LoadLibrary method in the ctypes library to load the dynamic link library:

from ctypes import cdll

#Load DLL
mylibrary = cdll.LoadLibrary('mylibrary.dll')

# Call functions in DLL
result = mylibrary.Add(3, 4)
print(result)

4. Use absolute paths

If the DLL file is not in the Python script’s current working directory, you can load it using an absolute path:

from ctypes import cdll
import os

# Get the absolute path of the DLL file
dll_path = os.path.abspath('mylibrary.dll')

#Load DLL
mylibrary = cdll.LoadLibrary(dll_path)

# Call functions in DLL
result = mylibrary.Add(3, 4)
print(result)

5. Use `os.add_dll_directory` (Python 3.8+)

If you are using Python 3.8 and above, you can use os.add_dll_directory to add the directory containing the DLL to the system path:

import os
from ctypes import cdll

# Add the directory containing the DLL to the system path
os.add_dll_directory(r'C:\path\to\dll\directory')

#Load DLL
mylibrary = cdll.LoadLibrary('mylibrary.dll')

# Call functions in DLL
result = mylibrary.Add(3, 4)
print(result)

This method can avoid some path problems that may be encountered when loading the DLL.

Please make sure to replace mylibrary.dll in the example with the filename of the DLL you actually want to load.

Compile your own python interpreter

Compiling the modified CPython source code on Windows platforms can be accomplished using the Microsoft Visual Studio compiler and some accompanying tools. Here are the detailed steps:

Install required software:
- Install Microsoft Visual Studio. It is recommended to install a full version, including C++ development tools.
- Install Git for cloning CPython repositories from GitHub.
Get source code:

Open a command prompt or PowerShell and run the following command to clone the CPython repository:
```
git clone https://github.com/python/cpython.git
```
Install Windows SDK:

In Visual Studio, use the Visual Studio Installer to install the “Desktop development with C++” workload and include the Windows 10 SDK.
Open Visual Studio:

Open Visual Studio and open the cpython\PCbuild\pcbuild.sln solution file.
Make edits:

Make your changes in Visual Studio.
Download the external dependencies required to compile cpython

Run PCbuild/get_externals.bat from the command line

After completion, there will be an externals folder in cpython, which contains the external dependencies needed to compile cpython.
Build:

Open pcbuild.sln and enter Visual Studio to compile. Compilation platforms can be selected from Win32, x64, ARM and ARM64. In addition to the ordinary debug and release, the compilation modes also include PGInstrument and PGUpdate modes.

In Visual Studio, select the Release or Debug configuration and press Ctrl + Shift + B or select Build > Solution Build ” to build the code.

If you configure your build using Debug in Visual Studio, you will be able to debug in Visual Studio during the build process.

PGInstrument and PGUpdate add PGO (Profile Guided Optimization) optimization in release mode. This compilation mode requires the Premium version of Visual Studio. The python downloaded from the python official website is compiled in PGO optimization mode.

Compiling CPython in Visual Studio is very fast. In debug + win32 mode, it can be compiled in half a minute on a laptop. After successful compilation, python_d.exe and python312_d.dll will be generated in the PCbuild/win32 path. 312 in the file name is the version number. This is the python compiled from the source code, which can be run by double-clicking python_d.exe (the suffix _d indicates that CPython was compiled in debug mode.)

The core functions of python are provided by python3.dll, and python.exe just acts as an entry point. python.exe just sets a shell for python3.dll and passes the command line parameters argc argv to the Py_Main function.

Directory

.so and .dll files

Compile go code into a dynamic link library

Compile C code into a dynamic link library

ctypes library

Basic data types

How to use

Basic data types

Array type

Pointer type

Structure type

Nested structure

Structure array

Structure pointer

Array of structure pointers

How ctypes loads DLL

1. Use the CDLL class

2. Use the WinDLL class (Windows specific)

3. Use the cdll.LoadLibrary method

4. Use absolute paths

5. Use os.add_dll_directory (Python 3.8+)

Compile your own python interpreter

1. Use the `CDLL` class

2. Use the `WinDLL` class (Windows specific)

3. Use the `cdll.LoadLibrary` method

5. Use `os.add_dll_directory` (Python 3.8+)