Python lightweight serialization and deserialization package marshmallow detailed usage guide 3

Marshmallow

Official documentation: https://marshmallow.readthedocs.io/en/latest/

Marshmallow, Chinese translation: cotton candy. It is a lightweight data format conversion module, also called serialization and deserialization module, which is often used to convert between complex orm model objects and python native data types. Marshmallow provides rich API functions. as follows:

  1. Serializing

    Serialization [can convert data objects into storable or transferable data types, for example: objects/object->list/dict, dict/list->string]

  2. Deserializing

    Deserializer [converts storable or transferable data types into data objects, for example: list/dict->objects/object, string->dict/list]

  3. Validation

    Data verification, you can perform type verification or custom verification on the content of the data to be converted during the deserialization stage.

Marshmallow itself is a separate library, based on the framework of our current project is flask and the database ORM framework uses SQLAlchemy, so we can integrate it into the project by installing flask-sqlalchemy and marshmallow-sqlalchemy.

Basic installation and configuration

Module installation:

pip install -U marshmallow-sqlalchemy
pip install -U flask-sqlalchemy
pip install -U flask-marshmallow

Module initialization:

import os

from flask import Flask
from flask_script import Manager
from flask_sqlalchemy import SQLAlchemy
from flask_redis import FlaskRedis
from flask_session import Session
from flask_migrate import Migrate,MigrateCommand
from flask_jsonrpc import JSONRPC
from flask_marshmallow import Marshmallow

from application.utils import init_blueprint
from application.utils.config import load_config
from application.utils.session import init_session
from application.utils.logger import Log
from application.utils.commands import load_command
# Create terminal script management object
manager = Manager()

# Create database link object
db = SQLAlchemy()

# redis link object
redis = FlaskRedis()

# Session storage object
session_store = Session()

# data migration instance object
migrate = Migrate()

# log object
log = Log()

# jsonrpc module instance object
jsonrpc = JSONRPC()

# Object creation for data converters
ma = Marshmallow()

def init_app(config_path):
    """Global initialization"""
    # create app application object
    app = Flask(__name__)
    # Project root directory
    app.BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))

    # load configuration
    Config = load_config(config_path)
    app.config.from_object(Config)

    # Database initialization
    db.init_app(app)
    redis.init_app(app)

    # Initialization of the data converter
    ma.init_app(app)

    # session storage initialization
    init_session(app)
    session_store.init_app(app)

    # Data migration initialization
    migrate.init_app(app, db)
    # Add data migration commands to the terminal script tool
    manager.add_command('db', MigrateCommand)

    # log initialization
    app.log = log.init_app(app)

    # Blueprint Registration
    init_blueprint(app)

    # jsonrpc initialization
    jsonrpc.service_url = "/api" # URL address prefix of api interface
    jsonrpc.init_app(app)

    # Initialize terminal script tool
    manager.app = app

    # Register custom commands
    load_command(manager)

    return manager

Basic constructor (Schema)

The marshmallow conversion data format is mainly done through the constructor class, while the Schema class provides the basic functions of data conversion: serialization, verification and deserialization. So in the process of using marshmallow, all constructor classes must directly or indirectly inherit from the Schema base class

Complete data serialization conversion based on Schema

application.apps.marsh.viewsCode:

from marshmallow import Schema,fields
from application.apps.users.models import User,UserProfile
class UserSchema(Schema): # Basic constructor (Schema)
    '''Conversion of user information has nothing to do with the model fields set field type '''
    name = fields. String()
    age = fields. Integer()
    email = fields. Email()
    money = fields. Number()
    classMeta:
        fields = ["name","age","money","email","info"] # ('Fields that need to be serialized')
        ordered = True # Convert to an ordered dictionary
def index():
    """Serialization"""
    """Serialization processing of single model data"""
    user1 = User( name="xiaoming", password="123456", age=16, email="[email protected]", money=31.50 )
    # print(user1)
    # Convert the model object to dictionary format
    data1 = UserSchema(). dump(user1)
    print(type(data1),data1)
    
    # Convert the model object to json string format
    data2 = UserSchema().dumps(user1) # dumps serializes any object into a str
    print(type(data2), data2)
    return "ok"

Schema common attribute data type

Type Description
fields.Dict(keys, type]] = None, values, …) dictionary type, often used to receive json type data
fields. List(cls_or_instance, type], **kwargs) List type, often used to receive array data
fields.Tuple (tuple_fields, *args, **kwargs) tuple type
fields.String(* , default, missing, data_key, …) String type
fields.UUID(*, default, missing, data_key, …) UUID format type string
fields.Number(*, as_string, **kwargs) Numeric basic types
fields.Integer(*, strict, **kwargs) Integer
fields.Decimal(places, rounding, *, allow_nan, …) numeric
fields.Boolean(*, truthy, falsy, **kwargs) Boolean
fields.Float(*, allow_nan, as_string, **kwargs) floating point type
fields .DateTime(format, **kwargs) DateTimeType
fields.Time (format, **kwargs) Time type
fields.Date(format, **kwargs) DateType
fields.Url(*, relative, schemes, Set[str]]] = None, …) url URL string type
fields.Email(*args, **kwargs) Email String type
fields.IP(*args[, exploded]) IP address string type
fields.IPv4(*args[, exploded]) IPv4 address string type
fields.IPv6(*args[, exploded]) IPv6 address string type
fields .Method(serialize, deserialize, **kwargs) Fields based on Schema class method return values
fields.Function(serialize, Any], Callable[[Any, …) Returns value fields based on function
fields.Nested(nested, type, str, Callable[[], …) Foreign key type

Common common properties of Schema data types

Attribute name Description
default Set the default value of the field in the serialization phase
missing Set the default value of the field in the deserialization phase
validate The built-in data validator or built-in validation set called by the deserialization phase
required Set the required fields of the current field
allow_none Whether it is allowed to be empty
load_only Whether the current field is used in the deserialization phase
dump_omly Is the current field used in the serialization phase
dump_omly Whether the current field is used in the serialization phase
error_messages dictionary type, which can be used to replace the default field exception prompt, format:
error_messages={“required”: “The username is required.”}

The serialization operation performed above belongs to the serialization of a single data object, and the serialization of multiple data objects can also be performed in MarshMallow.

from marshmallow import Schema,fields
from application.apps.users.models import User,UserProfile

class UserSchema(Schema):
    name = fields. String()
    age = fields. Integer()
    email = fields. Email()
    money = fields. Number()
    classMeta:
        fields = ["name","age","money","email","info"]
        ordered = True # Convert to an ordered dictionary

def index():
    """Serialization"""
    """Serialization of multiple model data"""
    user1 = User(name="xiaoming", password="123456", age=15, email="[email protected]", money=31.50)
    user2 = User(name="xiaohong", password="123456", age=16, email="[email protected]", money=31.50)
    user3 = User(name="xiaopang", password="123456", age=17, email="[email protected]", money=31.50)
    data_list = [user1, user2, user3]
    data1 = UserSchema(many=True).dumps(data_list) # dumps serializes any object into a str many=True allows multiple models to be serialized
    print(type(data1),data1)
      # <class 'str'> [{"name": "xiaoming", "age": 4, "money": 31.5, "email": "33@qq .com", "info": null}, {"name": "xiaohui", "age": 5, "money": 66.5, "email": " "[email protected]", "info": null}, {"name": "xiaobai", "age": 6, "money": 88.5, "email ": "[email protected]", "info": null}]

    return "ok"

Constructor nested use

from marshmallow import Schema,fields
from application.apps.users.models import User,UserProfile
class UserProfileSchema(Schema):
    education = fields. Integer()
    middle_school = fields. String()

class UserSchema(Schema):
    name = fields. String()
    age = fields. Integer()
    email = fields. Email()
    money = fields. Number()
    info = fields.Nested(UserProfileSchema,only=["middle_school"])
    # nested foreign key type only only displays a certain field
    classMeta:
        fields = ["name","age","money","email","info"]
        ordered = True # Convert to an ordered dictionary

def index():
    """Serialization"""
    """Serialization nested using """
    user1 = User(name="xiaoming", password="123456", age=15, email="[email protected]", money=31.50)
    user1.info = UserProfile(
        education=3,
        middle_school="qwq school"
    )
  data = UserSchema().dump(user1) #order sorted dictionary
    print(data) # OrderedDict([('name', 'xiaoming'), ('age', 15), ('money', 31.5), ('email', \ '[email protected]'), ('info', {'middle_school': 'qwqschool'})])
    data1 = UserSchema().dumps(user1) # object to string
    print(data1) # {"name": "xiaoming", "age": 15, "money": 31.5, "email": "[email protected]", "info": {"middle_school": "qwq\\学\\校"}}

    return "ok"

Complete data deserialization conversion based on Schema

code:

class UserSchema2(Schema):
    name = fields.String(required=True,error_messages={<!-- -->"required":"Must be filled in"}) # required=True must be passed parameters, if there is no error
    sex = fields. String()
    age = fields.Integer(missing=18) # set the default value when deserializing
    email = fields.Email(error_messages={<!-- -->'invalid':"Sorry, the email must be formatted correctly!"})
    mobile = fields. String()

    @post_load # Deserialize into an object, add post_load decorator to a method of schema
    def post_load(self, data, **kwargs):
        return User(**data)


def index():
    user_data = {<!-- -->"mobile":"1331345635", "email": "[email protected]","sex":"abc"}
    us2 = UserSchema2()
    result = us2.load(user_data,partial=True)
    # The default serializer must pass all required fields, otherwise a validation exception will be thrown. But we can use partial=True to avoid error
    print(result) # ==> <User xiaoming>
    return "ok"


Convert/ignore some data when deserializing

from marshmallow import Schema, fields, validate, ValidationError, post_load
class UserSchema2(Schema):
    name = fields. String()
    sex = fields. String()
    age = fields. Integer(missing=18)
    email = fields. Email()
    mobile = fields. String(required=True)

    @post_load
    def post_load(self, data, **kwargs):
        return User(**data)

def index():
    user_data = {<!-- -->"name": "xiaoming","sex":"abc"}
    us2 = UserSchema2()
    result = us2.load(user_data,partial=True)
    # The default serializer must pass all required fields, otherwise a validation exception will be thrown. But we can use the partial parameter to allow partial field updates

    print(result) # ==> <User xiaoming>
    return "ok"

The setting field is only enabled during the serialization or deserialization phase

class UserSchema2(Schema):
    name = fields. String()
    sex = fields.Integer(validate=validate.OneOf([0,1,2])) # Deserialization stage calls selection validation
    age = fields. Integer(missing=18)
    email = fields. Email()
    mobile = fields. String()
    password = fields.String(load_only=True) # Set the current field as a write-only field, "write-only" will only be enabled in the deserialization phase
    # password = fields.String(dump_only=True) # Set the current field as a write-only field, "read-only" will only be enabled in the serialization phase
    @post_load
    def post_load(self, data, **kwargs):
        return User(**data)

def index():
    user_data = {<!-- -->"name": "xiaoming","password":"123456","sex":1}
    us2 = UserSchema2()
    # deserialize
    result = us2. load(user_data)
    print(result) # ==> <User xiaoming>
    # Serialization
    us3 = UserSchema2(only=["sex","name","age","password"]) # Limit processing fields
    result2 = us3. dump(result)
    print(result2)
    return "ok"

Hook method in the deserialization phase

post_dump([fn, pass_many, pass_original]) Registers a method to be called after serializing the object, it will be called after the object is serialized.
post_load([fn, pass_many, pass_original]) Registers the method to be called after deserializing the object, which will be called after validating the data.
pre_dump([fn, pass_many]) Registers a method to be called before serializing the object, it will be called before serializing the object.
pre_load([fn, pass_many]) Before deserializing the object, register the method to be called, it will be called before validating the data.

from marshmallow import Schema, fields, validate, ValidationError, post_load, post_dump
class UserSchema2(Schema):
    name = fields. String()
    sex = fields.Integer(validate=validate.OneOf([0,1,2]))
    age = fields. Integer(missing=18)
    email = fields. Email()
    mobile = fields. String()
    password = fields.String(load_only=True) # Set the current field as a write-only field, which will only be enabled in the deserialization phase

    @post_load
    def post_load(self, data, **kwargs):
        return User(**data)

    @post_dump
    def post_dump(self, data, **kwargs):
        data["mobile"] = data["mobile"][:3] + "*****" + data["mobile"][-3:]
        return data

def index():
    user_data = {<!-- -->"name": "xiaoming","password":"123456","sex":1,"mobile":" 133123454656"}
    us2 = UserSchema2()
    # deserialize
    result = us2. load(user_data)
    print(result) # ==> <User xiaoming>
    # Serialization
    us3 = UserSchema2(only=["sex","name","age","mobile"]) # Limit the fields processed
    result2 = us3. dump(result)
    print(result2)
    return "ok"

Validation of data during deserialization

Data validation based on built-in validators

< /table>

code:

from marshmallow import Schema, fields, validate, ValidationError, post_load
class UserSchema3(Schema):
    name = fields. String(required=True)
    sex = fields.String(required=True,error_messages={<!-- -->"required":"Sorry, permission must be filled"})
    age = fields.Integer(missing=18,validate=validate.Range(min=18,max=40,error="Age must be between 18-40!")) # Limit the value range
    email = fields.Email(error_messages={<!-- -->"invalid":"Sorry, you must fill in the email format!"})
    mobile = fields.String(required=True, validate=validate.Regexp("^1[3-9]\d{9}$",error="The mobile phone number format is incorrect"),error_messages= {<!-- -->"Regexp":"The mobile phone format is incorrect"})

    @post_load
    def make_user_obj(self, data, **kwargs):
        return User(**data)

def index3():
    user_data = {<!-- -->"mobile":"1331345635","name": "xiaoming","age":40, "email": " [email protected]","sex":"abc"}
    us2 = UserSchema3()
    result = us2. load(user_data)
    result2 = us2. dumps(result)
    print(result)
    print(result2)
    return "ok"

Custom authentication method

from marshmallow import Schema, fields, validate, validates, ValidationError, post_load, validates_schema
class UserSchema4(Schema):
    name = fields. String(required=True)
    sex = fields.String(required=True,error_messages={<!-- -->"required":"Sorry, permission must be filled"})
    age = fields.Integer(missing=18,validate=validate.Range(min=18,max=40,error="Age must be between 18-40!")) # Limit the value range
    email = fields.Email(error_messages={<!-- -->"invalid":"Sorry, you must fill in the email format!"})
    mobile = fields.String(required=True, validate=validate.Regexp("^1[3-9]\d{9}$",error="The mobile phone number format is incorrect"),error_messages= {<!-- -->"Regexp":"The mobile phone format is incorrect"})
    password = fields. String(required=True, load_only=True)
    password2 = fields. String(required=True, allow_none=True)
    @post_load
    def make_user_obj(self, data, **kwargs):
        return User(**data)

    @validates("name")
    def validate_name(self, data, **kwargs):
        print("name=%s" % data)
        if data == "root":
            raise ValidationError({<!-- -->"Sorry, the root user is a super user! You do not have permission to register!"})

        # must have a return value
        return data

    @validates_schema
    def validate(self, data, **kwargs):
        print(data)
        if data["password"] != data["password2"]:
            raise ValidationError("The password and confirmation password must be the same!")

        data. pop("password2")
        return data

def index():
    user_data = {<!-- -->"password":"12345","password2":"123456","mobile":"13313345635","name\ ": "root1","age":40, "email": "[email protected]","sex":"abc"}
    us2 = UserSchema4()
    result = us2. load(user_data)
    print(result)
    return "ok"

Model constructor (ModelSchema)

Official documentation: https://github.com/marshmallow-code/marshmallow-sqlalchemy

? https://marshmallow-sqlalchemy.readthedocs.io/en/latest/

Note: flask_marshmallow has removed the two model constructor classes, ModelSchema and TableSchema, after version 0.12.0, and officially recommends the use of SQLAlchemyAutoSchema and SQLAlchemySchema. The usage of the two classes is similar.

create

class UserSchema(SQLAlchemyAutoSchema):
    classMeta:
        model = model class name # table = models.Album.__table__
        include_relationships = True # Whether to process foreign keys at the same time when outputting model objects
        include_fk = True # Whether the sequence stage also returns the primary key
        load_instance = True # In the deserialization phase, directly return the model object
        sql_session = db.session # database connection session object
        # fields= ["id","name"] # Started field list
        exclude = ["id","name"] # list of excluded fields

ModelSchema

Official documentation: https://github.com/marshmallow-code/marshmallow-sqlalchemy

? https://marshmallow-sqlalchemy.readthedocs.io/en/latest/

Note: flask_marshmallow has removed the two model constructor classes, ModelSchema and TableSchema, after version 0.12.0, and officially recommends the use of SQLAlchemyAutoSchema and SQLAlchemySchema. The usage of the two classes is similar.

code:

from marshmallow_sqlalchemy import SQLAlchemySchema,SQLAlchemyAutoSchema,auto_field
from marshmallow import Schema, fields
from application.apps.user.models import User, UserProfile, db
from marshmallow import Schema, fields, validate, ValidationError, post_load
class UserSchema5(SQLAlchemySchema):
    # The role of auto_field, set the type and option declaration of the current data field to automatically extract from the corresponding field in the model
    # name = auto_field()
    # Here, there is no username in the database at all, you need to declare the type of the current data dictionary and the field of the model from which the option declaration is extracted in the first parameter position
    username = auto_field("name",dump_only=True)
    # You can add or overwrite the original statement in the model on the basis of the original field
    created_time = auto_field(format="%Y-%m-%d") # format set time format
    # You can even declare some fields that are not models
    token = fields. String()
    classMeta:
        model = User
        fields = ["username","created_time","token"]

def index5():
    """Serialization processing of single model data"""
    from datetime import datetime
    user1 = User(
        name="xiaoming",
        password="123456",
        age=16,
        email="[email protected]",
        money=31.50,
        created_time = datetime.now(),
    )
    user1.token = "abc"
    # Convert the model object to dictionary format
    data1 = UserSchema5().dump(user1)
    print(type(data1),data1)
    # <class 'dict'> {'username': 'xiaoming', 'token': 'abc', 'created_time': '2020-12-02'}
    return "ok"

"""SQLAlchemySchema is much simpler to use than the above Schema, but it is still necessary to write all the small conversion fields uniformly before converting these fields
, if you don't want to write field information and copy directly from the model, you can also use SQLAlchemyAutoSchema. """
class UserSchema6(SQLAlchemyAutoSchema):
    token = fields. String()
    classMeta:
        model = User
        include_fk = False # enable foreign key relationship
        include_relationships = False # model relationship external attributes
        fields = ["name","created_time","info","token"] # If you want to replace all fields, just don't declare fields or exclude fields
        sql_session = db.session

def index():
    """Serialization processing of single model data"""
    from datetime import datetime
    user1 = User(
        name="xiaoming",
        password="123456",
        age=16,
        email="[email protected]",
        money=31.50,
        created_time = datetime.now(),
        info=UserProfile(position="teaching assistant")
    )
    # Convert the model object to dictionary format
    user1.token="abcccccc"
    data1 = UserSchema6().dump(user1)
    print(type(data1),data1)
    # class 'dict'> {'info': <UserProfile: None>, 'created_time': '2020-12-02T17:47:21.821940', 'token': 'abcccccc ', 'name': 'xiaoming'}
    return "ok"

syntaxbug.com © 2021 All Rights Reserved.
Built-in Validator Description
validate.Email (*, error) Mailbox validation
validate.Equal(comparable, *, error) Judge whether the values are equal
validate.Length(min, max, *, equal, error) value length/size validation
validate.OneOf(choices, labels, *, error) option validation
validate.Range([min, max]) Range validation
validate.Regexp(regex, bytes, Pattern][, flags]) regular validation
validate. URL(*, relative, schemes, Set[str]]] = None, …) Verify whether it is a URL