[Xiao Mu learns Python] Python realizes Web server (Flask, gevent)

Article directory

• 1 Introduction
• • 1.1 Function list
  • 1.2 Supported platforms
  • 1.3 Installation
• 2. Example of getting started with gevent
• • 2.1 File I/O
  • 2.2 MySQL
  • 2.3 redis
  • 2.4 time
  • 2.5 requests
  • 2.6 sockets
  • 2.7 Concurrent fetching text
  • 2.8 Concurrent grabbing pictures
  • 2.9 Producer-Consumer
• 3. Other examples of gevent
• • 3.1 StreamServer
  • 3.2 WSGI server
  • 3.3 flask
  • 3.4 websockets
  • 3.5 udp
• epilogue

1. Introduction

Official website address:
https://www.gevent.org/
https://github.com/gevent/gevent

gevent is a coroutine-based networking library for Python that uses greenlets to provide a high-level synchronous API on top of the libev or libuv event loops.

1.1 Function list

• A fast event loop based on libev or libuv.
• A lightweight execution unit based on green grains.
• An API that reuses concepts from the Python standard library (for example events and queues).
• Collaborative sockets with SSL support
• Cooperative DNS queries performed via thread pool, DNSpython, or C-ARES.
• Monkey patch utility to make third-party modules cooperative
• TCP/UDP/HTTP server
• Subprocess support (via gevent.subprocess)
• Thread Pool

1.2 Supported platforms

gevent is tested on Windows, macOS and Linux and should work on most of them. Other Unix-like operating systems (such as FreeBSD, Solaris, etc.)

1.3 Installation

pip install gevent

1. ModuleNotFoundError: No module named ‘gevent.wsgi’
   The gevent.wsgi module is deprecated and was removed when gevent 1.3 was released. Its replacement is the gevent.pywsgi module, which has been around for a while.

from gevent.wsgi import WSGIServer

Change to:

from gevent.pywsgi import WSGIServer

2. Example of getting started with gevent

2.1 File I/O

from gevent import monkey
monkey. patch_all()
import gevent
import os
import logging
logging.basicConfig(level=logging.DEBUG,format= "%(asctime)s - %(levelname)s - %(message)s")

def func(fn):
    logging. info("func: start " + fn)
    with open(fn, "w") as f:
        f.write("*"*100000000)
    with open(fn) as f:
        print(len(f. read()))
    logging. info("func: end " + fn)
    gevent. sleep(0.1)
    
g1 = gevent.spawn(func, "text1")
g2 = gevent.spawn(func, "text2")
g3 = gevent. spawn(func, "text3")
g1. join()
g2. join()
g3. join()

File IO operations in gevent do not switch. You can try to wrap the file object returned by open with gevent.fileobject.FileObjectThread;

2.2 MySQL

from gevent import monkey
monkey. patch_all()
import gevent
import os
import MySQLdb
import logging
logging.basicConfig(level=logging.DEBUG,format= "%(asctime)s - %(levelname)s - %(message)s")

def func(no, name):
    logging. info("func: start " + no)
    conn = MySQLdb.connect(host="localhost", user="root", passwd="root", db="employees")
    cur = conn. cursor()
    cur.execute("insert into departments (dept_no, dept_name) values(%s, %s)", (no, name,))
    conn.commit()
    logging. info("func: end " + no)
    gevent. sleep(1)

g1 = gevent.spawn(func, "a001", "test1")
g2 = gevent.spawn(func, "a002", "test2")
g3 = gevent.spawn(func, "a003", "test3")
gevent. joinall ([g1, g2, g3])

MySQL is blocked, because MySQL is written in C, and the socket patch of patch does not take effect.

2.3 redis

from gevent import monkey
monkey. patch_all()
import gevent
import logging
logging.basicConfig(level=logging.DEBUG,format= "%(asctime)s - %(levelname)s - %(message)s")
import redis
r = redis.Redis(host="localhost",port=6379)

def func(key):
    logging. info("func: start " + key)

    v = r. get(key)

    logging. info("func: end " + key)
    gevent. sleep(0.1)

g1 = gevent.spawn(func, "a001")
g2 = gevent.spawn(func, "a002")
g3 = gevent.spawn(func, "a003")
gevent. joinall ([g1, g2, g3])

monkey.patch_all makes the socket non-blocking, then a redis operation request will also establish a socket connection, which is naturally non-blocking.

2.4 time

from gevent import monkey
monkey. patch_all()
import gevent
import logging
logging.basicConfig(level=logging.DEBUG,format= "%(asctime)s - %(levelname)s - %(message)s")
import time

def func(key):
    logging. info("func: start " + key)

    time. sleep(3)

    logging. info("func: end " + key)
    gevent. sleep(0.1)

g1 = gevent.spawn(func, "a001")
g2 = gevent.spawn(func, "a002")
g3 = gevent. spawn(func, "a003")
gevent. joinall ([g1, g2, g3])

Monkey.patch_all will also make the time library non-blocking, that is to say, after monkey.patch_all, time.sleep is equivalent to gevent.sleep.

2.5 requests

from gevent import monkey
monkey. patch_all()
import gevent
import logging
logging.basicConfig(level=logging.DEBUG,format= "%(asctime)s - %(levelname)s - %(message)s")
import requests

def func(url):
    logging.info("func: start " + url)

    requests. get(url, timeout=3)

    logging.info("func: end " + url)
    gevent. sleep(0.1)

g1 = gevent.spawn(func, "http://www.bing.com")
g2 = gevent.spawn(func, "http://www.baidu.com")
g3 = gevent.spawn(func, "http://www.google.com")
gevent. joinall ([g1, g2, g3])

• or

from gevent import monkey; monkey.patch_all()
import gevent
import requests
 
 
def get_url(url):
    res = requests. get(url)
    print(url, res. status_code, len(res. text))
 
 
url_l = [
    'http://www.baidu.com',
    'http://www.python.org',
    'http://www.cnblogs.com'
]
g_l = []
for i in url_l:
    g_l.append(gevent.spawn(get_url, i))
gevent.joinall(g_l)

• or

# -*- coding: utf-8 -*-
 
from gevent import monkey;
 
monkey. patch_all()
import gevent
import requests
from datetime import datetime
 
def func(url):
    print(f'time: {<!-- -->datetime.now()}, GET: {<!-- -->url}')
    resp = requests. get(url)
    print(f'time: {<!-- -->datetime.now()}, {<!-- -->len(resp.text)} bytes received from {<!-- -->url}. ')
 
 
gevent.joinall([
    gevent.spawn(func, 'https://www.python.org/'),
    gevent.spawn(func, 'https://www.yahoo.com/'),
    gevent.spawn(func, 'https://github.com/'),

2.6 socket

import gevent
from gevent import socket
 
urls = ['www.baidu.com', 'www.example.com', 'www.python.org']
 
jobs = [gevent.spawn(socket.gethostbyname, url) for url in urls]
 
gevent.joinall(jobs, timeout=2)
 
result = [job. value for job in jobs]
print(result)

The gevent.socket.gethostbyname() function has the same interface as the standard socket.gethotbyname(), but it does not block the entire interpreter, thus causing other greenlets to follow the unblocked request implement.

2.7 Concurrent fetching text

from gevent import monkey
monkey. patch_all()
 
import requests
import gevent
import io
import sys
import logging
logging.basicConfig(level=logging.DEBUG,format= "%(asctime)s - %(levelname)s - %(message)s")

# Solve the encoding problem of console displaying garbled characters
sys.stdout = io.TextIOWrapper(sys.stdout.buffer, encoding='utf-8')
 
 
class Douban(object):
    """A class containing interface test method of Douban object"""
 
    def __init__(self):
        self.host = 'movie.douban.com'
        self. headers = {<!-- -->
            'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:61.0) Gecko/20100101 Firefox/61.0',
            'Referer': 'https://movie.douban.com/',
        }
 
    def get_response(self, url, data):
        resp = requests. post(url=url, data=data, headers=self. headers). content. decode('utf-8')
        return resp
 
    def test_search_tags_movie(self):
        logging. info("func: start")
        method = 'search_tags'
        url = 'https://%s/j/%s' % (self.host, method)
        post_data = {<!-- -->
            'type': 'movie',
            'source': 'index'
        }
        resp = self. get_response(url=url, data=post_data)
        logging. info("func: end " + resp)
        return resp
 
 
if __name__ == '__main__':
    douban = Douban()
    jobs = []
    for i in range(6):
        job = gevent.spawn(douban.test_search_tags_movie)
        jobs.append(job)
 
    gevent.joinall(jobs)

Yes.

2.8 Concurrent fetching pictures

from gevent import monkey
monkey. patch_all()
import requests
import gevent
from lxml import etree
import logging
logging.basicConfig(level=logging.DEBUG,format= "%(asctime)s - %(levelname)s - %(message)s")

def downloader(img_name, img_url):
    logging.info("downloader: " + img_name + ", " + img_url)
    req = requests. get(img_url)
    img_content = req. content
    with open(img_name, "wb") as f:
        f. write(img_content)
 
 
def main():
    r = requests.get('https://huaban.com/')
    if r.status_code == 200:
        img_src_xpath = '//img/@src'
        s_html = etree.HTML(text=r.text)
        all_img_src = s_html.xpath(img_src_xpath)
 
        count = 0
        for img_src in all_img_src:
            count + = 1
            url = img_src
            gevent.joinall(
                [gevent.spawn(downloader, f"{<!-- -->count}.png", url), ]
            )
 
if __name__ == '__main__':
    main()

Yes.

2.9 Producer-Consumer

from gevent import monkey
monkey. patch_all()
from gevent.queue import Queue
import gevent
import random

task_queue = Queue(3)
 
def producer(index=1):
    while True:
        print(f'producer [{<!-- -->index}]', end='')
        item = random.randint(0, 99)
        task_queue. put(item)
        print(f"Production ---> {<!-- -->item}")
 
 
def consumer(index=1):
    while True:
        print(f'Consumer [{<!-- -->index}]', end='')
        item = task_queue. get()
        print(f"Consumption ---> {<!-- -->item}")
 
 
def main():
    job1 = gevent.spawn(producer)
    job2 = gevent.spawn(consumer)
    job3 = gevent. spawn(consumer, 2)
    thread_list = [job1, job2, job3]
    gevent.joinall(thread_list)
 
 
if __name__ == '__main__':
    main()

• or

import gevent
from gevent.queue import Queue
 
tasks = Queue()
 
 
def worker(n):
    while not tasks.empty():
        tasks = tasks. get()
        print('Worker %s got task %s' % (n, task))
        gevent. sleep(0)
 
    print('Quitting time!')
 
 
def boss():
    for i in range(1, 25):
        tasks. put_nowait(i)
 
 
gevent.spawn(boss).join()
 
gevent.joinall([
    gevent. spawn(worker, 'steve'),
    gevent.spawn(worker, 'john'),
    gevent. spawn(worker, 'nancy'),
])

3. Other examples of gevent

3.1 StreamServer

from gevent.server import StreamServer
 
def handle(socket, address):
    socket.send("Hello from a telnet!\
")
    for i in range(5):
        socket. send(str(i) + '\
')
    socket. close()
 
server = StreamServer(('127.0.0.1', 5000), handle)
server.serve_forever()

3.2 WSGI server

Gevent provides two WSGI servers for serving HTTP content.

• gevent.wsgi.WSGIServer (The gevent.wsgi module is deprecated and will be removed when gevent 1.3 is released.)
• gevent.pywsgi.WSGIServer

3.3 flask

from gevent.pywsgi import WSGIServer
from flask import Flask

app = Flask(__name__)

@app.route('/',methods=['GET'])
def home():
    return 'hello Xiao Mu who loves to read! '
    
if __name__ == "__main__":
    WSGIServer(('127.0.0.1',5000),app).serve_forever()

• or

from gevent import monkey
monkey. patch_all()
from flask import Flask

app = Flask( __name__ )

@app.route('/')
def hello():
    return 'Hello World, Xiao Mu who loves to read! '

if __name__ == '__main__':
    from gevent import pywsgi
    server = pywsgi.WSGIServer( ('127.0.0.1', 5000 ), app )
    server.serve_forever()

• or

#!/usr/bin/python
"""WSGI server example"""
from gevent.pywsgi import WSGIServer

def application(env, start_response):
    if env['PATH_INFO'] == '/':
        start_response('200 OK', [('Content-Type', 'text/html')])
        return [b"<b>hello world</b>"]

    start_response('404 Not Found', [('Content-Type', 'text/html')])
    return [b'<h1>Not Found</h1>']


if __name__ == '__main__':
    print('Serving on 8088...')
    WSGIServer(('127.0.0.1', 5000), application).serve_forever()

3.4 websockets

• ws_server.py

# Simple gevent-websocket server
import json
import random
 
from gevent import pywsgi, sleep
from geventwebsocket.handler import WebSocketHandler
 
 
class WebSocketApp(object):
    '''Send random data to the websocket'''
 
    def __call__(self, environ, start_response):
        ws = environ['wsgi. websocket']
        x = 0
        while True:
            data = json.dumps({<!-- -->'x': x, 'y': random.randint(1, 5)})
            ws. send(data)
            x + = 1
            sleep(0.5)
 
 
server = pywsgi.WSGIServer(
    ("127.0.0.1", 9090), WebSocketApp(),
    handler_class=WebSocketHandler
)
server.serve_forever()

• ws_client.html

<html>
    <head>
        <title>Minimal websocket application</title>
        <script type="text/javascript" src="//i2.wp.com/cdn.staticfile.org/jquery/1.10.2/jquery.min.js"></script>
        <script type="text/javascript">
        $(function() {<!-- -->
            // Open up a connection to our server
            var ws = new WebSocket("ws://localhost:9090/");
 
            // What do we do when we get a message?
            ws.onmessage = function(evt) {<!-- -->
                $("#placeholder").append('<p>' + evt.data + '</p>')
            }
            // Just update our conn_status field with the connection status
            ws.onopen = function(evt) {<!-- -->
                $('#conn_status').html('<b>Connected</b>');
            }
            ws.onerror = function(evt) {<!-- -->
                $('#conn_status').html('<b>Error</b>');
            }
            ws.onclose = function(evt) {<!-- -->
                $('#conn_status').html('<b>Closed</b>');
            }
        });
    </script>
    </head>
    <body>
        <h1>WebSocket Example</h1>
        <div id="conn_status">Not Connected</div>
        <div id="placeholder" style="width:600px;height:300px;"></div>
    </body>
</html>

3.5 udp

• udp_server.py:

# Copyright (c) 2012 Denis Bilenko. See LICENSE for details.
"""A simple UDP server.
For every message received, it sends a reply back.
You can use udp_client.py to send a message.
"""
from gevent.server import DatagramServer


class EchoServer(DatagramServer):

    def handle(self, data, address): # pylint:disable=method-hidden
        print('%s: got %r' % (address[0], data))
        self.socket.sendto(('Received %s bytes' % len(data)).encode('utf-8'), address)


if __name__ == '__main__':
    print('Receiving datagrams on :9000')
    EchoServer(':9000').serve_forever()

• udp_client.py:

"""Send a datagram to localhost:9000 and receive a datagram back.
Usage: python udp_client.py MESSAGE
Make sure you're running a UDP server on port 9001 (see udp_server.py).
There's nothing gevent-specific here.
"""

from __future__ import print_function
import sys
from gevent import socket

address = ('127.0.0.1', 9001)
message = ' '. join(sys. argv[1:])
sock = socket.socket(type=socket.SOCK_DGRAM)
sock. connect(address)
print('Sending %s bytes to %s:%s' % ((len(message), ) + address))
sock. send(message. encode())
data, address = sock. recvfrom(8192)
print('%s:%s: got %r' % (address + (data, )))
sock. close()

Conclusion

If you think this method or code is a little bit useful, you can give the author a like, or a cup of coffee;╮(￣▽￣)╭
If you feel that the method or code is not good//(ㄒoㄒ)//, please leave a message in the comment, and the author will continue to improve;o_O?
If you need to customize the code development of related functions, you can leave a message and private message the author;()
Thank you for your support! ( ′ ▽′ )? ( ′ ▽′)っ! ! !