Collect and follow to avoid getting lost
Article directory
- Preface
- 1. Project introduction
- 2. Development environment
- 3. Function introduction
- 4. Core code
- 5. Effect drawing
- 6. Article Table of Contents
Foreword
In the past, whether you were browsing books or manually searching for information from the Internet through mobile phones, computers, etc., your field of vision was limited, the information surface was too narrow, and the amount of data was large and messy. The update speed of explosive information was fast and irregular. . To manually obtain massive amounts of information and analyze and sort it out, it takes a lot of time, energy, and is inefficient. However, through web crawlers, we can obtain weather data from a certain place or multiple places according to needs, clean the data, and store it in tables. Data visualization and feedback of analysis results to users. The main purpose of this topic is to design a web crawler program for directional systems. It also needs to meet different performance requirements and involves details and application aspects of directional web crawlers. In order to adapt to different needs, web crawlers can crawl specific topics based on preset topics. Study the principles of web crawlers and implement crawler-related functions, clean the crawled data and store it in the database for later visual display. Use Python’s specialized data visualization library pyecharts for visual display, so you need to select data sources with more data dimensions for collection and crawling. It mainly crawls weather data from the China Weather Network to obtain a large amount of weather and temperature data. Through data analysis, different renderings are displayed, which can analyze temperature changes in real time and help agricultural production reduce food production losses caused by sudden weather changes. To improve the economic benefits of farmers, it is imperative that we develop this system for crawling weather data and conducting data analysis.
Keywords: web crawler: pyecharts: weather data
1. Project introduction
Weather Forecast Life Planning System is an application based on weather forecast data. The system is designed to combine weather information with users’ personal preferences and daily itineraries to provide users with intelligent life planning suggestions based on different weather conditions. Its main research contents include the following aspects:
(1) Weather forecast data analysis: The system needs to collect, organize and analyze weather forecast data. Through the analysis of historical weather data and prediction of future weather trends, it can help users better understand the weather conditions of the day and the changing trends in the next few days.
(2) User demand survey: Through questionnaires, user interviews, etc., investigate users’ expectations on the impact of weather on daily life, daily travel habits, and life planning suggestions. This determines the factors and details that need to be considered for the system.
(3) Life planning algorithm design: Based on weather forecast data and user needs, design a life planning algorithm model suitable for different weather conditions, and continuously optimize the model through machine learning and other technologies to further improve the accuracy and practicality of life planning suggestions.
(4) System interface design: In order to enable users to better use the system, it is necessary to design a user-friendly interface and provide a good interactive experience. At the same time, the compatibility of different platforms and devices also needs to be considered to ensure that users can easily use the system on various terminals.
Through the integration of the above research content, the weather forecast life planning system can provide users with more personalized and practical life planning suggestions, helping users better cope with weather changes and improve the quality of life.
2. Development environment
Development language: Python
- python web crawler
- Data cleaning and formatting preprocessing (dirty data removal, data standardization, format conversion, etc.)
- Data analysis and processing of pandas and numpy
- Flask builds background services, bootstrap + echarts front-end rendering visualization
The framework includes, flask back-end framework, front-end bootstrap, echarts, JavaScript, data analysis uses pandas, numpy
—————-
3. Function introduction
1. System registration and login
2. Historical weather query for popular cities
The system provides weather data for popular cities (more than 30 such as Beijing, Shanghai, Tianjin, Chongqing, Guiyang, Haikou, etc.) according to year and month.
3. Statistical analysis of weather conditions in popular cities
Through statistical analysis of temperature changes, weather conditions, wind speed and direction and other indicators in popular cities in recent years, an overall understanding of urban weather changes can be achieved.
4. Temperature changes in popular cities
By analyzing the temperature changes in popular cities in recent years, including the minimum temperature and maximum temperature, it can be found that the temperature is basically increasing, which also reflects the seriousness of the global climate warming problem.
5. City temperature annual calendar heat map
Calculate the average daily temperature of a popular city in a certain year, and use the calendar component of echarts to draw the calendar’s temperature heat map.
6. Plan your daily life according to the weather.
Planning our daily routines according to the weather can help us better adapt to various weather conditions and improve the quality and efficiency of our lives. Reasonable arrangements need to be made according to local climate conditions and personal habits to ensure physical and mental health and quality of life. At the same time, it can also be combined with the python weather forecast life planning system to make daily life plans more accurately based on weather forecast data.
7. Recommend travel suggestions based on the weather.
Recommending travel suggestions based on the weather can help us better avoid the impact of bad weather on travel and improve the safety and comfort of the journey. It is necessary to arrange travel plans reasonably according to specific weather conditions and personal habits, and stay alert to ensure safety and comfort during the journey. At the same time, it can also be combined with the python weather forecast life planning system to make travel plans more accurately based on weather forecast data.
System architecture design is the process of dividing the final goal into multiple small tasks. After completing one small task after another, the final goal can be assembled and completed. During the entire system conception process, the most appropriate one is selected from several feasible solutions to achieve the goal of each small function. For each small function goal, some functional-related requirements must be searched before development. Analyze the information, then start the preliminary design of the system, and gradually optimize the functions, and finally design a clear and implementable system structure [8].
The system structure design of this system is shown in Figure 3-1.
Figure 3-1 System structure diagram
4. Core code
Part of the code:
def users_login(request):
if request.method in ["POST", "GET"]:
msg = {<!-- -->'code': normal_code, "msg": mes.normal_code}
req_dict = request.session.get("req_dict")
if req_dict.get('role')!=None:
del req_dict['role']
datas = users.getbyparams(users, users, req_dict)
if not datas:
msg['code'] = password_error_code
msg['msg'] = mes.password_error_code
return JsonResponse(msg)
req_dict['id'] = datas[0].get('id')
return Auth.authenticate(Auth, users, req_dict)
def users_register(request):
if request.method in ["POST", "GET"]:
msg = {<!-- -->'code': normal_code, "msg": mes.normal_code}
req_dict = request.session.get("req_dict")
error = users.createbyreq(users, users, req_dict)
if error != None:
msg['code'] = crud_error_code
msg['msg'] = error
return JsonResponse(msg)
def users_session(request):
'''
'''
if request.method in ["POST", "GET"]:
msg = {<!-- -->"code": normal_code,"msg":mes.normal_code, "data": {<!-- -->}}
req_dict = {<!-- -->"id": request.session.get('params').get("id")}
msg['data'] = users.getbyparams(users, users, req_dict)[0]
return JsonResponse(msg)
def users_logout(request):
if request.method in ["POST", "GET"]:
msg = {<!-- -->
"msg": "Exit successfully",
"code": 0
}
return JsonResponse(msg)
def users_page(request):
'''
'''
if request.method in ["POST", "GET"]:
msg = {<!-- -->"code": normal_code, "msg": mes.normal_code,
"data": {<!-- -->"currPage": 1, "totalPage": 1, "total": 1, "pageSize": 10, "list\ ": []}}
req_dict = request.session.get("req_dict")
tablename = request.session.get("tablename")
try:
__hasMessage__ = users.__hasMessage__
except:
__hasMessage__ = None
if __hasMessage__ and __hasMessage__ != "No":
if tablename != "users":
req_dict["userid"] = request.session.get("params").get("id")
if tablename == "users":
msg['data']['list'], msg['data']['currPage'], msg['data']['totalPage'], msg[ 'data']['total'], \
msg['data']['pageSize'] = users.page(users, users, req_dict)
else:
msg['data']['list'], msg['data']['currPage'], msg['data']['totalPage'], msg[ 'data']['total'], \
msg['data']['pageSize'] = [],1,0,0,10
return JsonResponse(msg)
5. Renderings
6. Article directory
Table of contents
Abstract I
Abstract II
1 Introduction 1
1.1 Research background and development significance 1
1.2 Domestic and foreign research 2
1.2.1 Domestic research 2
1.2.2 Foreign research 2
1.3 Research content 2
2 System Analysis 4
2.1 Technical Analysis 4
2.2.1 Model analysis of web crawlers 4
2.2.2 Development environment 5
2.2.3 Environment installation 6
2.2.4 Python technology 6
2.2.5 pycharm software 6
2.2.6 Flask framework 7
2.2 Functional analysis 7
2.4 Performance Analysis 7
2.5 Feasibility analysis 8
2.5.1 Economic feasibility analysis 8
2.5.2 Technical feasibility analysis 8
2.5.3 Operation feasibility analysis 8
3 System Design 9
3.1 System structure design 9
3.2 System business design 10
3.2.1 Business process 10
3.3 Clustering algorithm design 11
3.4 Database design 12
4 System implementation 13
4.1 Implementation of historical weather query 13
4.2 Implementation of annual weather change analysis 13
4.3 Implementation of analysis of annual changes in air quality 14
5 System Test 16
5.1 Test purpose 16
5.2 Functional testing 16
5.3 Non-functional testing 17
5.4 Test summary 17
6 Summary and Outlook 19
Reference 20
Acknowledgments 21