Python’s Pandas Library (Python Data Analysis Library) is a powerful assistant for data scientists and analysts. It provides powerful data processing and analysis tools, making data import, cleaning, conversion and analysis more efficient and convenient.
This article will provide an in-depth introduction to the various functions and usage of the Pandas library, including basic operations of DataFrame and Series, data cleaning, data analysis and visualization, etc.
1. Introduction to Pandas
Pandas is one of the most popular data analysis libraries in Python, created in 2008 by Wes McKinney. Its name comes from the abbreviation of “Panel Data”. The main data structures of Pandas include DataFrame and Series:
- DataFrame: Similar to a spreadsheet or SQL table, it is a two-dimensional data structure with rows and columns. Each column can contain different types of data (integers, floats, strings, etc.).
- Series: is a one-dimensional data structure, similar to an array or list, but has labels and can be indexed by labels.
Features of Pandas include:
- Data alignment: Pandas can automatically align data with different indexes, making data operations more convenient.
- Handling missing values: Pandas provides powerful tools to handle missing values, including deletion, filling and other operations.
- Powerful data analysis functions: Pandas supports various data analysis and statistical calculations, such as mean, median, standard deviation, etc.
- Flexible data import and export: Pandas can read and write multiple data formats, including CSV, Excel, SQL databases, JSON, and more.
- Data cleaning and transformation: Pandas provides a wealth of data cleaning and transformation functions for data preprocessing and organization.
Next, we will delve into various aspects of the Pandas library.
2. Basic operations of Pandas
1. Install and import Pandas
First, make sure you have the Pandas library installed. If it is not installed, you can install it using the following command:
Copy code pip install pandas
Once installed, Pandas can be imported into Python:
javascript Copy code import pandas as pd
2. Create DataFrame
Creating a DataFrame is the first step in data analysis. DataFrames can be created in a variety of ways, including from dictionaries, CSV files, Excel files, SQL databases, and more.
2.1 Create DataFrame from dictionary
ini
Copy code
data = {'Name': ['Alice', 'Bob', 'Charlie'],
'Age': [25, 30, 35]}
df = pd.DataFrame(data)
print(df)
This will create a DataFrame containing names and ages, with each column being a Series object.
2.2 Import DataFrame from CSV file
ini
Copy code
df = pd.read_csv('data.csv')
The above code will import data from a CSV file named ‘data.csv’ and store it as a DataFrame object.
3. View and process data
Once you have a DataFrame, you can start viewing and processing the data. Here are some commonly used operations:
3.1 View the first few rows of data
bash Copy code print(df.head()) # Display the first 5 rows of data by default
3.2 View basic information of data
bash Copy code print(df.info()) # Display basic information of data, including column name, data type, number of non-null values, etc.
3.3 View statistical summary
bash Copy code print(df.describe()) # Display the statistical summary of the data, including mean, standard deviation, minimum value, maximum value, etc.
3.4 Select columns
ini Copy code ages = df['Age'] # Select the column named 'Age' and return a Series object
3.5 Select rows
ini Copy code row = df.loc[0] # Select the first row and return a Series object
3.6
Conditional filtering
bash Copy code young_people = df[df['Age'] < 30] # Filter rows whose age is less than 30 years old
4. Data cleaning
Data cleaning is an important step in data analysis, including handling missing values, duplicates, and outliers.
4.1 Handling missing values
bash Copy code # Delete rows containing missing values df.dropna() # Fill missing values with specified values df.fillna(0)
4.2 Handling duplicates
bash Copy code df.drop_duplicates() # Delete duplicate rows
4.3 Handling outliers
bash Copy code # Select rows with ages between 0 and 100 df[(df['Age'] >= 0) & amp; (df['Age'] <= 100)]
3. Data analysis and statistics
Pandas provides rich data analysis and statistical calculation functions, making data exploration and analysis easy.
1. Statistics
1.1 Calculate average
ini Copy code average_age = df['Age'].mean()
1.2 Calculate the median
ini Copy code median_age = df['Age'].median()
1.3 Calculate standard deviation
ini Copy code std_age = df['Age'].std()
2. Data grouping
2.1 Group statistics
ini
Copy code
#Group by gender and calculate the average age of each group
gender_group = df.groupby('Gender')
average_age_by_gender = gender_group['Age'].mean()
2.2 Pivot table
ini Copy code # Create a pivot table to calculate the average salary for each gender and occupation combination pivot_table = pd.pivot_table(df, values='Salary', index='Gender', columns='Occupation', aggfunc=np.mean)
3. Data visualization
Pandas can be used in conjunction with visualization libraries such as Matplotlib and Seaborn for data visualization.
3.1 Draw a line chart
python
Copy code
import matplotlib.pyplot as plt
# Draw age line chart
plt.plot(df['Age'])
plt.xlabel('Sample number')
plt.ylabel('age')
plt.title('Age Distribution')
plt.show()
3.2 Draw histogram
bash
Copy code
# Draw age histogram
plt.hist(df['Age'], bins=10)
plt.xlabel('age')
plt.ylabel('Sample number')
plt.title('Age distribution histogram')
plt.show()
3.3 Draw box plot
kotlin
Copy code
import seaborn as sns
# Draw a boxplot of age
sns.boxplot(x='Age', data=df)
plt.title('Age distribution box plot')
plt.show()
4. Advanced skills in data processing
1. Data merging and connection
Pandas can be used to merge and join multiple data sets. Common methods include concat, merge, join, etc.
1.1 Use concat to merge
ini Copy code # Merge two DataFrames along the row direction combined_df = pd.concat([df1, df2], axis=0) # Merge two DataFrames along the column direction combined_df = pd.concat([df1, df2], axis=1)
1.2 Use merge connection
ini Copy code # Join two DataFrames using common columns merged_df = pd.merge(df1, df2, on='ID', how='inner')
2. Data reshaping
Pandas provides a variety of methods to reshape data, including pivot, melt, stack/unstack, etc.
2.1 Use pivot for data pivot
ini Copy code # Create a pivot table to calculate the average salary for each gender and occupation combination pivot_table = pd.pivot_table(df, values='Salary', index='Gender', columns='Occupation', aggfunc=np.mean)
2.2 Use melt for data fusion
ini Copy code # Convert wide format data to long format data melted_df = pd.melt(df, id_vars=['Name'], value_vars=['Math', 'Physics', 'Chemistry'], var_name='Subject', value_name='Score')
3. Time series analysis
Pandas is also very powerful in processing time series data, and can parse timestamps, perform time resampling, calculate rolling statistics, etc.
3.1 Parsing timestamps
bash Copy code df['Timestamp'] = pd.to_datetime(df['Timestamp'])
3.2 Time Resampling
ini
Copy code
# Resample the time series data by week and calculate the weekly average
weekly_mean = df.resample('W', on='Timestamp').mean()
Summary
Pandas is an indispensable data analysis tool in Python. It provides rich data processing, cleaning, analysis and visualization functions, making it easier for data scientists and analysts to explore and understand data.
Now, Pandas is still developing, and more functions and performance optimizations will be introduced to meet the growing needs of data analysis. Mastering Pandas is an important step to improve data processing efficiency.
Digression
In this era of rapidly growing technology, programming is like a ticket to a world of infinite possibilities for many people. Among the star lineup of programming languages, Python is like the dominant superstar. With its concise and easy-to-understand syntax and powerful functions, Python stands out and becomes one of the hottest programming languages in the world.
The rapid rise of Python is extremely beneficial to the entire industry, but “There are many popular people and not many people“, which has led to a lot of criticism, but it still cannot stop its popularity. development momentum.
If you are interested in Python and want to learn Python, here I would like to share with you a Complete set of Python learning materials, which I compiled during my own study. I hope it can help you, let’s work together!
Friends in need can click the link below to get it for free or Scan the QR code below to get it for free
CSDN Gift Package: Free sharing of the most complete “Python learning materials” on the entire network(safe link, click with confidence )
?
1Getting started with zero basics
① Learning route
For students who have never been exposed to Python, we have prepared a detailed Learning and Growth Roadmap for you. It can be said to be the most scientific and systematic learning route. You can follow the above knowledge points to find corresponding learning resources to ensure that you learn more comprehensively.
② Route corresponding learning video
There are also many learning videos suitable for beginners. With these videos, you can easily get started with Python~
③Exercise questions
After each video lesson, there are corresponding exercises to test your learning results haha!
2Domestic and foreign Python books and documents
① Documents and books
3Python toolkit + project source code collection
①Python toolkit
The commonly used development software for learning Python is here! Each one has a detailed installation tutorial to ensure you can install it successfully!
②Python practical case
Optical theory is useless. You must learn to type code along with it and practice it in order to apply what you have learned to practice. At this time, you can learn from some practical cases. 100+ practical case source codes are waiting for you!
③Python mini game source code
If you feel that the practical cases above are a bit boring, you can try writing your own mini-game in Python to add a little fun to your learning process!
4Python interview questions
After we learn Python, we can go out and find a job if we have the skills! The following interview questions are all from first-tier Internet companies such as Alibaba, Tencent, Byte, etc., and Alibaba bosses have given authoritative answers. I believe everyone can find a satisfactory job after reviewing this set of interview materials.
5Python part-time channel
Moreover, after learning Python, you can also take orders and make money on major part-time platforms. I have compiled various part-time channels + part-time precautions + how to communicate with customers into documents.
All the above information , if friends need it, you can scan the QR code below to get it for free
?