[Python] 15 Jupyter tricks to save time

Source: DeepHub IMBA

Jupyter Notebooks are very easy to use and come in handy for any python-oriented task. As long as its kernel is active, scripts can be run and tested with subsets of data without having to restart the program each time, which speeds up our development and testing.

But because it is so simple, we often make some mistakes, wasting our time and computational cost. In this post, we’ll discuss some techniques that can save you time and reduce your computational costs.

1. Magic command

In Jupyter notebooks, “magic commands” are special commands that are not part of the Python language but can make your life easier. These commands are preceded by a % sign.

Magic commands are useful and can be embedded directly in python code and solve common problems such as listing all files in the current directory or changing the current working directory.

Here are some common magic commands:

  • %run: Run the Python script in the current kernel.

  • %load: Load code from a script and run it in the current kernel.

  • %who: List all variables.

  • %timeit: Record the execution time of a line of code.

  • ?bug: Enter the debugger at the exception.

  • %matplotlib inline: Display graphics in the notebook.

  • %load_ext: Load extensions, such as IPython extensions.

  • %pwd: Print the current working directory.

  • %ls: Display all files in the current directory.

We can run %lsmagic to see a list of all magic commands.

77818a5e746ba58c4367b98e0cf8350d.png

To get more information about a specific magic command, you can use the ? operator, for example %run?.

2. Execute another Jupyter notebook file

You can use magic commands to do some interesting things. For example, execute python code from a py file, or execute jupyter notebook from an ipynb file.

%run will execute jupyter notebook and display the output, unlike importing a python module.

We can run the two-histogram notebook and get the following output:

%run ./two-histograms.ipynb

3. View document

The documentation for a method can easily be viewed by highlighting the method and pressing Shift+Tab. It will show the docstring that was written when the function was written. You can also open the modal by clicking the + button in the upper right corner.

You can also get more information about each magic command by highlighting it and pressing Shift+Tab.

2df7b1411287cb2fcc1c72a2e975ad52.gif

4. Add multiple cursors

If you need to rename a variable written in several places, or when editing code, you wish you had multiple cursors.

In Jupyter notebooks, text can be edited simultaneously using multiple cursors. This can be useful if you want to make the same change to multiple lines of text at once.

To use multiple cursors in Jupyter notebook, you can Alt-click the desired positions. This will create a cursor at each clicked position. You can then edit as usual and the changes will be applied everywhere simultaneously.

Windows: hold alt + left click and drag the cursor. Mac: Hold down the option key + left mouse button to drag the cursor.

You can also use the Shift + Alt + Up/Down Arrow key combination to select multiple lines of text and create a cursor at the beginning of each line.

87632271759eb9a0847352581defe953.gif

Remember that using multiple cursors can cause problems, so it’s best to save your code before using this feature in case you make any unexpected changes.

5. Insert code from another external python script

The content of the cell can be replaced with an external python script. You can use any python file on your computer, or a URL as a source.

# Before Running
 %load ./hello_world.py

In the next cell you can use:

if __name__ == "__main__":
  print("Hello World!")
 
 output
 Hello World!

6. Run CMD/Shell commands?

In Jupyter notebook, use the ! prefix before the command to run the command of the operating system. For example, to list the files in the current directory, use the ls command:

!ls

You can also pass arguments by appending them to the command. For example, to list files under a specific directory, use the -l option to display files in long format, specifying the directory path as an argument:

!ls -l /path/to/directory

You can also assign the command’s output to a variable and use it in your code. For example:

files = !ls
 print(files)

The above code will assign the list of files in the current directory to the files variable and print them out.

You can use this technique to run any shell command in a Jupyter notebook, as long as that command is available on the system the notebook is running on.

7. Set an alarm clock for program completion?

It’s always helpful to get a notification when your model finishes training or any task.

In windows 10 we can set it using win10toast module.

pip install win10toast

then use this code

from win10toast import ToastNotifier
 toaster = ToastNotifier()
 
 #Your program
 
 toaster. show_toast("Execution complete",
                    "Your calculation completed",
                    duration=10)

We are notified for us when the program finishes executing.

import winsound
 
 # set an alarm of 440HZ for one second (1000ms)
 duration = 1000
 freq = 440
 
 winsound. Beep(freq, duration)

Mac and Linux can use the os module to play sounds using the afplay command (on macOS) or the aplay command (on Linux).

import os
 
 # Run your program here
 # Play a sound when the program completes
 os.system("afplay /path/to/sound.mp3") # macOS
 os.system("aplay /path/to/sound.wav") # Linux

You can replace /path/to/sound.mp3 with the path to the sound file you want to play. Or any audio file supported by the afplay or aplay command, such as MP3, WAV, or AIFF.

This will only work if the afplay or aplay command is available on the system running Jupyter Notebook.

On the Mac, you can also use the built-in command say to say something when the program is complete.

import os
 
 os.system('say "hi siri"')

8. Display execution time

To measure the execution time of a cell in Jupyter notebook, you can use the %timeit magic command. This command will execute a single statement and return the execution time. Here is an example of how to use %timeit:

%timeit sum(range(100))

This calculates the execution time of the sum function and returns the average time it takes to execute the function.

d277c8da7b4c197eed45749dda756c0f.png

You can also use %%timeit to measure the execution time of the entire cell:

%%timeit
 
 total = 0
 for i in range(1000):
     total + = i

1e43a20b37eefeb8cdeb7c840baa765f.png

You can see that the above code is similar to the result of using the time module in Python

import time
 
 start_time = time. time()
 
 # code to measure
 sum(range(100))
 
 end_time = time. time()
 
 elapsed_time = end_time - start_time
 print(f'Execution time: {elapsed_time:.2f} seconds')

e84581aa81c3e7d4cd88f207feb812d0.png

Note: These methods only measure the execution time of the code in the cell. If a Compute Unit depends on other Compute Units or external resources, the execution time will not include the time required to execute those dependencies.

9. Pass variables between notebooks

In Jupyter notebooks, the %store magic command can pass variables between notebooks.

Here’s an example of using it:

var1 = 10
 %store var1

In another notebook, you can use the following command to get the value of the variable

%store -r var1
 print(var1)

The %store magic command has the following operations

%store var1: store variable var1

%store -r var1: Retrieves the stored variable var1 and assigns it to a variable of the same name in the current notebook

%store -d var1: delete the stored variable var1

%store -z: delete all stored variables

You can also store multiple values with a single %store command, such as

%store var1 var2

The %store command only works within the same Jupyter session.

The value of the %store command is also accessible after a kernel restart, see the following example.

c016a0bd3b1a280755e08051f027d1b9.gif

10. List all keyboard shortcuts?

Learning keyboard shortcuts will save you a lot of time. We can check them under the top menu: Help > Keyboard Shortcuts, or by pressing the H key in command mode. Here is a list of some commonly used keyboard shortcuts in Jupyter notebook:

  • Enter: The current cell enters edit mode

  • Esc: The current cell enters command mode

  • Shift + Enter: run the current cell and move to the next cell

  • Ctrl + Enter: Run the current cell

  • Alt + Enter: run the current cell and insert a new cell below

  • Shift + Tab: Display the documentation for the current function or object

  • Ctrl + S: Save

  • A: Insert a new cell above the current cell (in command mode)

  • B: Insert a new cell below the current cell (in command mode)

  • M: Change the current cell to a Markdown cell (in command mode)

  • Y: Change the current cell to a code cell (in command mode)

  • D + D: delete the current cell (in command mode)

  • Z: Undo the last cell deletion (in command mode)

  • X: Cut selected cells (in command mode)

  • C: Copy selected cells (in command mode)

  • V: Paste selected cells (in command mode)

  • Ctrl + Shift + – Splits the current cell into two from where the cursor is. (in command mode)

  • Esc + F: Find and replace your code, but not the output. (in command mode)

  • Esc + O: toggle cell output (in command mode)

Select multiple cells:

  • Shift + Down selects the next cell in the downward direction.

  • Shift + Up selects the next cell in the upward direction. (in command mode)

  • Shift + M: Merge multiple selected cells. (in command mode)

You can also use the %shortcuts magic command to see a list of keyboard shortcuts for the output range of the current cell:

%shortcuts

This will display a list of all shortcut keys and their corresponding actions.

11. Hide unnecessary output

It’s annoying to see unnecessary in-memory code or object ids when you create reports from Jupyter notebooks. To hide this unnecessary output use a semicolon;

Take a look at the following code:

0604558409040860d52cb0587644691d.png

If you want to shield the output of matplotlib, look at the following code, pay attention to the semicolon:

plt.plot(x,y);

9ff9beae1767f9efde1e8f8c6ffd18c5.png

12. Write functions in languages other than python

If you’re dealing with lots of large datasets, and numpy isn’t fast enough, you can write some c or fortran code directly in python code.

If you want to start writing functions in c then you will need the cython library.

!pip install Cython

Load and use:

%load_ext Cython
 
 %%cython
 def myltiply_by_2(float x):
 return 2.0 * x
 
 myltiply_by_2 (23.)

To write fortran functions, you need another library fortrain-magic.

!pip install fortran-magic

code show as below:

%load_ext fortranmagic
 
 %%fortran subroutine compute_fortran(x, y, z)
 real, intent(in) :: x(:), y(:)
 real, intent(out) :: z(size(x, 1))
 z = sin(x + y)
 end subroutine compute_fortran
 
 compute_fortran([1, 2, 3], [4, 5, 6])

13. Expand the number of columns and rows in pandas output

By default, a pandas dataframe can only display a limited number of rows and columns. There are several ways to expand the number of rows and columns displayed in a pandas DataFrame in Jupyter Notebook.

method 1:

Use the pd.options.display.max_rows and pd.options.display.max_columns options.

For example, to display up to 100 rows and 50 columns, you can use the following code:

import pandas as pd
 pd.options.display.max_rows = 100
 pd.options.display.max_columns = 50

Method 2: Set these options using the pd.set_option function. For example:

pd.set_option("display.max_rows", 100)
 pd.set_option("display.max_columns", 50)

Or you can use the head and tail methods to display the first or last rows of the DataFrame. For example:

df.head(10)
 df.tail(5)

Method 3: Use the IPython.display module in IPython to control the display. For example:

from IPython.display import display
 display(df, max_rows=100, max_columns=50)

This will display a DataFrame with 100 rows and 50 columns.

14. Extract input and output unit data

When you’re done executing a cell and you realize you forgot to assign a value to a variable, what do you do?

When we execute the cell in jupyter notebook, it will assign a line number ln:

When the cell finishes executing, we get an output and can access it by passing the execution number as the index

960e872826b8064766b6ba308771c9d0.png

Out is a python dictionary that stores all the output of the cell. We can access the output using the number as an index.

15. Export the content of the cell

When the jupyter test is done we may want to export the contents of the jupyter unit to a python file. The easiest way is to create a py file and copy paste the code, but this is obviously not the best way.

%%writefile is a Jupyter Notebook magic command that saves the contents of a cell as a Python file. For example in a cell with the following code:

%%writefile example.py
 def add(a, b):
     return a + b
 print(add(3, 4))

After you can run the cell, a file named example.py will be created in the directory where the Jupyter Notebook is located. The content of the file is the python code in the cell

%pycat is another Jupyter Notebook magic command that displays the contents of a Python file in a Notebook cell. If you have the following code in a cell:

%pycat example.py

It will display the content of the example.py file in the output of the cell. This is useful for quickly viewing the contents of Python files.

Summary

The above are some of the skills we have summarized, I hope it will be helpful to you.

Author: Anup Das



4d72cbcfca522f95b77157008d01a893.jpeg










Excellent Review of Past Issues




Suitable for beginners to get started with artificial intelligence routes and data downloads (graphics + videos) machine learning introductory series download machine learning and deep learning notes and other materials print "Statistical Learning Methods" code reproduction album machine learning exchange qq group 955171419, join WeChat Group please scan code