[python\import] When importing, the local project code (library constructed with setup.py and other methods) is given priority instead of the library installed by pip || Code test

【start:20231103】

Article directory

introduction
- Problem Description
- Specific case
Construct your own python library
- Files involved in constructing the library
- - pyproject.toml
  - setup.py
- Commands involved in constructing the library
- - `pip install -e .`
  - `python setup.py develop`
  - `python setup.py install`
other problems
- Problems with linkage between construction library and importlib
Practical combat: Installation of local Cellpose library
- Case 1: LookupError: setuptools-scm was unable to detect version
- - Solution
- Case 2: fatal error: libaec.h: No such file or directory
- - Solution

Introduction

Problem description

Question: Suppose I am executing the A.py file. If I have a folder called cellseg_models_pytorch, the location is under the parent folder of the executable file, but I have previously installed the official version of cellseg_models_pytorch using the pip install cellseg_models_pytorch command.
Now when I import cellseg_models_pytorch in the A.py file, I want to import my cellseg_models_pytorch folder instead of the one installed before pip. What should I do?

conclusion of issue:

How to import your own folder as a library instead of pip install?
After importing your own folder as a library, how do you specify the location of the library as the source code instead of starting from scratch?

Specific cases

I once pip install cellseg_models_pytorch,

But now, I want to in the pannuke_nuclei_segmentation_cppnet.ipynb file,

When using the from cellseg_models_pytorch.datamodules import PannukeDataModule statement,

Instead of using the library downloaded by pip, use the local project code with cellseg_models_pytorch as the folder name as the imported cellseg_models_pytorch library:

How should this be achieved?

[ref]About python giving priority to calling the code in the project directory instead of the installed package

Construct your own python library

Files involved in constructing the library

During Python development, we often need to use third-party libraries and modules to extend our functionality. When we use the pip command to install these libraries, we sometimes encounter an error that the directory is not installable. This error means that Python does not recognize the current directory as an installable package or module.

This error is usually caused by a missing setup.py or pyproject.toml file. These files are the build configuration files for the Python package and are used to tell Python how to build and install the package.

[ref]Python ERROR: Directory cannot be installed. There is neither ‘setup.py’ nor ‘pyproject.toml’

Both the pyproject.toml file and the setup.py file play important roles in Python projects. pyproject.toml is used to The project’s metadata and build configuration, while setup.py is used for the project’s installation and packaging configuration. Understanding the role of these files can help you better manage and configure your Python projects.

pyproject.toml

The pyproject.toml file is typically used to define a Python project’s metadata, build system configuration, and dependencies. It is a new configuration file format that replaces the old setup.cfg file. In pyproject.toml, you can specify the metadata information of the project, such as name, version, author, etc., and you can also specify the project’s build tools and configuration, such as build-system part is used to specify the build tools used by the project, such as flit, setuptools, etc. In addition, you can also specify the project's dependencies in pyproject.toml, including the required Python version and other dependent libraries.


setup.py
The setup.py file is usually used to define the installation and packaging information of Python projects. In this file, you can use the setuptools library to define the installation requirements, version and other information of the project. Typically the project's name, version, author, description, and other configurations are defined. You can also specify your project's dependencies and other installation-time configuration options.
Commands involved in constructing the library
If there is a pyproject.toml file or a setup.py file, you can enter the corresponding parent folder path in cmd (delete the original .eggs and other files), and then run the following command Install local libraries:

pip install -e . is a source installation, suitable for development environments, allowing immediate viewing of modifications to the code without reinstallation.
python setup.py develop is a development mode installation method. Modifications to the source code will be immediately reflected in the installed package, which is suitable for continuous editing and testing during development.
python setup.py install is a traditional binary installation method, suitable for formal environments. The edited source code modification will not take effect immediately, and the installation command needs to be re-run.

pip install -e .
This is an editable install command used to install Python packages in development mode. Key features include:


Symbolic link or copy source code: Link the package's source code to the installation path by creating a symbolic link or copying the source code. This means you can make edits based on the original code without reinstalling.


For development environments: Typically used during development to allow immediate viewing of changes to code. Useful for sharing code and instant feedback.



Use the pip tool: Use the pip tool, which is a general package management tool that can handle dependencies and other related matters.


Usage example (you can use the --editable or -e option):
pip install -e .

For example, after using this method to generate the cellpose library, the library is the original code. The location of __init__.py is as follows:
/home/linxq/code/cell_seg_workflow/src_cite/segment/cellpose-main/cellpose/__init__.py

python setup.py develop
Similar to pip install -e ., it is also a source installation method that links the source code of the package to the installation path by creating a symbolic link or copying the source code.


Purpose: Used to install packages in development mode, allowing you to edit based on the source code without reinstalling.


Effect: Create a symbolic link (or copy) to link the source code of the package to the installation path, allowing you to make modifications based on the original code without reinstalling.


Effect after editing: If you make modifications to the source code after installation, these modifications will be immediately reflected in the installed package without re-running the installation command.


Usage example:
python setup.py develop

the difference:

pip install -e . uses the pip tool, which is a general package management tool that can handle dependencies and other related matters.
python setup.py develop uses the installation script provided by setuptools, a tool for building, publishing, and installing Python packages.

python setup.py install
This is a traditional binary installation command, used to install the binary files of Python packages into the system. Key features include:


Copy binary: Copies the package's binary (usually a compiled .pyc file) to the specified installation directory.


For formal installation: Typically used when preparing a package for distribution to others or for use in a production environment.


Using the setuptools tool: Used the installation script provided by setuptools, which is a tool for building, publishing and installing Python packages.


Usage example:
python setup.py install

For example, after using this method to generate the cellpose library, the library is saved in the "site-packages" of conda envs. The location of __init__.py is as follows:
/home/linxq/.conda/envs/seg38/lib/python3.8/site-packages/cellpose-3.0.0-py3.8.egg/cellpose/__init__.py

Other questions
Problems with linkage between construction library and importlib
use:
import importlib
importlib.reload(xxx)

When xxx is a local project that has gone through the "construction" operation mentioned above, the reload method seems to be invalid.
Practice: Installation of local Cellpose library
implement:
(seg38) linxq@lab:~/code/cell_seg_workflow/src_cite/segment/cellpose-main$ python setup.py install

Finally, the effect we expect is:

However, in fact, you will encounter the following abnormal situations:
Case 1: LookupError: setuptools-scm was unable to detect version
run:
(cpp310) linxq@lab:~/code/cell_seg_workflow/src_cite/segment/cellpose-main$ python setup.py install

return:
linxq@lab:~/code/cell_seg_workflow/src_cite/segment/cellpose-main$ python setup.py install
/opt/anaconda3/lib/python3.11/site-packages/setuptools/__init__.py:84: _DeprecatedInstaller: setuptools.installer and fetch_build_eggs are deprecated.
!!

        *************************************************** ****************************
        Requirements should be satisfied by a PEP 517 installer.
        If you are using pip, you can try `pip install --use-pep517`.
        *************************************************** ****************************

!!
  dist.fetch_build_eggs(dist.setup_requires)
Warning: 'classifiers' should be a list, got type 'tuple'
WARNING setuptools_scm.pyproject_reading toml section missing 'pyproject.toml does not contain a tool.setuptools_scm section'
Traceback (most recent call last):
  File "/home/linxq/code/cell_seg_workflow/src_cite/segment/cellpose-main/setup.py", line 73, in <module>
    setup(
  File "/opt/anaconda3/lib/python3.11/site-packages/setuptools/__init__.py", line 107, in setup
    return distutils.core.setup(**attrs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 147, in setup
    _setup_distribution = dist = klass(attrs)
                                 ^^^^^^^^^^^^
  File "/opt/anaconda3/lib/python3.11/site-packages/setuptools/dist.py", line 486, in __init__
    _Distribution.__init__(
  File "/opt/anaconda3/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 283, in __init__
    self.finalize_options()
  File "/opt/anaconda3/lib/python3.11/site-packages/setuptools/dist.py", line 925, in finalize_options
    ep(self)
  File "/opt/anaconda3/lib/python3.11/site-packages/setuptools/dist.py", line 945, in _finalize_setup_keywords
    ep.load()(self, ep.name, value)
  File "/home/linxq/code/cell_seg_workflow/src_cite/segment/cellpose-main/.eggs/setuptools_scm-8.0.4-py3.11.egg/setuptools_scm/_integration/setuptools.py", line 101, in version_keyword
    _assign_version(dist, config)
  File "/home/linxq/code/cell_seg_workflow/src_cite/segment/cellpose-main/.eggs/setuptools_scm-8.0.4-py3.11.egg/setuptools_scm/_integration/setuptools.py", line 56, in _assign_version
    _version_missing(config)
  File "/home/linxq/code/cell_seg_workflow/src_cite/segment/cellpose-main/.eggs/setuptools_scm-8.0.4-py3.11.egg/setuptools_scm/_get_version_impl.py", line 112, in _version_missing
    raise LookupError(
LookupError: setuptools-scm was unable to detect version for /home/linxq/code/cell_seg_workflow/src_cite/segment/cellpose-main.

Make sure you're either building from a fully intact git repository or PyPI tarballs. Most other sources (such as GitHub's tarballs, a git checkout without the .git folder) don't contain the necessary metadata and will not work.

For example, if you're using pip, instead of https://github.com/user/proj/archive/master.zip use git + https://github.com/user/proj.git#egg=proj

Solution
First, make sure the python version is correct (python==3.8): change the environment cpp310 to seg38;
Secondly, you must specify the version of your own library, for example, in the setup variable of the setup.py file:
exist,
setup(
    name="cellpose",
    license="BSD",
    ...
)

, insert version (you can edit the version number yourself):
setup(
    name="cellpose",
    version="3.0.0",
    license="BSD",
    ...
)

details as follows:

Then execute in sequence:
conda create -n seg38 python==3.8
conda activate seg38

pip install "/home/linxq/.conda/whl/torch-1.13.1 + cu117-cp38-cp38-linux_x86_64.whl"
pip install -r environment2.yml
python setup.py install

Case 2: fatal error: libaec.h: No such file or directory
/home/linxq/.conda/envs/seg38/lib/python3.8/site-packages/setuptools/_distutils/cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated.
!!

        *************************************************** ****************************
        Please avoid running ``setup.py`` directly.
        Instead, use pypa/build, pypa/installer or other
        standards-based tools.

        See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.
        *************************************************** ****************************

!!
  self.initialize_options()
In file included from /home/linxq/.conda/envs/seg38/lib/python3.8/site-packages/numpy/core/include/numpy/ndarraytypes.h:1940,
                 from /home/linxq/.conda/envs/seg38/lib/python3.8/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
                 from /home/linxq/.conda/envs/seg38/lib/python3.8/site-packages/numpy/core/include/numpy/arrayobject.h:5,
                 from imagecodecs/_aec.c:1215:
/home/linxq/.conda/envs/seg38/lib/python3.8/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
   17 | #warning "Using deprecated NumPy API, disable it with " \
      | ^~~~~~~
imagecodecs/_aec.c:1222:10: fatal error: libaec.h: No such file or directory
 1222 | #include "libaec.h"
      | ^~~~~~~~~~
compilation terminated.
error: Setup script exited with error: command '/usr/bin/gcc' failed with exit code 1

Solution
Reinstall the imagecodecs library:
pip install imagecodecs==2021.4.28
conda install -c conda-forge imagecodecs

return:
...
Using /home/linxq/.conda/envs/seg38/lib/python3.8/site-packages
Finished processing dependencies for cellpose==3.0.0

【ref】install imagecodecs on mac big sur