Use ktra to build a private Cargo registry

The latest articles are only posted at: https://zjp-cn.github.io/posts/ktra-cargo-registry/

use tools:

  • ktra: A command-line tool for building a Cargo registry written in pure Rust

Install it:

cargo install ktra

Notice:

  • This enables secure-auth and db-sled, the database is the built-in sled
  • If you need redis or mongo as the backend database, add features yourself, see https://book.ktra.dev/installation/cargo.html
  • If you don’t want to run via command line, but deploy via docker, see https://book.ktra.dev/installation/docker.html

Step 1: Create a Git Index repository

Since Cargo uses git or sparse protocols, the latter is only recently stable (2023), while ktra
It appeared earlier, so it is currently only based on the git protocol.

Create a new warehouse on gitee, such as ktra-cargo-registry, then create a new warehouse locally (the main branch is main), create a warehouse configuration file, and submit and push:

mkdir ktra-cargo-registry
cd ktra-cargo-registry
git remote add origin [email protected]:xxx/ktra-cargo-registry.git
echo '{"dl":"http://localhost:8000/dl","api":"http://localhost:8000"}' > config.json
git add config.json
git commit -am "initial commit"
git push origin main

Notice:

  • This article describes the private registry deployment (the Index warehouse is private, and the published crate is stored in its own private server)
  • If you want wider public use based on a private registry, you might be interested in ktra’s OpenId (it’s more complicated)

Step 2: Add a local Cargo registry

In the configuration file of Cargo, such as the global configuration file ~/.cargo/config

[registries]
ktra = { index = "[email protected]:xxx/ktra-cargo-registry.git" }

Notice:

  • The global configuration file can be ~/.cargo/config or ~/.cargo/config.toml, see https://doc.rust-lang.org/cargo/ reference/config.html
  • You can fill in the HTTPS address, git verification will be easier (because the SSH method needs to set Cargo’s git-fetch-with-cli, and gitee to set the SSH public key)

    • Specifically, in the case of HTTPS, logging in or obtaining the registry requires manually entering the account and password

      $ cargo b
    Updating `ktra` index
Username for 'https://gitee.com': xxx
Password for 'https://[email protected]': xxx

    • So, you may also need to set git config --global credential.helper store or git config --global credential.helper cache
      To reduce the number of manual logins. Well, this is the most basic login setup for git

Step 3: Create ktra configuration file and deploy service

A TOML file, passed to the ktra command line, which configures ktra’s built-in database. I store it in ~/.config/ktra/ktra.toml:

[index_config]
branch = "main"

remote_url = "https://gitee.com/xxx/ktra-cargo-registry.git"
https_username = "xxx"
https_password = "xxx"

# These two settings don't seem to work
git_email = ""
git_name = "ktra"

# ssh always appears git error: username not defined
# remote_url = "ssh://[email protected]/xxx/ktra-cargo-registry.git"
# ssh_username = "xxx"
# ssh_privkey_path = "~/.ssh/id_ed25519"
# ssh_pubkey_path = "~/.ssh/id_ed25519.pub"

cd ~/.config/ktra/
ktra # or ktra -c ./ktra.toml, this command deploys the service to the local port 8000

Step 4: Prepare account

Since the service needs to be running all the time, open a new terminal and run the following command:

# Create account admin and password, get {"token":"xxx"}
curl -X POST -H 'Content-Type: application/json' -d '{"password":"PASSWORD"}' http://localhost:8000/ktra/api/v1/new_user/admin

# cargo login ktra: publish packages, such as cargo publish/yank, etc.
CARGO_NET_GIT_FETCH_WITH_CLI=true cargo login --registry ktra
#please paste the token found on http://localhost:8000/me below
# During the dialogue, paste the token and see the following line to log in successfully
# Login token for `ktra` saved

Note that the environment variable here means to use the git command line for verification. If you don’t do this, you need to refer to it yourself
https://doc.rust-lang.org/cargo/appendix/git-authentication.html#ssh-authentication.

Self-authentication seems complicated, and it does not support full SSH authentication (I haven’t tried it), it is recommended to use git cli to verify.

(If your git doesn’t want to mess with SSH, use the HTTPS address in the Cargo registry, step 2 has been described.)

Note: Since the registry is written in the SSH address, it means that SSH verification is required to pull the library from ktra. It is recommended not to use the local environment variable of the command line as above.

I did that for quick verification, you’d better set it as a global environment variable or write it in the global or local .cargo/config.toml configuration file

[net]
git-fetch-with-cli=true

Step 5: Try your own registry

Repository

cargo new ktra-test --lib
cd ktra-test
cargo publish --allow-dirty --registry=ktra

# cargo publish output
   Packaging ktra-test v0.1.0 (/rust/tmp/ktra-test)
   Verifying ktra-test v0.1.0 (/rust/tmp/ktra-test)
   Compiling ktra-test v0.1.0 (/rust/tmp/ktra-test/target/package/ktra-test-0.1.0)
    Finished dev [unoptimized + debuginfo] target(s) in 1.13s
    Packaged 3 files, 941.0B (731.0B compressed)
   Uploading ktra-test v0.1.0 (/rust/tmp/ktra-test)
    Updating `ktra` index
remote: Enumerating objects: 6, done.
remote: Counting objects: 100% (6/6), done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 5 (delta 0), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (5/5), 486 bytes | 486.00 KiB/s, done.
From ssh://gitee.com/xxx/ktra-cargo-registry
   7c6b370..90c97e3 -> origin/HEAD

From my observations, publishing a library to ktra does the following:

  • Package the library locally, then compile, after success

    • Push the information of this library to the INDEX warehouse (just a json file)

    {<!-- -->"name":"ktra-test","vers":"0.1.0","deps":[],"cksum":"89c08ea6c8ea00f06a7f0dd2ec468cf1116458aa38cd6aa5caf1cde05b7e0849", "features":{<!-- -->},"yanked":false,"links":null}
    • Upload the packaged library data to the server running ktra
      • If you follow my above steps, then you are running ktra in ~/.config/ktra/, then the library is under crates/
    $ ll ~/.config/ktra/
total 28K
drwxr-xr-x 3 root root 4.0K Mar 24 18:44 crates/
drwxr-xr-x 2 root root 4.0K Mar 24 16:09 crates_io_caches/
drwxr-xr-x 3 root root 4.0K Mar 24 21:33 db/
drwxr-xr-x 4 root root 4.0K Mar 24 18:44 index/
-rw-r--r-- 1 root root 348 Mar 24 22:57 ktra.toml

$ ll ~/.config/ktra/crates/ktra-test/0.1.0/download
-rw-r--r-- 1 root root 731 Mar 24 18:44 /root/.config/ktra/crates/ktra-test/0.1.0/download

Pull repository

cargo new use-ktra
cargo add serde # library from crates.io
cargo add ktra-test --registry ktr
cargo check # bingo!

# cat Cargo.toml
[dependencies]
serde = "1.0.158"
ktra-test = { version = "0.1.0", registry = "ktra" }

Answer some basic questions

Why do I need a private registry?

If your code is public for others to use, of course you don’t need a private registry.

If not, then you probably want to get something privately controlled and small in scope.

Are there any other options besides ktra?

See https://github.com/rust-lang/cargo/wiki/Third-party-registries for other options.

ktra is written in Rust, and its functions are simple and small, especially suitable for personal use or small group use.

The relationship between the Index warehouse and ktra?

I haven’t looked into it carefully, but I think:

  • The Index warehouse only stores the basic information of the crate (name, version, its dependency information, etc., without code), there is a real example in step 5, or go to
    https://github.com/rust-lang/crates.io-index Take a look.
  • Ktra can be regarded as a server, manages crate and Index, and stores the crate in the local directory where ktra is running. (The data packaged by cargo publish is the project source code)

Is there a lighter weight alternative to ktra?

Cargo supports pulling from private repositories over HTTPS/SSH, so it’s easiest to do something like:

[dependencies.my-private-crate]
git = "ssh://xxx"
tag = "xxx"

But git tags don’t have semver. You can follow this discussion.

Mix different registries in the project?

sure!

Cargo uses crates.io as the default registry. If it needs to be specified as ktra, only relevant command line parameters or configuration parameters are required.

Step five is an example. But doing so will prevent your project from being published on crates.io, because it does not support libraries from other registries. while ktra
Pulling from crates.io is also not supported – but what you need is completely customizable.