~cnx/ipwhl-docs

ref: c478171b3e5e808bf5d1857f162102783acb6338 ipwhl-docs/index.md -rw-r--r-- 7.6 KiB
c478171bNguyễn Gia Phong Document checks' criteria a month ago

#title: Floating cheeses

#What?

Floating cheeses are collections of carefully handpicked cheeses from the cheese shop and other sources. Analogies aside, the interplanetary wheels (IPWHL) are platform-unique, single-versioned Python binary distributions backed by IPFS. It aims to be a downstream wheel supplier in a similar fashion to GNU/Linux distributions, whilst take advantage of a content-addressing peer-to-peer network to provide a reproducible, easy-to-mirror source of packages.

On IPWHL, for each platform (architecture, operating system and Python implementation and version), there exists only one single binary distribution. The collection of these distribution packages are given as a single IPFS CID. An installer can use solely this content ID and packages names to reproduce the exactly same environment on every platform.

The official IPWHL repository will provide exclusively free software. However, deriving the repository should be trivial and is a supported use case.

#Why?

The cheese shop is great, but choosing cheeses from it can often be confusing. Dependency resolution is expensive, and version requirements are not future-proof. In order to avoid breakage, people usually have to pin packages on the installer side, which is redundant and difficult to validate manually. Additionally, we believe it is not the packaging users' job to do this; they should be able to save their time doing what they do best: writing and using software.

Moreover, there are millions of ways for a piece of cheese to rot on the way home from the (almost) lawless cheese shop. Everyone can sell at the shop, and thus typosquatting is a common exploit. In addition, cheeses from the shop are not independently verifiable: the checksums are provided along with the files so the shop is the single point of failure for security attacks. There are ongoing efforts to integrate TUF into Python packaging toolchain, however it is unlikely that they can entirely mitigate this due to the centralized nature of the inherent architecture.

Centralization also makes it really difficult for mirrors to be useful for the users: the cheese shop is not aware if any of its mirrors, let alone redirecting to the closest one. Mirroring is hardly a collaborative effort, one either provide everything for an entire region, or give up. On the other hand, many organizations host their Python packages and their dependencies on dedicated machines running 24/7, but the resources are mostly gone to waste when unused by the companies themselves.

IPWHL makes use of IPFS and statically declared and carefully curated metadata to try to solve most the listed problems. In addition to providing only one wheel version at a time, source distributions are not supplied to avoiding executing untrusted code on the users' machine.

#How?

In order to add a wheel file to IPWHL, it must be statically declared and checked, then the declaration collection shall be published.

#Declare

A IPWHL declaration is a flat TOML file consisting of the following fields:

  • source: Upstream URL to the wheel
  • content-id: CIDv0 of the wheel
  • requires-python (optional): Specifier set representing Python versions the wheel is compatible with, read from Requires-Python in metadata 1.2
  • extras: Array of optional features, read from Provides-Extra in metadata 2.1
  • dependencies: Array of requirements from Requires-Dist in metadata 1.2, converted to PEP 508 format

The file shall be placed in the following path:

$prefix/$project/${wheelname%.whl}.toml

where project is the (pre-normalized) project name, prefix is the smallest 8 bits in lowercase hexadecimal of the sum of characters in project and wheelname is the file name of the wheel as specified in PEP 427.

#Check

#File Organization

The following criteria are used to check the paths to declarations:

  • Location: The path to the parent directory of a declaration must be in the format of $prefix/$project as described above.
  • Filename: The extension must be .toml and were it .whl the hypothetical name must make up a valid wheel filename according to PEP 427.
  • Uniqueness: No pair of wheels declared in one project can be installed on a same system. This is examined based on the tags in the filenames.

#Declarations

The integrity of a declaration is verified by comparing the original wheel with the declared fields. Before performing this, one must ensure that the source URL has not been compromised. Due to the lack of warranty for the long-term availability of the source and to save resources, an implementation may only do this for recently modified declarations, e.g. ones added or updated by a patch.

#Dependencies

Every wheels declared must have their dependencies, including extras, satisfied on every supported platform.

#Publish

If all checks pass, a new repository release can be published from the declarations.

#Build

For a fresh build, every wheel declared is downloaded and placed according to PEP 503:

$project/$wheelname

The parent directory is then recursively added to IPFS. To leverage subsequent builds, IPFS mutable file system (MFS) can be utilized for caching: only new declarations and those not matching the content ID are downloaded and added to the previous repository version. This way, we need not rely on the upstream URLs' availability; however wheels from declarations having been updated or removed must be garbage collected.

#Garbage Collection

Should MFS be used, the following files and directories must be removed to maintain the property of being platform-unique and single-versioned of IPWHL:

  • Projects not declared in the current metadata
  • Wheels not declared in the current metadata
  • Wheels whose CIDs mismatches with the declarations (these shall be replaced with the correct ones)

#Announcement

New releases must be tagged along with their respective CIDs and cryptographically signed on the data Git repository. Afterwards, the maintainers should announce it on the announce mailing list.

#Where?

#Git Repositories

The mainline Git trees are hosted on SourceHut:

#Mailing Lists

In ascending order of traffic:

#Tickets

Bugs and requests are tracked on SourceHut TODO, which also supports submitting via email to ~cnx/ipwhl@todo.sr.ht.

#Instant Messaging

The maintainers of the floating cheeses can be found on Matrix at #ipwhl:matrix.org. However, there is no concept of an official chat room, so feel free to create ones on other platforms. We will consider bridging them to the Matrix room upon requests.