~ehmry/blobsets

Simple object store
README: Add link to blobservices
Fix README hyperlinks
Update to latest CBOR API

clone

read-only
https://git.sr.ht/~ehmry/blobsets
read/write
git@git.sr.ht:~ehmry/blobsets

You can also use your local clone with git send-email.

#A post-UNIX storage layer

UNDER CONSTRUCTION

This repository hosts a proof-of-concept storage system and synthetic file- system frontend. The concept of the file has been replaced with the blob, a finite and linear stream of bytes. Blobs are identified by 192 bit digests using the Tiger hash in tree mode (this is the ADC protocol hash).

A blobset is flat map of key hashes to blob hashes. Key hashes are 64 bits and sets are implemented as lazy-loaded hash array mapped tries with a 64 node fan-out. A store contains both data blobs and blobs of set metadata and is the public API to both blobs and sets.

Two stores are provided, one that stores blobs to a UNIX file-system backend and the other as a HTTP client. A primitive HTTP server is provided to serve the UNIX backend to the client. A Lisp-like REPL is provided to build and explore sets and a Genode file-system server is provided to serve sets as file-systems.

Exposing a set as a read-only UNIX file-system is trivial because set keys are hash digests of arbitrary byte-strings, so names containing the / character may be mapped in the set.

There is no plan to create any FUSE like interface, but exposing blobsets from Genode to Linux via 9P over TCP is known to work.

#Anti-features

  • Blob "names" cannot be traversed, only opaque key and blob pairs.
  • No partial blob deduplication via blocks.
  • Set metadata has low locality.
  • No block-level storage backend.

#Features

  • Simple, functional API. Available via an Lisp REPL.
  • Keys for referencing blobs are arbitrary, can be text or binary.
  • Blob data has high locality.
  • Can be hosted on traditional file-systems.
  • Simple to replicate or backup.

#Storing music

A plugin for the Picard tagger is available at extras/picard_plugin. This plugin ingests audio files into a rolling set and indexes them by MusicBrainz recording UUID. A simple playlist generator for playing these files may be found at https://git.sr.ht/~ehmry/musicbrainz_playlist_generator