~ach/vtdb

Fixed size value tracking database.
Add does not exist error.
Make api more intuitive.
Flush output files when done.

clone

read-only
https://git.sr.ht/~ach/vtdb
read/write
git@git.sr.ht:~ach/vtdb

You can also use your local clone with git send-email.

#vtdb

Value tracking database - Track counters, values and events in a simple fixed size file via the command line and a C library.

This tool:

  • Tracks and counts events/values over time with a configurable resolution and retention.
  • Stores data in a crash resistent fixed size file.
  • Never runs out of space once the file is preallocated.

Backstory:

I wanted to plot a graph of my bupstash backup repository sizes over time in a simple way and resilient way. I decided a ring buffer on disk where new values overwrite old values makes sense, but I didn't have a suitable tool to do that.

Demo asciicast:

asciicast

#Why not...?

Why not rrdtool? Rrdtool requires fixed interval updates and is not really designed for tracking irregular events. I did use it as an inspiration however.

Why not prometheus? Prometheus doesn't like high cardinality data. Prometheus also requires http endpoints, and it is quite a headache to run one for every metric you want to collect.

Why not implement it in rust/go/...? I am a fan of rust and other languages, however this tool is so simple that it does not require dynamic memory allocation. we manually perform bounds checks when writing to the database to minimize the risk of out of bounds access. C also has some advantages these tools do not... we are able to mmap the database (which was not possible in my initial Go prototype) and it is easier to distribute and embed a C library in other projects. If warranted, it would definitely be a fun project to use Frama-C to prove the absence of undefined behavior.

#User Manual

NAME
       vtdb - Value tracking database.

SYNOPSIS
       Track values over time in a fixed size file.

       vtdb init [-s starttime] <file> <resolution> <retention> <name>...
       vtdb add <file> [-a] <time> <value>...
       vtdb set <file> [-a] <time> <value>...
       vtdb current-values [-H] <file>
       vtdb values [-H] [-i interval] [-s starttime] [-e endtime] <file>
       vtdb dump <file>

SUBCOMMANDS
       init [-s starttime] file resolution retention name
       Initialize a database file for tracking 1-7 values.

       add [-a] file time value ...
       Add  to  the  current  values.  Passing -a (async flush) increases speed, but reduces data
       durability.

       set [-a] file time value ...
       Set the current values.  Passing -a (async flush) increases speed, but reduces data  dura‐
       bility.

       current-values [-H] file
       Print the current values in TSV format.  with an optional header (specify -H).

       values [-H] [-i internal] [-s starttime] [-e endtime] file
       Print values in TSV format with an optional header (specify -H).

       dump file
       Perform a debug dump of the database.

TIME FORMAT
       Duration:
              [0-9]+(ms|s|m|h|d)

       Time (seconds since unix epoch):
              ([0-9]+('.'[0-9]{1,3})?)|now|(now'-'DURATION)

NOTES
       Concurrency:
              The database uses POSIX locking to control concurrent access.

       Crash Safety:
              The  current  values and database header fit in one filesystem page so they are un‐
              likely to be corrupted by a crash.  Historic values may be affected  by  unexpected
              power loss.

EXAMPLES
       $ vtdb init ./data.vtdb 1h 30d v1 v2
       $ vtdb add ./data.vtdb now 1 -4.1
       $ vtdb set ./data.vtdb now 6 6
       $ vtdb set ./data.vtdb now _ 0
       $ vtdb current-values -H ./data.vtdb
       $ vtdb values -H -i 6h -s now-3d -e now ./data.vtdb

#Design notes

#File format

See vtdb.h for the main data structure (a header and ring buffer), this is memory mapped and updated directly.

Because the file is memory mapped, the format is endian specific, which is checked via the magic value in the header.

Debug dumping a vtdb database is a good way to understand the format.

$ vtdb init ./bytes.vtdb 500ms 5s bytes
$ vtdb dump ./bytes.vtdb
start_millis=1648073778736
step_millis=500
n_steps=10
ring_index=0
n_values=1
value_names[0]=bytes
last_update_millis=1648073778736
current_values.bytes=0.000000
ring_buffer[0].bytes=0.000000
ring_buffer[1].bytes=0.000000
ring_buffer[2].bytes=0.000000
ring_buffer[3].bytes=0.000000
ring_buffer[4].bytes=0.000000
ring_buffer[5].bytes=0.000000
ring_buffer[6].bytes=0.000000
ring_buffer[7].bytes=0.000000
ring_buffer[8].bytes=0.000000
ring_buffer[9].bytes=0.000000