~charles/rq

CLI tool for evaluating Rego Queries
add mailing list and issue tracker
tag v0.0.4 release
v0.0.3 release

clone

read-only
https://git.sr.ht/~charles/rq
read/write
git@git.sr.ht:~charles/rq

You can also use your local clone with git send-email.

#Rego Query

builds.sr.ht status go.dev reference Go Report Card documentation buy me a coffee issue tracker mailing list

Early Stage Project: this project is still very much experimental. Expect things to change and break without notice.

rq (Rego Query) is inspired by jq, and aims to offer a similar set of capabilities oriented around running Rego queries. In the long run, rq is intended to make it as easy as possible to utilize Rego both interactively as a shell command, and within shell scripts. Though it is already possible to use Rego in this with via opa eval using the official opa command, rq is intended to be easier and simpler to use specifically in a shell context, and has additional convenience features to support this use case, such as more input and output formats, syntax highlighted output, and so on.

A simple demonstration (see the Showcase section for more):

$ data='{"foo": 7, "bar": 3, "baz": 12, "quux": 4, "spam": 6}'
$ # find all keys in data where the value is at least 5
$ echo "$data" | rq '{k | some k; input[k] > 5}'
[
	"baz",
	"foo",
	"spam"
]

#Features

  • Supported input formats
    • CSV (including comments, arbitrary delimiters, headers, and leading line skip)
    • JSON
    • YAML
    • TOML
  • Supported output formats
    • JSON
    • YAML
    • CSV
    • raw (primitive types, and lists of primitive types, are printed one per line without quotes, otherwise behaves like the JSON output format)
  • Syntax-highlighted output using chroma

#Installation Instructions

With go install:

$ go install git.sr.ht/~charles/rq/cmd/rq@latest

With GNU Make:

$ git clone https://git.sr.ht/~charles/qr
$ cd rq
$ git checkout <tag name>  # skip if you want to use the latest development version
$ make install

#Usage Instructions

See rq --help. You might also like the getting started guide.

#Showcase

For more rq usage examples, check out the cookbook.

Sample data:

$ cat sample_data/books.json
[
    {
        "title": "Writing An Interpreter In Go",
        "authors": ["Thorsten Ball"],
        "isbn": "978-3982016115",
        "year": 2018
    },
    {
        "title": "Writing A Compiler In Go",
        "authors": ["Thorsten Ball"],
        "isbn": "978-3982016108",
        "year": 2018
    },
    {
        "title": "The Go Programming Language",
        "authors": ["Alan A. A. Donovan", "Brian W. Kernighan"],
        "isbn": "978-0134190440",
        "year": 2015
    },
    {
        "title": "Hands-On GUI Application Development in Go",
        "authors": ["Andrew Williams"],
        "isbn": "978-1789138412",
        "year": 2019
    },
    {
        "title": "Building Cross-Platform GUI Applications with Fyne",
        "authors": ["Andrew Williams"],
        "isbn": "1800563167",
        "year": 2021
    }
]
$ cat sample_data/books.csv
title,"first author",isbn,year
Writing An Interpreter In Go,Thorsten Ball,978-3982016115,2018
Writing A Compiler In Go,Thorsten Ball,978-3982016108,2018
"The Go Programming Language","Alan A. A. Donovan",978-0134190440,2015
Hands-On GUI Application Development in Go,Andrew Williams,978-1789138412,2019
Building Cross-Platform GUI Applications with Fyne,Andrew Williams,1800563167,2021

Read JSON data:

$ rq < sample_data/books.json
[
	{
		"authors": [
			"Thorsten Ball"
		],
		"isbn": "978-3982016115",
		"title": "Writing An Interpreter In Go",
		"year": 2018
	},
	{
		"authors": [
			"Thorsten Ball"
		],
		"isbn": "978-3982016108",
		"title": "Writing A Compiler In Go",
		"year": 2018
	},
	{
		"authors": [
			"Alan A. A. Donovan",
			"Brian W. Kernighan"
		],
		"isbn": "978-0134190440",
		"title": "The Go Programming Language",
		"year": 2015
	},
	{
		"authors": [
			"Andrew Williams"
		],
		"isbn": "978-1789138412",
		"title": "Hands-On GUI Application Development in Go",
		"year": 2019
	},
	{
		"authors": [
			"Andrew Williams"
		],
		"isbn": "1800563167",
		"title": "Building Cross-Platform GUI Applications with Fyne",
		"year": 2021
	}
]

Read CSV data, with headers:

$ rq -i csv -H < sample_data/books.csv
[
	{
		"first author": "Thorsten Ball",
		"isbn": "978-3982016115",
		"title": "Writing An Interpreter In Go",
		"year": 2018
	},
	{
		"first author": "Thorsten Ball",
		"isbn": "978-3982016108",
		"title": "Writing A Compiler In Go",
		"year": 2018
	},
	{
		"first author": "Alan A. A. Donovan",
		"isbn": "978-0134190440",
		"title": "The Go Programming Language",
		"year": 2015
	},
	{
		"first author": "Andrew Williams",
		"isbn": "978-1789138412",
		"title": "Hands-On GUI Application Development in Go",
		"year": 2019
	},
	{
		"first author": "Andrew Williams",
		"isbn": 1800563167,
		"title": "Building Cross-Platform GUI Applications with Fyne",
		"year": 2021
	}
]

Select all books published after 2018:

$ rq -i csv -H '{book.title | book := input[_]; book.year > 2018}' < sample_data/books.csv
[
	"Building Cross-Platform GUI Applications with Fyne",
	"Hands-On GUI Application Development in Go"
]

As above, but with CSV output:

$ rq -o csv '{b | b := input[_]; b.year > 2018}' < sample_data/books.json
isbn,title,year,authors[0]
1800563167,Building Cross-Platform GUI Applications with Fyne,2021,Andrew Williams
978-1789138412,Hands-On GUI Application Development in Go,2019,Andrew Williams

Select all authors who have published at least 2 books:

$ rq '{author | book := input[_]; author := book.authors[_]; count({b | b := input[_]; author in b.authors}) > 1}' < sample_data/books.json
[
	"Andrew Williams",
	"Thorsten Ball"
]

As above, but with YAML output:

$ rq -o yaml '{author | book := input[_]; author := book.authors[_]; count({b | b := input[_]; author in b.authors}) > 1}' < sample_data/books.json
- Andrew Williams
- Thorsten Ball

As above, but with "raw" output:

$ rq -o raw '{author | book := input[_]; author := book.authors[_]; count({b | b := input[_]; author in b.authors}) > 1}' < sample_data/books.json
Andrew Williams
Thorsten Ball

Run a shell command and parse the output as JSON:

$ rq 'rq.run(["jq", ".[0]"], {"stdin_spec": {"file_path": "sample_data/books.json"}, "stdout_spec": "json:"})'
{
	"exitcode": 0,
	"stderr": "",
	"stdout": {
		"authors": [
			"Thorsten Ball"
		],
		"isbn": "978-3982016115",
		"title": "Writing An Interpreter In Go",
		"year": 2018
	}
}

#Performance

Performance numbers use the jeopardy.json from this blog post. Each test was run 6 times, with the first run dropped and the remaining 5 runs averaged. jq-1.6 is used. The test system is running Arch Linux with kernel 5.17.4-zen1-1-zen on a Framework laptop with an i7-1165G7, 32GB memory, and a WD Black NVMe SSD with ZFS. Time is measured using the time command, and only the real time is considered. rq is build against go version go1.18.1 linux/amd64. The version of rq used for this test was level with b2e830f006fa4051c8c027ae41aea86a8af7745c. Output is written to disk, so this implies no colorization for either jq or rq.

Note that by default, the input file is parsed as YAML. I also show performance when using -i json which uses the standard library's JSON implementation, which is faster than parsing YAML. --ugly is used for some tests to disable pretty-printing output as well.

  • Identity, read the input in JSON, write it to disk in JSON
    • jq: mean=1.757s, stdev=0.08641s
    • rq: mean=4.126s, stdev=0.06777s
    • rq --ugly: mean=3.821s, stdev=0.6064s
    • rq -i json --ugly: mean=1.927s, stdev=0.07547s
  • Select only the subset of the question which has category HISTORY
    • jq '.[] | select(.category=="HISTORY")': mean=0.6974s, stdev=0.01293s
    • rq '{q | q := input[_]; q.category == "HISTORY"}': mean=3.327s, stdev=0.04233s
    • rq -i json --ugly '{q | q := input[_]; q.category == "HISTORY"}': mean=1.305s, stdev=0.02595s
  • Select only the question field of those questions which have category HISTORy
    • jq '.[] | select(.category=="HISTORY") | .question': mean=0.7524s, stdev=0.1202s
    • rq '{q.question | q := input[_]; q.category == "HISTORY"}': mean=3.303s, stdev=0.02696s
    • rq -i json --ugly '{q.question | q := input[_]; q.category == "HISTORY"}': mean=1.305s, stdev=0.01077s

#License

See ./LICENSE.

#Changelog

See ./CHANGELOG.md.

#Documentation

See ./doc/README.md.

#Community & Contribution

Feel free to discuss or send patches via the rq mailing list.

If you have found an issue with rq, please do so via the rq issue tracker.

For instructions on how to properly prepare patches for use with SourceHut projects such as rq, see the git.srr.ht docs.

#Supporting rq

If you would like to support the project financially, donations are accepted via buy me a coffee. Your support is greatly appreciated.

#Disclaimer

I am a Styra (the creators of OPA) employee. This project is not endorsed, affiliated with, or supported by Styra or the OPA project. You have no expectation of support from Styra or the OPA team if you use this software.