~elektito/gemplex

A Gemini Search Engine
Remove some extraneous url parsing in crawler
Remove unused import

refs

master
browse  log 

clone

read-only
https://git.sr.ht/~elektito/gemplex
read/write
git@git.sr.ht:~elektito/gemplex

You can also use your local clone with git send-email.

#Gemplex: A Gemini Search Engine

Gemplex is an experimental Gemini search engine currently available at gemplex.space.

Gemplex is written in Go. In order to build it, you need to have go available on your machine (tested with Go 1.20). You can build all the required executable by running make.

Gemplex consists of three executables: gemplex which is the main executable, gpcgi which is a CGI script for exposing gemplex over Gemini, and gpctl which provides some utilities.

You will need a postgres database before running Gemplex. See the README in the db sub-directory for more details.

#gemplex executable

The gemplex executable accepts one or more of the following sub-commands:

  • crawl: Runs the crawler.
  • rank: Periodically performs the Page Rank algorithm on the URLs and links stored in the database and stores the results in the database.
  • index: Periodically indexes the contents of the database. At every point in time one of two indices (named "ping" and "pong") is active. Upon each re-indexing, the indexer switches the active index which is used by the search daemon.
  • search: Starts the search daemon which is normally accessed by the CGI script.

You can also pass the all pseudo-command to run all sub-commands at the same time.

The gemplex executable can read its configuration from a toml formatted config file. You can pass the address to this file using the -config flag.

#gpcgi executable

This executable is normally run using a CGI-capable Gemini server (for example, Hodhod which is used to host the service at gemplex.space).

For testing purposes, you can pass the -serve argument to gpcgi which causes a development Gemini server to be launched.

#gpctl executable

This executable provides a number of utilities to manage and monitor a Gemplex installation. The following sub-commands are available:

  • addseed: Add a URL to the database.
  • delhost: Delete all URLs and links for a given hostname (that are not referenced by any other rows) from the database.
  • index: Indexes the database contents.
  • pagerank: Updates URL/host rankings in the database.
  • reparse: Re-parses all the pages stored in the database and extracts metadata from them (like title, language, etc.) and stores them back to the database. This can be useful if a change is made to the parsing routines and we want it applied back to the content that is already crawled and stored.
  • url: Displays information about a given URL.