~poptart/stagnant

199fd521dfcf7f4602c8c68111fcbff2e674c646 — terrorbyte 7 months ago
Started refactoring for public consumption. This will take some time and will absolutely need some squashing
A  => .gitignore +10 -0
@@ 1,10 @@
temp/
build/
util/src/
hmdtohtml
backup/
test/
genz.sh
buildz/
.*.sw*
expr/

A  => README.md +117 -0
@@ 1,117 @@


A pipeline based static "site" generator focused on allowing flexible composability using built in UNIX tools. Fundamentally, this project is simple and only enforces a simple set of rules, filesystem structures, and a small subset of exposed environment variables.

Reasoning
---------

I write my entire life in markdown, and things that aren't in markdown get generators to turn them into markdown for my own consumption. While some static site generators have a common markdown backend I found that often the generated code was hard to pipeline into other outputs, such as Gopher. 

Additionally, my CI of choice is Laminar-CI which takes the opinionated stance of "just use job shell scripts". I found this rather freeing from implementation specific DSLs or janky non-reproducible web interface configuration management.

I wanted something that allowed me to write a simple step-by-step sychronous pipeline using POSIX.1-2008 shell scripts and allow me to write the pipeline steps in any language I desired. Use the tools you already have.

Example Structure
-----------------

The only directories that are fundamental to the creation of a "site" is the `site` directory which contains the initial format of the site and the functions (`stages`) directory that contains the staged pipeline pieces of code. It is also highly suggested to create a `util` directory that contains executables or scripts used during pipelining.

My personal project root looks like this (directories marked with `<` are required for the generator to work by default):

```
.
|-- README.md
|-- build/
|-- stages/	<
|-- gen.sh*
|-- prod/
|-- site/	<
|-- tmpl/
`-- util/	<
```

```
.
|-- build/
|   |-- 1581996075/
|   |-- 1581996127/
|   |-- 1581996133/
|   `-- latest@ -> 1581996133
|-- stages/
|   |-- 00-depends.sh*
|   |-- 01-slides.sh*
|   |-- 02-http.sh*
|   |-- 03-removemd.sh*
|   |-- 04-minify.sh*
|   `-- scripts/
|       |-- clean.sh*
|       `-- deploy.sh*
|-- gen.sh*
|-- prod/
|   |-- 403.html
|   |-- 404.html
|   |-- 50x.html
|   |-- a/
|   |-- d/
|   |-- e/
|   |-- f/
|   |-- favicon.ico
|   |-- favicon.png
|   |-- i/
|   |-- id.html
|   |-- index.html
|   |-- keybase.txt
|   |-- keys.txt
|   |-- n/
|   |-- p/
|   |-- pub.key
|   |-- resume.pdf
|   |-- rss.xml
|   |-- style.css
|   |-- t/
|   `-- talks.html
|-- site/
|   |-- 403.md
|   |-- 404.md
|   |-- 50x.md
|   |-- f/
|   |-- favicon.ico
|   |-- favicon.png
|   |-- i/
|   |-- id.md
|   |-- index.md
|   |-- keys.txt
|   |-- p/
|   |-- pub.key
|   |-- rss.xml
|   |-- t/
|   `-- talks.md
|-- tmpl/
|   |-- footer.html
|   |-- header.html
|   |-- meta.html
|   |-- rss.xml
|   |-- rss_item.xml
|   |-- style.css
|   `-- template.html
`-- util/
    |-- hdev*
    |-- hmdtohtml*
    |-- hosakahashi*
    `-- minify*

``` 

Exposed Variables
-----------------

The following variables are exposed to all running stage executables/scripts and to any generator scripts:

* `_FUNCDIR`
* `_SCRIPTDIR`
* `_BUILDROOT`
* `_UTIL`
* `_PROD`
* `_ID`
* `_BUILD`
* `_DEPLOY`

A  => doc/examples/complex/site/403.md +1 -0
@@ 1,1 @@
# ACCESS DENIED

A  => doc/examples/complex/site/404.md +1 -0
@@ 1,1 @@
# ☠ Page does not exist...

A  => doc/examples/complex/site/50x.md +1 -0
@@ 1,1 @@
# ☠ BAN HAMMER

A  => doc/examples/complex/site/index.md +20 -0
@@ 1,20 @@
//META:title Hosaka Corp - Shell Distributors
//META:description Index of Research and Blog Posts
//META:style /style.css
<div id=post>0x0B <a href=/p/ssh-pki.html>creating a PKI for OpenSSH</a></div>
<div id=post>0x0A <a href=/p/slides.html>hosakahashi: takahashi in pure css</a></div>
<div id=post>0x09 <a href=/p/voidmap-pentest.html>using voidmap for penetration test project management</a></div>
<div id=post>0x08 <a href=/p/ldpreload-hashcat.html>LD_PRELOAD, hashcat, and bad ideas</a></div>
<div id=post>0x07 <a href=/p/riscv-asm.html>RISC-V assembly and shellcode creation series</a>
<ul>
 <li>Part 1: <a href="/p/riscv-asm-1.html">the basics</a></li>
 <li>Part 2: <a href="/p/riscv-asm-2.html">some complexity</a></li>
 <li>...</li>
</div>
<div id=post>0x06 <a href=/p/systemd-user-msf.html>systemd user persistence in metasploit</a></div>
<div id="post">0x05 <a href="/t/ipv6.html">talk: ipv6 for pentesters</a></div>
<div id="post">0x04 <a href="/p/go-spark.html">Go+SPARK</a></div>
<div id="post">0x03 <a href="/p/wireguard-basic.html">wireguard quickstart</a></div>
<div id="post">0x02 <a href="/p/systemd-user.html">abusing systemd user services</a></div>
<div id="post">0x01 <a href="/p/unicode-rot.html">unicode rotation cipher</a></div>
<div id="post">0x00 <a href="/p/tinc-1.1-overview.html">tinc 1.1 overview</a></div>

A  => doc/examples/complex/stages/00-depends.sh +7 -0
@@ 1,7 @@
#!/bin/sh -e
[ ! -z "$_ROOT" ] || exit 1
[ -f "$_ROOT/util/minify" ] || exit 2
[ -f "$_ROOT/util/smu" ] || exit 2
[ -f "$_ROOT/util/hmdtohtml" ] || exit 2
[ -f "$_ROOT/util/hosakahashi" ] || exit 2
[ -d "$_ROOT/site/" ] || exit 2

A  => doc/examples/complex/stages/01-slides.sh +6 -0
@@ 1,6 @@
#!/bin/sh -e
[ ! -z "$_BUILD" ] || exit 2
find "${_BUILD}" -name '*.sent' | while IFS= read -r file
do
       "$_UTIL"/hosakahashi < "$file" > "$(echo "$file" | sed 's/\.sent/\.html/')"
done

A  => doc/examples/complex/stages/02-http.sh +15 -0
@@ 1,15 @@
#!/bin/sh
META="$(cat "$_ROOT"/tmpl/meta.html)"
HEADER="$(cat "$_ROOT"/tmpl/header.html)"
FOOTER="$(cat "$_ROOT"/tmpl/footer.html)"
export TMPL_TITLE="⊕ hosaka corp"
export TMPL_DESCR="Digital voodoo, cyber-witchcraft, and shell conjuring."
export TMPL_STYLE="/style.css"
find "${_BUILD}" -name '*.md' | while IFS= read -r file; do
        # Search for template meta tags
        [ -n "$(grep '//META:style' "$file")" ] && TMPL_STYLE="$(grep '//META:style' "$file" | sed -e 's%//META:style %%')"
        [ -n "$(grep '//META:title' "$file")" ] && TMPL_TITLE="$(grep '//META:title' "$file" | sed -e 's%//META:title %%')"
        [ -n "$(grep '//META:description' "$file")" ] && TMPL_DESCR="$(grep '//META:description' "$file" | sed -e 's%//META:description %%')"
        BODY="$(grep -v '//META:' "$file" | "$_UTIL"/hmdtohtml | "$_UTIL"/minify --type html)"
        echo "$META$HEADER$BODY$FOOTER" | sed -e "s/{{title}}/$TMPL_TITLE/g" -e "s/{{description}}/$TMPL_DESCR/g" -e "s%{{style}}%$TMPL_STYLE%g" > "$(echo "$file" | sed 's/\.md/\.html/')"
done

A  => doc/examples/complex/stages/03-removemd.sh +2 -0
@@ 1,2 @@
#!/bin/sh
find "$_BUILD" -iname '*.md' -exec rm {} \;

A  => doc/examples/complex/stages/04-minify.sh +3 -0
@@ 1,3 @@
#!/bin/sh
"$_UTIL"/minify --type css < "$_ROOT"/tmpl/style.css > "$_BUILD"/style.css
find "$_BUILD" -type f -name '*.html' -exec sh -c 'cat "$1" | $_UTIL/minify --type html -o "$1"' -- {} \;

A  => doc/examples/complex/stages/scripts/clean.sh +6 -0
@@ 1,6 @@
#!/bin/sh -e
NUM="$(find "${_BUILDROOT}"/* -type d -print0 -maxdepth 1 -prune | xargs -0 -I {} basename {} | wc -l | tr -d ' ')"
if [ $NUM -gt 5 ]; then
	#Extra because of the symlink latest
	find "${_BUILDROOT}"/* -type d -print0 -maxdepth 1 -prune | xargs -0 -I {} basename {} | sort -n -r | tail -n +6 | xargs -I {a} find "${_BUILDROOT}"/{a} -type d -maxdepth 1 -prune -exec rm -rf "{}" \;
fi

A  => gen.sh +80 -0
@@ 1,80 @@
#!/bin/sh -e
export _ORIGIN
export _ROOT

_ORIGIN="$(pwd)"
_ROOT=$(dirname "$(readlink -f "$0")")

export _FUNCDIR="$_ROOT/stages"
export _SCRIPTDIR="$_ROOT/stages/scripts"
export _BUILDROOT="$_ROOT/build"
export _UTIL="$_ROOT/util"
export _PROD="$_ROOT/prod"
export _ID=""
export _BUILD=""
export _DEPLOY=0
_ID="$(date +%s)"
_BUILD="$_BUILDROOT/$_ID"

HELPDIAG="[-p]\\n   -p: tag and stage for deployment\\n   -s [script]: list scripts with no arguments, or run a script"

runscript() {
	if [ ! -n "$1" ]; then
		find "$_SCRIPTDIR" -type f -name '*.sh' -print0 | xargs -0 -I {} basename {} | sed 's/\.sh//g'
		exit 0
	fi
	COMM="$(printf "%s/%s.sh" "$_SCRIPTDIR" "$1")"
	if [ ! -f "$COMM" ]; then
		printf "Script command '%s' not found...\\n" "$1" >&2
		exit 1
	fi
	find "$COMM" -type f -print0 | xargs -0 -I {} sh -c "{}"
}

#TODO build locking
while :; do
        case $1 in
                -h|-\?|--help)
                        printf "%s %b\\n" "$0" "$HELPDIAG"
                        exit
                        ;;
		-p)
			_DEPLOY=1
			;;
		-s)
			shift
			runscript "$1"
			exit
			;;
                -?*)
                        printf 'Unknown option: %s\n' "$1" >&2
                        exit
                        ;;
                *)
                        break
        esac
	shift
done

cd "$_ROOT" || exit 3
mkdir -p "$_BUILD"
printf "%s\\n" "$_ID"
cp -r "${_ROOT}"/site/* "$_BUILD"
[ -h "$_BUILDROOT/latest" ] && rm "$_BUILDROOT/latest"
ln -s "$_BUILD" "$_BUILDROOT/latest"
cd "$_BUILD" || exit 3
find "$_FUNCDIR/" -type f -name '[0-9]*-*.sh' -print0 | sort -n -z | xargs -0 -r sh -c '"$@" || exit 255' 

# This is the old order rendering function that was based around staged execution. In reality just making the "stages" numeric makes more sense
#export _ORDER="pre:render:post"
#for f in $(printf "%s\\n" "$_ORDER" | tr ':' '\n'); do 
#	find "$_FUNCDIR/$f" -type f -print0 -name '[0-9]*-*.sh' | xargs -0 -I {} sh -c "{}"
#done 

if [ "$_DEPLOY" -ne 0 ]; then
	[ -d "$_PROD" ] && rm -rf "$_PROD"
	mkdir -p "$_PROD"
	cp -r "$_BUILDROOT/latest/"* "$_PROD"
fi

cd "$_ORIGIN" || exit 3

A  => site/403.md +1 -0
@@ 1,1 @@
# ACCESS DENIED

A  => site/404.md +1 -0
@@ 1,1 @@
# ☠ Page does not exist...

A  => site/50x.md +1 -0
@@ 1,1 @@
# ☠ BAN HAMMER

A  => site/index.md +20 -0
@@ 1,20 @@
//META:title Hosaka Corp - Shell Distributors
//META:description Index of Research and Blog Posts
//META:style /style.css
<div id=post>0x0B <a href=/p/ssh-pki.html>creating a PKI for OpenSSH</a></div>
<div id=post>0x0A <a href=/p/slides.html>hosakahashi: takahashi in pure css</a></div>
<div id=post>0x09 <a href=/p/voidmap-pentest.html>using voidmap for penetration test project management</a></div>
<div id=post>0x08 <a href=/p/ldpreload-hashcat.html>LD_PRELOAD, hashcat, and bad ideas</a></div>
<div id=post>0x07 <a href=/p/riscv-asm.html>RISC-V assembly and shellcode creation series</a>
<ul>
 <li>Part 1: <a href="/p/riscv-asm-1.html">the basics</a></li>
 <li>Part 2: <a href="/p/riscv-asm-2.html">some complexity</a></li>
 <li>...</li>
</div>
<div id=post>0x06 <a href=/p/systemd-user-msf.html>systemd user persistence in metasploit</a></div>
<div id="post">0x05 <a href="/t/ipv6.html">talk: ipv6 for pentesters</a></div>
<div id="post">0x04 <a href="/p/go-spark.html">Go+SPARK</a></div>
<div id="post">0x03 <a href="/p/wireguard-basic.html">wireguard quickstart</a></div>
<div id="post">0x02 <a href="/p/systemd-user.html">abusing systemd user services</a></div>
<div id="post">0x01 <a href="/p/unicode-rot.html">unicode rotation cipher</a></div>
<div id="post">0x00 <a href="/p/tinc-1.1-overview.html">tinc 1.1 overview</a></div>

A  => stages/00-depends.sh +7 -0
@@ 1,7 @@
#!/bin/sh -e
[ ! -z "$_ROOT" ] || exit 1
[ -f "$_ROOT/util/minify" ] || exit 2
[ -f "$_ROOT/util/smu" ] || exit 2
[ -f "$_ROOT/util/hmdtohtml" ] || exit 2
[ -f "$_ROOT/util/hosakahashi" ] || exit 2
[ -d "$_ROOT/site/" ] || exit 2

A  => stages/01-http.sh +15 -0
@@ 1,15 @@
#!/bin/sh
META="$(cat "$_ROOT"/tmpl/meta.html)"
HEADER="$(cat "$_ROOT"/tmpl/header.html)"
FOOTER="$(cat "$_ROOT"/tmpl/footer.html)"
export TMPL_TITLE="⊕ hosaka corp"
export TMPL_DESCR="Digital voodoo, cyber-witchcraft, and shell conjuring."
export TMPL_STYLE="/style.css"
find "${_BUILD}" -name '*.md' | while IFS= read -r file; do
        # Search for template meta tags
        [ -n "$(grep '//META:style' "$file")" ] && TMPL_STYLE="$(grep '//META:style' "$file" | sed -e 's%//META:style %%')"
        [ -n "$(grep '//META:title' "$file")" ] && TMPL_TITLE="$(grep '//META:title' "$file" | sed -e 's%//META:title %%')"
        [ -n "$(grep '//META:description' "$file")" ] && TMPL_DESCR="$(grep '//META:description' "$file" | sed -e 's%//META:description %%')"
        BODY="$(grep -v '//META:' "$file" | "$_UTIL"/hmdtohtml | "$_UTIL"/minify --type html)"
        echo "$META$HEADER$BODY$FOOTER" | sed -e "s/{{title}}/$TMPL_TITLE/g" -e "s/{{description}}/$TMPL_DESCR/g" -e "s%{{style}}%$TMPL_STYLE%g" > "$(echo "$file" | sed 's/\.md/\.html/')"
done

A  => stages/02-removemd.sh +2 -0
@@ 1,2 @@
#!/bin/sh
find "$_BUILD" -iname '*.md' -exec rm {} \;

A  => stages/03-minify.sh +3 -0
@@ 1,3 @@
#!/bin/sh
"$_UTIL"/minify --type css < "$_ROOT"/tmpl/style.css > "$_BUILD"/style.css
find "$_BUILD" -type f -name '*.html' -exec sh -c 'cat "$1" | $_UTIL/minify --type html -o "$1"' -- {} \;

A  => stages/scripts/clean.sh +6 -0
@@ 1,6 @@
#!/bin/sh -e
NUM="$(find "${_BUILDROOT}"/* -type d -print0 -maxdepth 1 -prune | xargs -0 -I {} basename {} | wc -l | tr -d ' ')"
if [ $NUM -gt 5 ]; then
	#Extra because of the symlink latest
	find "${_BUILDROOT}"/* -type d -print0 -maxdepth 1 -prune | xargs -0 -I {} basename {} | sort -n -r | tail -n +6 | xargs -I {a} find "${_BUILDROOT}"/{a} -type d -maxdepth 1 -prune -exec rm -rf "{}" \;
fi