Don't pass the FILE: prefix to llvm-symbolizer
Make llvm-symbolizer errors non-fatal
Make llvm-symbolizer errors non-fatal
Alys is a Crystal memory usage tracer, tracking memory allocs, re-allocs, and frees for later analysis.
THIS IS ALPHA QUALITY SOFTWARE!
Add the dependency to your shard.yml
:
dependencies:
alys:
git: https://git.sr.ht/~refi64/alys
Run shards install
require "alys"
Alys.setup_from_env
Now, to enable tracing run your application with:
ALYS_TRACING=file crystal myapp.cr
This will trace to a file TIME.alys
, where TIME is an ISO 8601 timestamp. To
change the file's name, use:
ALYS_TRACING=file:myfile.alys ./myapp
alys_converter
The .alys
file is in a custom binary format, but it can be converted into
JSON, Go pprof, folded stacks (for use with flamegraph), or a direct flamegraph
via:
# NOTE: passing `--symbolize` with the binary is *required* on systems where
# ALYS_BACKTRACE_TYPE=addr by default (see below).
# Outputs myfile.alys.json:
bin/alys_converter --symbolize myprog myfile.alys
# You can rename the output file:
bin/alys_converter --symbolize myprog myfile.alys myfile.json
# and/or format & indent the JSON file:
bin/alys_converter --symbolize myprog --indent myfile.alys
bin/alys_converter --symbolize myprog --indent myfile.alys myfile.json
# Outputs myfile.pb.gz (pprof format):
bin/alys_converter --symbolize myprog -f pprof myfile.alys
# You can rename the output file:
bin/alys_converter --symbolize myprog -f pprof myfile.alys myfile.pb.gz
# Outputs myfile.folded (folded stacks):
bin/alys_converter --symbolize myprog -f folded-stacks myfile.alys
# You can rename the output file:
bin/alys_converter --symbolize myprog -f folded-stacks myfile.alys myfile.folded
# Outputs myfile.svg (flamegraph, via inferno-flamegraph):
bin/alys_converter --symbolize myprog -f inferno-flamegraph myfile.alys
# You can rename the output file:
bin/alys_converter --symbolize myprog -f inferno-flamegraph myfile.alys myfile.svg
(Note that .alys files are only compatible with an identical alys_converter version!)
alys_converter's JSON format is simply an array of event objects like so:
{
"id": 29,
"time": 1.009839517,
"kind": "alloc",
"addr": 281471443894272,
"size": 96,
"stack": [
{
"ip": 4721576,
"line": 279,
"file": "/home/ryan/code/alys/src/alys.cr",
"name": "record_alloc"
},
...
]
}
id
is a unique ID assigned to each allocation, with reallocation & free
calls using the same ID as the original allocation. In other words, you can
use this to link together allocation and free calls.time
is the number of seconds since program start that this occurred.kind
is one of alloc
, realloc
, or free
.addr
is the memory address that was allocated or freed.size
is the number of bytes allocated.realloc
events, both of the above point to the new size and address,
and two additional keys store the previous values, prev_addr
and
prev_size
.stack
is an array of stack frames, each containing (only ip
is guaranteed
to be present):
ip
is the instruction pointer of this frame.file
is the full filenameline
is the line numbername
is the function nameYou can see some (ugly) example usage of the JSON file for analysis at this Colab notebook)
pprof files can be visualized with Google's pprof CLI tool. Four sample indexes are available (with similar meanings as to Go's):
alloc_objects
: number of objects allocated over the course of the programalloc_space
(default): amount of bytes allocated over the course of the
programinuse_objects
: number of objects allocated at the time of program
terminationinuse_space
(default): amount of bytes allocated at the time of program
terminationMost likely, you'd want to analyze the data with the pprof web UI; to start it
on PORT
, use:
pprof -http localhost:PORT myfile.pb.gz
Folded stacks files are usually converted to flamegraphs
flamegraphs; you can also go
straight from .alys -> flamegraph via the inferno-flamegraph
format (requires
Inferno to be installed). In the latter
case, additional options can be passed to the inferno-flamegraph
CLI via
--inferno-opt
:
# Pass --flamechart and --title=test to inferno-flamegraph:
bin/alys_converter -f inferno-flamegraph myfile.alys \
--inferno-opt flamechart --inferno-opt title=test
By default, Alys will also capture a backtrace on each allocation or
reallocation, which can result in significant slowdowns. You can control this
via ALYS_BACKTRACE_TYPE
:
ALYS_BACKTRACE_TYPE=none
will capture no backtrace at all.ALYS_BACKTRACE_TYPE=addr
will create backtraces that only contain the
function's memory addresses (i.e. no names, files, or line numbers). A
backtrace like this, however, is not very useful, so alys_converter
can
resolve them when running alys_converter
via:
bin/alys_converter --symbolize ./myapp myfile.alys
-Wl,-no_pie
. M1 macs do not
support this option.
PIE randomizes the memory location of the executable at runtime, so it's
impossible to know what was where after the fact.llvm-symbolizer
must be installed.
This is the default on non-macOS systems.ALYS_BACKTRACE_TYPE=name
will create backtraces that contain the function
names but no files or line numbers. This is usually only slightly slower than
ALYS_BACKTRACE_TYPE=addr
, and it supports position-independent executables
on macOS, but the symbol names aren't as detailed as using alys_converter
's
symbolization.
This is the default on macOS.ALYS_BACKTRACE_TYPE=full
will create backtraces that contain the function
names, as well as the source file paths and line numbers.TODO: add support for & document flipping Alys on / off at runtime
Instead of writing an event for every trace, you can have Alys write an event
for every N bytes allocated by setting ALYS_SAMPLE_INTERVAL=bytes:N
. This
should significantly increase the performance of allocations at the cost of
some granularity.
Gathering backtraces on release builds is significantly slower than debug builds!
This is because release builds omit the frame pointer / base pointer, which is used to quickly go back up the call stack. Without this, backtrace creation needs to look into the embedded unwind information, which is significantly slower.
These were gathered by running the Lucky website locally, then timing:
wget --recursive --max-redirect 0 127.0.0.1:5000
In other words, this would run the Lucky website and then download every single page on the site.
Build Type | Tracing Enabled? | Backtrace Type | Backtrace Limit | Time |
---|---|---|---|---|
debug | N | N/A | N/A | 1.2s |
debug | Y | none | N/A | 2.7s |
debug | Y | addr | 5 | 7s |
debug | Y | addr | 10 | 10s |
debug | Y | addr | unlimited | 23s |
debug | Y | name | 5 | 9s |
debug | Y | name | 10 | 14s |
debug | Y | name | unlimited | 33s |
debug | Y | full | 5 | 1m8s |
debug | Y | full | 10 | 3m25s |
debug | Y | full | unlimited | 8m47s |
release | N | N/A | N/A | 0.9s |
release | Y | none | N/A | 2.2s |
release | Y | addr | 5 | 34s |
release | Y | addr | 10 | 1m16s |
release | Y | addr | unlimited | 1m21s |
release | Y | name | 5 | 36s |
release | Y | name | 10 | 1m18s |
release | Y | name | unlimited | 1m25s |
release | Y | full | 5 | 8m4s |
release | Y | full | 10 | timeout |
release | Y | full | unlimited | timeout |
On Linux systems, the default glibc unwind implementation is significantly
slower than alternatives, which will result in significant performance loss if
backtraces are enabled. As a result, Alys will try to configure your program
to use LLVM's libunwind if available, printing a warning when shards
is run
otherwise.
IMPORTANT NOTE: There are multiple implementations of libunwind available that may not be compatible. In particular, there's nongnu libunwind, which is incompatible with the standard libunwind and will not work. Alys will automatically avoid using this libunwind, but you may have some confusion if you think you installed "libunwind" but Alys is not using it.
After running any of the below steps, you can run
lib/alys/tools/detect_libunwind.sh
to ensure that LLVM libunwind is detected.
Alys can build and use its own local copy of libunwind. In order to do this, just run:
$ lib/alys/tools/build_libunwind.sh
$ sudo dnf install llvm-libunwind
# Find the libunwind version corresponding to the Debian release's primary
# LLVM package.
$ LIBUNWIND_VERSION=$(apt-cache depends llvm | egrep -o 'llvm-[0-9]+' | cut -d- -f2)
$ sudo apt install libunwind-$LIBUNWIND_VERSION
Add the official LLVM APT repo, then run:
$ sudo apt install libunwind-13
$ sudo apk add llvm-libunwind
TODO
Please see the guide for submitting patches on
git.sr.ht. (If you
choose to use git send-email
, the patches should be sent to
~refi64/alys-devel@lists.sr.ht.)ODO