Supplemental materials for: Challenging common interpretability assumptions in feature attribution explanations
add proper citation
add arxiv link
update hosted models


browse  log 



You can also use your local clone with git send-email.

#Challenging common interpretability assumptions in feature attribution explanations

As machine learning and algorithmic decision making systems are increasingly being leveraged in high-stakes human-in-the-loop settings, there is a pressing need to understand the rationale of their predictions. Researchers have responded to this need with explainable AI (XAI), but often proclaim interpretability axiomatically without evaluation. When these systems are evaluated, they are often tested through offline simulations with proxy metrics of interpretability (such as model complexity). We empirically evaluate the veracity of three common interpretability assumptions through a large scale human-subjects experiment with a simple "placebo explanation" control. We find that feature attribution explanations provide marginal utility in our task for a human decision maker and in certain cases result in worse decisions due to cognitive and contextual confounders. This result challenges the assumed universal benefit of applying these methods and we hope this work will underscore the importance of human evaluation in XAI research.


make docker         (re)build the Docker image for the experiment 
make repl           IPython REPL mirroring environment of the experiment
make serve          Run local web server of app for the experiment
make download       Download fitted `brms` models used in the paper (~2GB)         
make models         Fit `brms` models (only fits non-existent models)


Parameters Convergence Fit
Fixed summary.txt plots.pdf brms.rds (500 MB)
1PL summary.txt plots.pdf brms.rds (700 MB)

#Getting in touch

For any questions, comments, suggestions, or feedback on anything in here or in the paper, we have a mailing list meant to serve as something of a discussion section or forum: https://lists.sr.ht/~hyphaebeast/research

You may subscribe to the list by emailing ~hyphaebeast/research+subscribe@lists.sr.ht. You may unsubscribe with +unsubscribe. You may post new threads to this list by writing to the address with no + command.

Before posting, please read:

#Local experiment


The only hard requirement to run a local instance of the experiment is Docker (you don't even need to download/clone the source git repository).

# run experiment @ http://localhost:5001
docker run -p 5001:5001 -e PORT=5001 hyphaebeast/challenging-xai:experiment

Simply run the above command (or make serve) and visit http://localhost:5001 in your web browser to walk through an anonymous and ephemeral version of the experiment.


If you would like to replicate/modify/remix/edit/etc. the experiment, the Docker container is configured to enable direct deployments to Heroku (or any other container service).

NOTE: the default configuration for the container is to use an ephemeral local Postgres instance. If you are deploying to Heroku and would like to persist data, you will need to configure a hosted database like Heroku Postgres (i.e. heroku addons:create heroku-postgresql:hobby-dev -a [app name])


# get source for experiment
git clone https://git.sr.ht/~hyphaebeast/challenging-xai
cd challenging-xai/experiment

# login to the Heroku container registry
heroku container:login

# create a new Heroku app for experiment
# should output a Heroku haiku as the app name 
# (something like: salty-fortress-4191)
heroku create

# build Docker container and add to Heroku registry
heroku container:push web -a [app name]

# deploy container to live web app
heroku container:release web -a [app name]

# open experiment in webpage
heroku open -a [app name]

To remix or modify the experiment (like even just the color of the charts), simply edit the files in challenging-xai/experiment then rebuild and deploy the container:

# must be run in the same directory as the Dockerfile (i.e. challenging-xai/experiment)
heroku container:push web -a [app name]
heroku container:release web -a [app name]

If you are deploying the container to run an actual experiment and need to persist data/results, point the DATABASE_URL environment variable to a hosted database. If you are using Heroku Postgres this is automatically set for you and no further action is necessary (simply just heroku addons:create heroku-postgresql:hobby-dev -a [app name]).

#Software Colophon




    title={Challenging common interpretability assumptions
        in feature attribution explanations}, 
    author={Jonathan Dinu and Jeffrey Bigham and J. Zico Kolter},


Zero clause BSD (effectively public domain). Do whatever you want....