~johnhamelink/elfeed-paywall

Avoid paywalls and retrieve content from a feed entry's link
Balance parens in README
Add documentation on `elfeed-paywall-proxy-url-pattern`
Turn the paywall proxy into a customizable variable.

refs

main
browse  log 

clone

read-only
https://git.sr.ht/~johnhamelink/elfeed-paywall
read/write
git@git.sr.ht:~johnhamelink/elfeed-paywall

You can also use your local clone with git send-email.

#Introduction

Three bug-bears when using RSS feeds with Elfeed are:

  1. The content in the feed is abbreviated
  2. The content in the link is behind a paywall
  3. The content on a website includes lots of distracting cruft which we don't want.

This package solves these three problems:

  1. The content in the feed is replaced by content from the link
  2. The content is retrieved with a User Agent which should trigger the full content
  3. The content is filtered using the same algorithm used by Firefox for its "Reader View"

#Installation

NB: Before you use elfeed-paywall, you'll need rdrview in your $PATH.

;; Add the `paywall' tag to a feed
    (setq elfeed-feeds
          '(("https://www.theregister.com/headlines.atom" tech paywall)))

    (straight-use-package
     '(elfeed-paywall :type git :repo "https://git.sr.ht/~johnhamelink/elfeed-paywall"))

    (require 'elfeed-paywall)

    (defun my-elfeed-transform-entry (entry)
      "Transformation logic for ENTRYs."
      (elfeed-paywall-with-tag
       entry 'paywall
       (lambda ()
         (elfeed-log 'info "Processing Entry %s" (entry-deref (elfeed-entry-title entry)))
         ;; Remove the analytics URL forwarder that is put in front of
         ;; "The Register" articles
         (elfeed-paywall-replace-regexp-in-link
          entry "go.theregister.com/feed/" "")

         ;; Prefix the link for use with https://12ft.io (A direct
         ;; URL, no JS required)
         (elfeed-paywall-add-paywall-proxy entry)

         ;; Visit the entry link, bypass the paywall, and extract the
         ;; content from the page, then replace the content in the
         ;; entry with it
         (elfeed-paywall-extract-from-url entry))

       ;; Delete the tag after running the lambda
       t))

    (add-hook 'elfeed-new-entry-hook #'my-elfeed-transform-entry)

#Configuration

#elfeed-paywall-rdrview-user-agent

The User-Agent String that is passed to `elfeed-paywall-rdr-executable`. The default value is taken from declutter.el.

#elfeed-paywall-rdrview-path

The executable path for rdrview.

#Examples

;; Add the `paywall' tag to a feed
    (setq elfeed-feeds
          '(("https://www.youtube.com/feeds/videos.xml?channel_id=UC0uTPqBCFIpZxlz_Lv1tk_g" emacs video)
            ("https://www.theregister.com/headlines.atom" tech paywall)))

    (straight-use-package
     '(elfeed-paywall :type git :repo "https://git.sr.ht/~johnhamelink/elfeed-paywall"))

    (require 'elfeed-paywall)

    (defun my-elfeed-transform-entry (entry)
      (elfeed-paywall-with-tag
       entry 'paywall
       (lambda ()
         (elfeed-log 'info "Thwarting Paywall: %s" (entry-deref (elfeed-entry-title entry)))
         ;; Remove the analytics URL forwarder that is put in front of
         ;; "The Register" articles
         (elfeed-paywall-replace-regexp-in-link
          entry "go.theregister.com/feed/" "")

         ;; Prefix the link for use with https://12ft.io (A direct
         ;; URL, no JS required)
         (elfeed-paywall-add-paywall-proxy entry)

         (elfeed-log 'debug "Entry Link is now: %s" (elfeed-deref (elfeed-entry-link entry)))

         ;; Visit the entry link, bypass the paywall, and extract the
         ;; content from the page, then replace the content in the
         ;; entry
         (elfeed-paywall-extract-from-url entry))

      ;; Delete the tag after running the lambda
      t)

    ;; Replace Youtube links with their Invidious counterparts.
    ;; This is desirable for privacy and to avoid ads.
    ;; See: https://invidious.io/
    (elfeed-paywall-with-tag
     entry 'video
     (lambda ()
       (elfeed-log 'info "YouTube -> Invidious: %s" (entry-deref (elfeed-entry-title entry)))
       (elfeed-paywall-replace-regexp-in-link
        entry "[w\\.]*youtube\\.com" "yt.artemislena.eu"))))

    (add-hook 'elfeed-new-entry-hook #'my-elfeed-transform-entry)

#Thanks

This project wouldn't have existed without the following projects:

  • Elfeed, the best RSS reader I've ever used.
  • Declutter.el, which provided me with the inspiration to build this package.
  • Rdrview, for converting the content beautifully.