I have been a long time user (and fan!) of the excellent Harbor Freight Coupon Database website, but I wanted to improve my (previously manual) coupon searching and monitoring process.
This script is the result.
It checks that website daily (or on specified interval) and then notifies you (via XMPP message) when coupons become available for whatever lot number(s) you are interested in.
cron). Although I suppose you could run it manually...
This is a (relatively) simple bash script that first searches through your
todo.txt file for a certain (configurable) keyword and one or more Harbor Freight lot numbers. If no lot numbers (or keywords) are specified in the
todo.txt file, that's it and the script exits.
todo.txtfile, or don't know what that is, see here.
Otherwise, if there are matches, the hfqpdb.com website will be downloaded (with
wget) and then searched for any of the lot numbers you have specified.
After that (and whether there were matches or not), you will receive an XMPP message telling you what happened and also some feedback like whether the web page downloaded successfully or not, as well as the results of a 'regex test.'
hf-price-check script running on <hostname>, downloading HFQPDB.com and searching it for 39 lot number(s)... Web page downloaded successfully. Lot no. regex test returned 100 results (if 0, the website probably changed and broke our regex). Searched for 39 lot numbers, but no hits.
This 'regex test' is the total number of lines (results) that our lot number regex matched on the entire downloaded coupon page of hfqpdb.com. We use this as a check of the regex itself (as oppsed to the particular lot numbers you may be looking for). If this returns zero, the website probably changed and our regex broke. In that case, please create a ticket or maybe even better send an email to the mailing list to bring it to everyone's attention. If I end up with any subscribers on that list, maybe I will even publish if and when I update / fix things.
clone this repository somewhere, or simply copy the files. Cloning will allow you to update easier in the future, should the script break or otherwise be necessary (i.e., with a simple
git pull). The default settings (and examples) assume you have cloned this project directory to
hfpc.conf to your own settings:
test_web_page_result_file. There is a
sampleN.html file included within the repo, make sure this variable points to it.
testing is set to
yes. This is to make sure everything is working properly before you start hammering hfqpdb.com with requests.
Set other variables accordingly.
Make sure you install and configure all Dependencies if you did not do so already. Test and make sure they are all working. That's out of scope for this README.
Because there are a lot of moving parts here (
sendxmpp, the script itself, etc.) do us all a favor and do a few test runs and make sure everything is working correctly with the
sampleN.html test web page:
Make sure you can send / receive XMPP messages properly.
Make sure your
todo_txt_file is found, and any lot numbers you put in there are recognized by the script.
Add some lot numbers that you find in the sample web page to your
todo.txt file and make sure they hit.
If you have any trouble at this stage, I guess send an email to the mailing list and we can see about helping and/or improving the docs.
Only after you are satisfied with all of above, change
no. Then try (only once!) to run it live, to make sure everything works.
Finally (optionally) set up
cron to run the script once daily (or as often as you want (within reason!)). As always with
cron, set it once initially like 5 minutes from now and make sure it fires and everything works before you set it for the middle of the night (or early morning, or whatever).
Last but certainly not least, thanks to the guy who put up hfqpdb.com and keeps it running! Without him, we wouldn't be having any Nice Things(TM) at all in the first place!
FWIW, I did reach out to the guy before I published this and told him about my script and asked him if he would mind that I share it with other people (in addition to my own use). He was OK with it, so here we are.
Maybe even consider throwing him a few bucks so he can buy himself a beer / coffee. Or click through on some of his links (I think he is ad supported), but only if you plan on actually buying something, anyway.[^1]
The guy running hfqpdb.com seems like a pretty swell guy, and he is doing us all a favor (see previous section). So please don't ruin this by being a dumb ass and hammering his site with requests. For example, I run my script only once per day. With the length of time these coupons are usually valid, that should give you plenty of notice to avail yourself of any opportunities that may come of it. In fact, you could probably get away with running this script only once every 2 or even 3 days.
todo.txt is a simple text file format that is great for keeping lists like TODOs or shopping lists.
I designed this script to integrate with todo.txt, because I already use it to keep my shopping lists. So it's very handy to just add this price monitoring feature where I am already keeping my list of things to pick up at Harbor Freight.
However if you don't use
todo.txt, you can simply create a plain text file like the following, and set the variable
todo_txt_file to point at it. On the beginning of each line, use whatever you have set for
todo_txt_keyword (here we use the default) followed by one or more lot numbers you are interested in, separated by spaces:
+HFPriceWatch: 11111 22222 +HFPriceWatch: 33333 +HFPriceWatch: 44444
Why more than one on the same line? Some times there can be more than one lot number for the same item. Usually hfqpdb.com is pretty good about listing all of them, however some times I add more than one, just to try and make sure I don't miss something.
In case you are interested...
I wrote this in bash because it was what I was most comfortable with at the time. If I did it today I would probably use Python and something like BeautifulSoup.
The hfqpdb.com website is pretty straightforward, however, and seems to be stable. I have been running this script personally for some months before sharing it, and in that time there have been no changes I am aware of (my script has not broken yet).
There are tons of comments throughout the script, so hopefully the actual (nuts and bolts) "implememtation details" should not need any further explanation.
FWIW, this is the first project I ever pushed up to sr.ht, in fact it's the first project that I ever really shared, publicly, anywhere.
I have a few more things I would also like to share, once I've polished them up and feel they are ready.
[^1]: Apparently clicking thorough on ads but not actually buying anything negatively affects his ad (payment?) rate. I learned this when visiting the HFQPDB.com site recently, where there is now a prominent notice to this effect.