Of course, just doing the tourist thing is a bit boring, so why not hike up to the castle (Stejpan Grad) instead?
It turns out to be quite easy, although you have to know about the "zigzag way" unless you want to go straight uphill all the way, see map. We saw a couple doing this, but it seems nearly impossible without proper footwear, especially getting back down.
Starting from the parking space, where all the tourist buses stop, next to Caffe Bar Alea, you walk north around the hill with the castle on top. Coming up behind the hill you'll have to continue a bit until you see the path straight up to the castle. A little bit after that, the easier path up starts.
You'll have amazing views from the top of the castle of Blagaj. You can even see the cross on Hum. Hum seems a lot smaller looking at it from here in relation to the bigger mountains behind it. It was amazing and totally worth it, you'll run into few tourists.
Getting to and from Blagaj is a bit of a challenge. During the week it is possible to go by bus, but in the weekend there are barely any buses! We ended up taking a taxi for 15 KM to go and 20 KM to get back as the taxi had to come from Mostar first. We asked the nice guy from the parking for help and he ordered a taxi for us. We didn't try hitchhiking, but that seems quite possible there as well, as for walking from Mostar, two way is a bit far, around 30 kilometer, but the bad part is that lots of the way is right along the cars, so not really an option.
Map source: OpenStreetMap.