~kota/paste.cf

ref: 27d7fc2a3a8d9d7c3e526d8ed95ff630aa52e7a8 paste.cf/index.html -rw-r--r-- 14.2 KiB
27d7fc2a — Dakota Walsh fix head language and indentations 2 years ago
                                                                                
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
<!DOCTYPE html>
<html lang="en">
<head>
	<meta charset="utf-8">
	<link rel="stylesheet" href="./highlight/highlight.css">
	<script src="./highlight/highlight.pack.js"></script>
	<script>hljs.initHighlightingOnLoad();</script>
	<title>paste.cf</title>
	<style>
		body {
			margin:40px auto;
			max-width:650px;
			line-height:1.6;
			font-size:18px;
			color:#333;
			padding:0 10px;
			font-family: Tahoma, Verdana, Arial, sans-serif;
			background-color:#fafafa
		}

		h1,h2,h3 {
			line-height:1.2;
		}

		code {
			color:rgb(165, 42, 42);
		}

		a {
			color:rgb(82, 157, 204);
			text-decoration: none;
		}

		li {
			margin:20px auto;
		}
	</style>
</head>
<body>
	<h1>Welcome to paste.cf</h1>
	<p>
	This is a public <a href="https://en.wikipedia.org/wiki/Pastebin">pastebin</a>
	server. You may upload files up to 10MB in size and have them shared
	publically. <a href="https://en.wikipedia.org/wiki/File_Transfer_Protocol">FTP</a>
	is used to upload files to the server.
	</p>
	<p>
	The very minimal code needed for this server is open sourced under the
	3-Clause BSD License and can be found further down on this page or <a
	href="https://git.sr.ht/~kota/paste.cf">at this git repo.</a> The full
	license can be found at <a href="./LICENSE.md">paste.cf/LICENSE.md</a>
	</p>

	<h2>Why</h2>
	<p>
	I often paste things (code, text, logs, pics, gifs, etc) I want to
	share with friends, forums, or irc online, but it feels like every time
	I got used to a <a href="https://github.com/ptpb/pb/issues/246">pastebin</a>
	or <a href="https://old.reddit.com/r/rant/comments/6jbyhp/photobucket_has_changed_their_tos_to_not_allow/">imagehosting</a>
	site it went to shit within a year or two. So I figured I could spare a
	few dollars a month on a vps to host my own paste server as that's the
	only way you KNOW it will last.
	</p>
	<p>
	Turns out all the paste server software out there I could find 
	<a href="https://suckless.org/">sucked.</a> A paste server doesn't need 
	js frameworks, big dynamic servers, node, ads, fluff, a web gui, fonts, 
	complex slow clients, or any other bullshit people will invent in the 
	coming years. FTP, an HTTP server, inotify, cron, and
	2 &lt;100 line scripts is the most it should ever be.
	</p>
	<h2>Usage</h2>
	<p>
	Uploading to paste.cf is quite simple. Using your favourite FTP client
	you can <code>ftp paste.cf</code>, <code>cd incoming</code> (the
	public upload directory), and <code>put file.whatever</code> then
	run <code>sha1sum</code> on your file and the public address will be
	<code>paste.cf/yoursha1.extension</code>
	</p>
	<p>
	Your pastes will be deleted eventually. Storage space costs money; if
	you need your files hosted forever read the section below and have your
	own private server running very easily.
	</p>
	<p>
	To make uploading a bit quicker I wrote a tiny client that FTPs the file
	to the server, calculates the hash, and prints what the resulting url
	should be. Basically <code>pcf notes.md</code> will return the (soon to
	be) live url. Then you can download the client <a href="./pcf.py">here</a> or scroll further
	down to read the code. Obviously you can combine <code>pcf</code> with
	other programs to do cool stuff. Like take a screenshot, upload it, and
	put the url in your clipboard. <code>scrot -q 85 scrot.png && pcf
	scrot.png | xclip -in -selection c</code>
	</p>

	<h2>How</h2>
	<p>
	This server runs an FTP server configured to allow public uploads into
	a specific directory. It also runs a simple static web server. That
	public FTP directory is watched for linux inotify updates with <a href="http://inotify.aiken.cz/?section=incron&page=about&lang=en">incrond.</a>
	Incrond is configured to run a python script (found below) when new
	files are placed in the public FTP directory. The script only does a
	few things. First it checks that the file is under 10Mb (the size can
	be specified). Then it caculates the <a href="https://en.wikipedia.org/wiki/Sha1sum">sha1sum</a> 
	of the file and renames + moves the file into the web server's public
	directory (wherever you specify that to be).
	</p>
	<p>
	Additionally the server uses <a href="https://en.wikipedia.org/wiki/Cron">cron</a> 
	to run another script once every hour. This script checks if the disk usage is
	above a certain percentage, if so it deletes uploaded files based on age
	That script can also be found below.
	</p>
	<p>
	With basic computer knowledge you should be able to duplicate this
	server for yourself (and even have your own private version if you'd
	like). Below are basic setup instructions as well as the aforementioned
	scripts.
	</p>
	<ol>
		<li>
			Install a reasonable server operating system. My favourites are <a href="https://alpinelinux.org/">alpine,</a> <a href="https://www.debian.org/">debian,</a> and <a href="https://www.archlinux.org/">arch.</a> Linux 2.6.13+ will make this much easier, but with some effort this can be setup on *BSD.
		</li>
		<li>
			Install <a href="https://ftp.isc.org/isc/cron/">cron,</a> <a href="http://inotify.aiken.cz/?section=incron&page=about&lang=en">incron,</a> <a href="https://security.appspot.com/vsftpd.html">an ftp server</a> (or <a href="https://www.openssh.com/">sftp</a> for a private server), <a href="https://nginx.org/">a web server,</a> and <a href="https://www.python.org/">python 3.</a>
		</li>
		<li>
			Optionally get a domain name to point to your server and use <a href="https://letsencrypt.org/">letsencrypt</a> to get a free https cert.
		</li>
		<li>
			<a href="https://wiki.archlinux.org/index.php/Very_Secure_FTP_Daemon#Anonymous_login">Configure your ftp server to allow anonymous uploads.</a> Or use sftp and <a href="https://www.digitalocean.com/community/tutorials/how-to-set-up-ssh-keys-on-debian-9">setup a user plus keys</a> to make things easier for yourself. You probably wont need to do any web server configuration (other than maybe adding the ssl certs from above). You simply need to know which directory is being served. It is probably <code>/var/www/html/</code> or similar.
		<li>
			Create a user on your server to run the scripts. On debian <code>adduser --no-create-home paste</code> will do that.
		</li>
		<li>
			Make sure incrond and crond are running on your system and enabled at startup. On debian they are enabled and running if you installed them with apt, you can verify with <code>systemctl status crond</code>, figure out how the init system on your operating system works and this should be pretty simple.
		</li>
		<li>
			Add the <a href="./rename.py">rename.py</a> script to your system and make sure it's marked executable <code>chmod +x rename.py</code>. Do the same thing for remove.py.
		</li>
		<li>
			Login as your paste user, <code>su paste</code>, then edit your incron tab with <code>incrontab -e</code> so that it says something like <code>/public/ftp/directory/here/ IN_CLOSE_WRITE,IN_MOVE_TO /path/to/rename.py --file $# --hold /public/ftp/directory --web /public/web/directory --max 10</code>. Read <code>incrontab(1)</code>, <code>incrontab(5)</code>, and <code>rename.py --help</code> for more info. If you we're unable to run <code>incrontab -e</code> you probably need to add your username to <code>/etc/incron.allow</code>
		</li>
		<li>
			Now for <a href="./remove.py">remove.py</a>, edit the script so that the <code>web_dir</code> variable at the top points to the webdirectory with all the pictures. Then you can simply place the script in <code>/etc/cron.hourly/</code> if that doesn't exist on your system you'll need to also edit your cron tab to run that script once per hour.
		</li>
		<li>
			Consider donating to one or more of the open source projects you used to build your paste server, without them much of the internet would not exist.
		</li>
	</ol>
	<h3><a href="./rename.py">rename.py</a></h3>
	<pre><code>
#!/usr/bin/env python3
# rename.py version 1.0
# Copyright (C) 2019 Dakota Walsh

import os	  # file operations
import sys	  # system operations
import argparse   # argument parsing
import hashlib	  # calculate hash

def tooBig(old_path, max_size):
	# check is the file is over the max size
	if (os.path.getsize(old_path) <= (max_size*1024*1024)):
		return False
	else:
		return True

def getHash(hold, f_name):
	# return a hash of the file in question with the extension
	root = ''
	ext	 = ''
	# however first we'll get the extension if there is one
	if '.' in f_name:
		root, ext = os.path.splitext(f_name)
	f_path = os.path.join(hold, f_name)
	# calculate the hash of the file
	f_hash = hashlib.sha1(open(f_path,'rb').read()).hexdigest()
	return(f_hash + ext)

def getArgs():
	# get the arguments with argparse
	arg = argparse.ArgumentParser(description="rename.py")
	arg.add_argument('--file', '-f', required=True, help='The file which incrond detected.')
	arg.add_argument('--hold', '-d', required=True, help='The directory which was watched.')
	arg.add_argument('--web', '-w', required=True, help='The web directory.')
	arg.add_argument('--max', '-m', help='The max file size in MB. If no max size is set there will be no max size.')
	return arg.parse_args()

def checkDirs(hold, web):
	# check that the directories exist
	if not (os.path.isdir(hold)):
		print ("ERROR: Hold Directory not found! Check your spelling and ensure the directory exists.")
		sys.exit(1)
	if not (os.path.isdir(web)):
		print ("ERROR: Web Directory not found! Check your spelling and ensure the directory exists.")
		sys.exit(1)

def main():
	# get the passed arguments
	arguments = getArgs()
	old_name  = arguments.file
	hold	  = arguments.hold
	web	  = arguments.web
	old_path  = os.path.join(hold, old_name)

	# make sure the hold and web directories exist
	checkDirs(hold, web)

	# check max size
	if arguments.max:
		max_size = int(arguments.max)
		if tooBig(old_path, max_size):
			os.remove(old_path)
			sys.exit(0)

	# calculate sum and rename file
	new_name = getHash(hold, old_name) # can't use old_path due to ext check
	new_path = os.path.join(web, new_name)
	os.rename(old_path, new_path)

if __name__ == '__main__':
	main()
	</pre></code>
	<h3><a href="./remove.py">remove.py</a></h3>
	<pre><code>
#!/usr/bin/env python3
# remove.py version 1.0
# Copyright (C) 2019 Dakota Walsh

import os  # file operations
import sys # system operations

def sorted_ls(path, ignore):
	# sort the web_dir by age
	mtime = lambda f: os.stat(os.path.join(path, f)).st_mtime
	by_age = list(sorted(os.listdir(path), key=mtime))
	for i in ignore:
		for ii in by_age:
			if ii == i:
				by_age.remove(ii)
	return by_age

def main():
	# public web directory with the images
	web_dir = "/var/www/html/" # EDIT THIS FOR YOUR SERVER
	max_disk_percent = 0.80 # Max disk usage percentage out of 1
	# add any files in the dir which should be skipped to the ignore list
	ignore = ["highlight",  "humans.txt",  "index.html",  "LICENSE.md",  "remove.py",  "rename.py"]
	# determine the current disk usage
	stat  = os.statvfs(web_dir)
	total = stat.f_blocks*stat.f_bsize
	free  = stat.f_bfree*stat.f_bsize
	used  = 1-(free/total)
	# if it's more than the max disk percent remove oldest file
	while (used > max_disk_percent):
		oldest_file = sorted_ls(web_dir, ignore)[0]
		oldest_path = os.path.join(web_dir, oldest_file)
		os.remove(oldest_path)
		stat  = os.statvfs(web_dir)
		total = stat.f_blocks*stat.f_bsize
		free  = stat.f_bfree*stat.f_bsize
		used  = 1-(free/total)

if __name__ == '__main__':
	main()
	</pre></code>
	<h3><a href="./pcf.py">pcf.py</a></h3>
	<pre><code>
#!/usr/bin/env python3
# pcf.py version 1.0
# Copyright (C) 2019 Dakota Walsh

import os         # file operations
import sys        # system operations
import ftplib     # ftp operations
import argparse   # argument parsing
import hashlib    # calculate hash

def getHash(f_name):
	# return a hash of the file in question with the extension
	root = ''
	ext	 = ''
	# however first we'll get the extension if there is one
	if '.' in f_name:
		root, ext = os.path.splitext(f_name)
	# calculate the hash of the file
	f_hash = hashlib.sha1(open(f_name,'rb').read()).hexdigest()
	return(f_hash + ext)

def checkFile(upload_file, max_file):
	# check that the directories exist
	if not (os.path.isfile(upload_file)):
		print ("ERROR: File not found! Check your spelling and ensure the file exists.")
		sys.exit(1)
	if not (os.path.getsize(upload_file) <= (max_file)):
		print ("ERROR: File too large. If this is a mistake, edit pcf.py to change max_file.")
		sys.exit(1)

def getArgs():
	# get the arguments with argparse
	arg = argparse.ArgumentParser(description="pcf.py")
	arg.add_argument('file', metavar='F', help='The file which should be uploaded.')
	return arg.parse_args()

def main():
	# get the passed arguments
	arguments    = getArgs()
	upload_file  = arguments.file
	server_addr  = "paste.cf" # the server's address
	server_pub   = "incoming" # public subdirectory on the server
	max_file     = 10*1024*1024 # error if the file is over this size

	# check that the file exists and is less than max_file
	checkFile(upload_file, max_file)

	# login to paste.cf and change into the public directory
	ftp = ftplib.FTP(server_addr, "anonymous", "pcfclient")
	ftp.cwd(server_pub)

	# upload the file then close the connection
	ftp.storbinary("STOR " + upload_file, open(upload_file, 'rb'))
	ftp.quit()

	# calculate the sha1sum and print the url
	new_url = "https://" + server_addr + "/" + getHash(upload_file)
	print(new_url)

if __name__ == '__main__':
	main()
	</pre></code>
	<h2>Privacy and Permanency</h2>
	<p><b>
	Do not upload anything which must remain private to paste.cf as
	everything that is uploaded is public.
	</b></p>
	<p>
	Additionally anything you upload may be removed at any time so do not
	rely on paste.cf as your only copy of something important. The fact
	that things can be removed does not guarantee their removal. In fact
	virtually any uploaded file could be crawled and archived elsewhere.
	</p>
	<p>
	Finally it should be noted that the FTP server running on paste.cf is
	configured to log IPs for each uploaded file. Meaning if illegal content
	is uploaded and the authories send me a letter I'm handing them your
	IP. I'm a broke ass college student and I'm simply not interested in
	fighting your legal battles for you.
	</p>
	<p>
	<b>You have been warned.</b> If you need to upload something private or
	permanent follow the instructions above and setup your own paste server.
	A server just like this one can be rented for $5 a month and give you
	and your mates plenty of bandwidth and storage.
	</p>
</body>
</html>