Fix: (X-Drive-Version header) Always return latest version of drive
Fix: (fileHistory) Catch only BLOCK_NOT_AVAILABLE error
Change: (serveHead) Use db.peek() to check directory existence
Implementation of Fetch that uses the Hyper SDK for loading p2p content
import makeHyperFetch from 'hypercore-fetch'
import * as SDK from 'hyper-sdk'
// Create in-memory hyper-sdk instance
const sdk = await SDK.create({storage: false})
const fetch = await makeFetch({
sdk: true,
writable: true
})
const someURL = `hyper://TODO_REAL_URL_HERE_PLEASE`
const response = await fetch(someURL)
const data = await response.text()
console.log(data)
makeHyperFetch({sdk, writable=false, extensionMessages = writable, renderIndex}) => fetch()
Creates a hypercore-fetch instance.
The sdk
argument should be an instance of hyper-sdk.
The writable
flag toggles whether the PUT
/POST
/DELETE
methods are available.
extensionMessages
enables/disables Hypercore Extension Message support which is used for sending extra data to peers on top of hypercore replication streams.
renderIndex
is an optional function to override the HTML index rendering functionality. By default it will make a simple page which renders links to files and folders within the directory.
This function takes the url
, files
array and fetch
instance as arguments.
After you've created it, fetch
will behave like it does in browsers.
Each response will contain a header for the canonical URL represented as a Link
header with rel=canonical
.
There is also an ETag
header which will be a JSON string containging the drive's current version
, or the file's sequence number.
This will change only when the drive has gotten an update of some sort and is monotonically incrementing.
The ETag
representing a file's sequence number represents the version the Hyperdrive was at when the file was added.
Thus you can get the previous version of a file by using hyper://NAME/$/version/${ETAG}/example.txt
.
The X-Drive-Size
header represents the size in bytes that the
requested hyperdrive (the whole drive, not just the requested
file/directory) locally takes up on disk.
The X-Drive-Version
header represents the latest version of the
drive.
If the resource is a file, it may contain the Last-Modified
header if the file has had a metadata.mtime
flag set upon update.
If the resource is a directory, it will contain the Allow
header to
indicate whether a hyperdrive is writable ('HEAD,GET'
) or not
('HEAD,GET,PUT,DELETE'
).
The X-File-Block-Length
and X-File-Block-Length-Downloaded
headers
are integers representing how many blocks a file consists of and how
many of those blocks are already downloaded locally.
fetch('hyper://NAME/example.txt', { headers: {'X-Fully-Replicate': 'db' | 'blobs' | 'all' }})
This will fully replicate the inode db, blob store, or both, when you
pass the X-Fully-Replicate
header with db
, blobs
, or all
value,
respectively. Returns status 400
if there is an error when replicating or status 200
on success.
fetch('hyper://NAME/example.txt', {method: 'GET'})
This will attempt to load example.txt
from the archive labeled by NAME
.
It will also load index.html
files automatically for a folder.
You can find the details about how resolution works in the resolve-dat-path module.
NAME
can either be the 52 character z32 encoded key for a Hyperdrive or Hypercore , or a domain to parse with the DNSLink standard.
fetch('hyper://NAME/example/', {method: 'GET'})
When doing a GET
on a directory, you will get a directory listing.
By default it will return a JSON array of files and folders in that directory.
You can differentiate a folder from files by the fact that it ends with a /
.
You can set the Accept
header to text/html
in order to have it return a basic HTML page with links to files and folders in that directory.
This can be overrided with the renderIndex
option if you want custom index pages.
e.g.
["example.txt", "posts/", "example2.md"]
Files in the directory will be listed under their name, sub-directories will have a /
appended to them.
Alternatively, you can set the Accept
header to application/json; metadata=full
in order to receive the directory contents with full
metadata:
[{
seq: 1,
key: '/example.txt',
value: {
executable: false,
linkname: null,
blob: { byteOffset: 0, blockOffset: 0, blockLength: 1, byteLength: 11 },
metadata: { mtime: 1691832753029 }
}
},
{
key: '/posts/
}]
With Accept: application/json; metadata=full
, each object's key
is
the full path to the file/folder. You can distinguish directories from
files by their lack of metadata.
NAME
can either be the 52 character z32 encoded key for a Hyperdrive or Hypercore , or a domain to parse with the DNSLink standard.
fetch('hyper://NAME/example/?noResolve', {method: 'GET'})
Adding ?noResolve
to a URL will prevent resolving index.html
files and will attempt to load the path as is.
This can be useful for list files in a directory that would normally render as a page.
NAME
can either be the 52 character z32 encoded key for a Hyperdrive or Hypercore , or a domain to parse with the DNSLink standard.
fetch('hyper://localhost/?key=NAME', {method: 'POST'})
In order to create a writable Hyperdrive with its own URL, you must first generate a keypair for it.
NAME
can be any alphanumeric string which can be used for key generation in Corestore.
The response body will contain a hyper://
URL with the new Hyperdrive.
You can then use this with PUT
/DELETE
requests.
Note that this is only available with the writable: true
flag.
fetch('hyper://localhost/?key=NAME', {method: 'GET'})
If you want to resolve the public key URL of a previously created Hyperdrive, you can do this with the GET
method on the key creation URL.
NAME
can be any alphanumeric string which can be used for key generation in Corestore.
The response body will contain a hyper://
URL with the new Hyperdrive.
You can then use this with PUT
/DELETE
requests.
Note that this is only available with the writable: true
flag.
fetch('hyper://NAME/example.txt', {method: 'PUT', body: 'Hello World'})
You can add files to archives using a PUT
method along with a
body
. Note that this is only available with the writable: true
flag.
The body
can be any of the options supported by the Fetch API such as a String
, Blob
, FormData
, or ReadableStream
.
NAME
can either be the 52 character z32 encoded key for a Hyperdrive or Hypercore , or a domain to parse with the DNSLink standard.
The mtime metadata is automatically set to the current time when
uploading. To override this value, pass a Last-Modified
header with a value
set to a date string according to RFC
7231.
An attempt to PUT
a file to a hyperdrive which is not writable will
fail with status 403
.
fetch('hyper://NAME/folder/', {method: 'PUT', body: new FormData()})
You can add multiple files to a folder using the PUT
method with a FormData body.
You can append to a FormData with formData.append('file', content, 'filename.txt')
where fieldname
gets ignored (use something like file
?) the content
can either be a String, Blob, or some sort of stream.
The filename
will be the filename within the directory that gets created.
Note that you must use the name file
for uploaded files.
NAME
can either be the 52 character z32 encoded key for a Hyperdrive or Hypercore , or a domain to parse with the DNSLink standard.
fetch('hyper://NAME/', {method: 'DELETE'})
You can purge all the stored data for a hyperdrive by sending a DELETE
to it's root.
If this is a writable drive, your data will get fully clearned and trying to write to it again will lead to data corruption.
If you try to load this drive again data will be loaded from scratch.
NAME
can either be the 52 character z32 encoded key for a Hyperdrive or Hypercore , or a domain to parse with the DNSLink standard.
fetch('hyper://NAME/example.txt', {method: 'DELETE'})
You can delete a file or directory tree in a Hyperdrive by using the DELETE
method.
NAME
can either be the 52 character z32 encoded key for a Hyperdrive or Hypercore , or a domain to parse with the DNSLink standard.
Note that this is only available with the writable: true
flag.
An attempt to DELETE
a file in a hyperdrive which is not writable
will fail with status 403
.
fetch('hyper://NAME/example.txt', {method: 'POST'})
With the Cache-Control: no-store
header, DELETE
requests remove
the blob for the file specified by the URL from the local filesystem,
but do not touch the hyperdrive metadata. Clearing the local cache of
a hyperdrive file does not require that the hyperdrive be writable. If
URL specifies a directory, its file contents will be recursively
cleared.
Clearing the cache for a file or directory does not clear blobs for
all file versions. To clear the blob for a specific version, send a
POST
request to hyper://NAME/$/version/VERSION_NUMBER/example.txt
.
An attempt to POST
to a file or directory without the
Cache-Control: no-store
header will fail with status 400
.
fetch('hyper://NAME/$/extensions/')
You can list the current hypercore extensions that are enabled by doing a GET
on the /$/extensions/
directory.
This will give you a directory listing with the names of all the extensions.
Note that this requires the extensionMessages: true
flag.
fetch('hyper://NAME/$/extensions/EXTENSION_NAME')
You can list the peers that you are replication with which have registered this extension by doing a GET
to the directory for the extension.
This is also how you can register an extension that hasn't been registered yet.
The list will be a JSON array with objects that contain the fields remotePublicKey
and remoteHost
.
Note that this requires the extensionMessages: true
flag.
fetch('hyper://NAME/$/extensions/', {headers: {'Accept': 'text/event-stream'}})
Using the text/event-stream
content type in the Accept
header will get back an event stream with the extension events.
You can get the browser's EventSource API over hypercore-fetch by using the @rangermauve/fetch-to-eventsource module.
The event
will be the name of the extension you got the data for, the id
(accessible by e.lastEventId
in EventSource) will be set to the ID of the peer that sent it.
Only extension messages that have been queried before via a GET
to the EXTENSION_NAME will be visible in this stream.
There are also two special events: peer-open
which gets emitted whena new peer has connected, and peer-remove
which gets emitted when an existing peer disconnects.
Note that this requires the extensionMessages: true
flag.
fetch('hyper://NAME/$/extensions/EXTENSION_NAME', {method: 'POST', body: 'Example'})
You can broadcast an extension message to all peers that are replicating that extension type with a POST
to the extension's URL.
The body
of the request will be used as the payload.
Please note that only utf8 encoded text is currently supported due to limitations of the event-stream encoding.
Note that this requires the extensionMessages: true
flag.
fetch('hyper://NAME/$/extensions/EXTENSION_NAME/REMOTE_PUBLIC_KEY', {method: 'POST', body: 'Example'})
You can send an extension message to a specific peer by doing a POST
to the extension with their remote public key ID.
The body
of the request will be used as the payload.
Please note that only utf8 encoded text is currently supported due to limitations of the event-stream encoding.
Note that this requires the extensionMessages: true
flag.
fetch('hyper://NAME/$/version/VERSION_NUMBER/example.txt')
You can get older views of data in an archive by using the special /$/version
folder with a version number to view older states.
VERSION_NUMBER
should be a number representing the version to check out based on the ETag
of the root of the archive.
From there, you can use GET
and HEAD
requests with allt he same headers and querystring paramters as non-versioned paths to data.
Note that you cannot PUT
or DELETE
data in a versioned folder.
fetch('hyper://NAME/$/history/example.txt')
You can get a file's version history by sending a GET
request to the
special /$/history
folder. Response body will be a JSON array
containing the history of changes to /example.txt
. Each item in the
history array will be an object with the following properties:
{
exists: Boolean | 'unknown',
blockLengthDownloaded: Number | undefined,
version: Number
...entry (without seq)
}
Requests to /$/history
do not download any content from
the network, but merely return the history metadata which has already
been loaded.
$
, you cannot store files in this folder. If this is a major blocker, feel free to open an issue with alternative folder names we should consider.