~monsieuricon/grokmirror

b5ffa134 — Konstantin Ryabitsev 7 months ago master
pi-piper: fix crasher on "d" messages

When we have a "d" operation in the public-inbox repository (a message
removal from the index), we aren't properly catching the exception and
the hook dies. Properly deal with this situation and just move on to the
next commit.

Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
ceb53836 — Konstantin Ryabitsev 11 months ago
Don't force url to contain hostname

If we force the url= entry to contain the hostname, his makes it look
extra busy on the wwwlisting page.

Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
373c2b89 — Konstantin Ryabitsev 11 months ago
Fix crash on trying to remove repository we never fsck'd

If a repository is gone from manifest before we evern managed to fsck
it, we may get a crash on trying to remove it from status.

Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
e5e79d0a — Konstantin Ryabitsev 11 months ago
Update manpage for grok-pi-indexer

Added -t and -p flags for when public-inbox databases are separate from
git trees.

Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
02d020c0 — Konstantin Ryabitsev 11 months ago
Set default boost to 1 when boosts are used

Make the configuration file more consistent when using boost values for
specific list-ids.

Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
ed04235f — Konstantin Ryabitsev 1 year, 1 day ago
Add --toplevel and --pi-toplevel to pi-indexer

It is useful to be able to keep public-inbox databases separate from the
git repositories. Passing --pi-toplevel should allow us to do this.

Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
60836f0a — Konstantin Ryabitsev 1 year, 5 days ago
Fix pull_threads logic error

The logic error caused pull_threads to always be 1 if pull_threads is
set in the config file.

Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
52e6d95d — Konstantin Ryabitsev 1 year, 12 days ago
Adjust config and mangpage for grok-pi-indexer

We've changed a few things around, so reflect this in the documentation.

Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
ddb227b5 — Konstantin Ryabitsev 1 year, 12 days ago
Init all new inboxdirs before indexing them

Indexing takes a long time, so it's possible that something will go
wrong before grokmirror is done. Init-ing all new inboxdirs first allows
us to more easily recover from this situation.

Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
fdd92c7c — Konstantin Ryabitsev 1 year, 12 days ago
Rework how we pass extra flags to public-inbox

It was too messy to pass double-dashed parameters via hook
configuration, so instead we'll just use -j, -L, and --no-fsync global
flags and set index* parameters via the config file itself.

Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
e56929df — Konstantin Ryabitsev 1 year, 13 days ago
Add ability to pass native public-inbox flags

Instead of duplicating all possible public-inbox flags, add ability to
set --pi-index-flags, --pi-init-flags, and --pi-extindex-flags so they
are passed along verbatim to proper commands.

Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
77c62d68 — Konstantin Ryabitsev 1 year, 14 days ago
Fix missed post_work_hook invocation

There are two places where we can be done, depending on the invocation,
so there was a situation where the post_work hook wasn't firing on
completed work.

Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
4293e802 — Konstantin Ryabitsev 1 year, 15 days ago
Document changes for 2.1 (thus far)

Will probably be a few more things, but nothing drastic, I don't think.

Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
4692feb6 — Konstantin Ryabitsev 1 year, 15 days ago
Document pi-indexer and set version to 2.1.0-dev

We've added two important hooks, so let's call it 2.1.x and roll back
2.0.x back to what it was just after 2.0.8.

Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
3047754f — Konstantin Ryabitsev 1 year, 19 days ago
More work to support public-inbox processing

- add another hook post_work_complete_hook, so that we can call
  public-inbox-extindex once per update cycle, as is recommended
- adjust grok-pi-indexer to handle three different hook invocations

Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
6dc859cd — Konstantin Ryabitsev 1 year, 19 days ago
Better logic for auto-detecting pull_threads

On a single-CPU vm, we ended up with 0 pull_threads, which resulted in
the pull process getting stuck.

Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
967f9abd — Konstantin Ryabitsev 1 year, 21 days ago
Add support for boost and listid priorities

The latest public-inbox master adds support for boost values and passing
extra configuration parameters via -c during the public-inbox-init
stage. Use both of these to stop using git-config directly and just let
init write the configuration for us.

Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
0cebec3a — Konstantin Ryabitsev 1 year, 23 days ago
Support reindexing and setting description

We may want to trigger a reindex without needing to run manual
public-inbox commands.

Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
07745fc8 — Konstantin Ryabitsev 1 year, 23 days ago
Up version to 2.0.10-dev

Maybe this will become 2.1, not sure.

Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
6be3a636 — Konstantin Ryabitsev 1 year, 23 days ago
Initial grok-pi-indexer implementation

When mirroring public-inbox repositories, add support to run
public-inbox-init and public-inbox-index as hooks.

Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
Next