Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
139 changes: 139 additions & 0 deletions apps/loopover-ui/content/docs/ams-unattended-scheduling.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,139 @@
---
title: Unattended scheduling & failure alerting
description: Run the miner's scheduled commands -- manage poll and discover -- unattended on cron or systemd, and alert reliably when a run fails.
---

Operational guidance for running the miner's scheduled commands — `manage poll` and `discover` —
unattended on a timer (cron or systemd), and for alerting when a run fails. These are the two
commands most likely to run on a schedule; everything they need is local and they make no
interactive prompts.

<Callout variant="note">
Scope: scheduling + failure alerting for `manage poll` / `discover`. For local-state recovery
see the [AMS operations runbook](/docs/ams-operations-runbook); for deployment layout see the
[AMS deployment guide](/docs/ams-deployment).
</Callout>

## The exit-code contract (what to alert on)

Both commands follow the same convention, so any scheduler can detect a failed run from the exit
code:

<FeatureRow
items={[
{
title: "0 — Success",
description: "The command completed.",
},
{
title: "2 — Failure",
description:
"Invalid arguments, or the run hit an error (network / API / local state). Alert on this.",
},
]}
/>

For scheduled runs, two flags matter:

- `--no-update-check` (or `LOOPOVER_MINER_NO_UPDATE_CHECK=1`) — skip the npm-registry version
nudge so an unattended run never depends on / prints it.
- `--json` — machine-parseable stdout, so an alert handler can attach the structured output.

## cron

<CodeBlock
lang="cron"
code={`# crontab env applies to every job below.
MAILTO=you@example.com
LOOPOVER_MINER_NO_UPDATE_CHECK=1

# Poll a tracked PR every 10 minutes. The || branch fires on any non-zero exit: it logs the failing
# code to syslog AND re-raises it with exit "$status", so the failure stays visible to exit-status
# monitoring instead of being masked by logger's own success.
*/10 * * * * /usr/local/bin/loopover-miner manage poll acme/widgets 42 --json || { status=$?; logger -t loopover-miner "manage poll failed (exit $status)"; exit "$status"; }

# Discover + enqueue candidate work hourly.
0 * * * * /usr/local/bin/loopover-miner discover --search "label:good-first-issue" --json || { status=$?; logger -t loopover-miner "discover failed (exit $status)"; exit "$status"; }`}
/>

Two cron facts to get right here:

<Callout variant="warn" title="MAILTO mails output, not exit status">
cron emails whatever a job writes to stdout/stderr to `MAILTO` — it does not send a message
"because" the exit code was non-zero. A job that fails *silently* (non-zero exit, no output)
produces no mail, so don't rely on `MAILTO` alone as the failure signal.
</Callout>

<Callout variant="warn" title="A bare || logger … hides the failure">
`logger` succeeds (exit 0), so `cmd || logger …` makes the whole cron job exit 0 — any
exit-status-based monitoring then sees success. Capture the code first (`status=$?`) and
re-raise it (`exit "$status"`) as shown, so the real failing code survives.
</Callout>

## systemd (service + timer)

A `oneshot` service plus a timer is the more observable option: `systemctl status` /
`journalctl` capture each run, and `OnFailure=` is a first-class alerting hook.

`loopover-miner-discover.service`:

<CodeBlock
lang="ini"
code={`[Unit]
Description=loopover-miner discover
OnFailure=loopover-miner-alert@%n.service

[Service]
Type=oneshot
Environment=LOOPOVER_MINER_NO_UPDATE_CHECK=1
# A non-zero exit (2) marks the unit failed and triggers OnFailure=.
ExecStart=/usr/local/bin/loopover-miner discover --search "label:good-first-issue" --json`}
/>

`loopover-miner-discover.timer`:

<CodeBlock
lang="ini"
code={`[Unit]
Description=Run loopover-miner discover hourly

[Timer]
OnCalendar=hourly
Persistent=true

[Install]
WantedBy=timers.target`}
/>

Enable with `systemctl enable --now loopover-miner-discover.timer`.

## Alerting on failure

Every option keys on the same exit-code contract (`2` = failure).

**cron.** Append `|| { status=$?; <alert-command>; exit "$status"; }` (as in the cron example
above) — capture `$?` before the alert command runs and re-raise it, so the failure isn't masked.
Substitute `logger` with a webhook `curl`, a PagerDuty/Slack CLI, etc. (`MAILTO` still mails any
output, but is not a reliable signal for a silent failure — see the cron note above.)

**systemd.** `OnFailure=loopover-miner-alert@%n.service` runs a templated alert unit on any
non-zero exit. A minimal alert unit:

<CodeBlock
lang="ini"
code={`# loopover-miner-alert@.service
[Service]
Type=oneshot
ExecStart=/usr/local/bin/notify-failure "loopover-miner unit %i failed"`}
/>

**Wrapper script.** For any scheduler, wrap the command and preserve its exit code:

<CodeBlock
lang="bash"
code={`#!/bin/sh
loopover-miner "$@" || { status=$?; notify-failure "loopover-miner $* exited $status"; exit "$status"; }`}
/>

Keep `--json` on scheduled runs so the alert handler can forward the structured output; the
human-readable form is for interactive use.
1 change: 1 addition & 0 deletions apps/loopover-ui/src/components/site/docs-nav.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,7 @@ export const docsNav: DocsGroup[] = [
{ to: "/docs/ams-deployment", label: "Deployment guide" },
{ to: "/docs/ams-operations-runbook", label: "Operations runbook" },
{ to: "/docs/ams-observability", label: "Observing your miner" },
{ to: "/docs/ams-unattended-scheduling", label: "Unattended scheduling" },
],
},
],
Expand Down
22 changes: 22 additions & 0 deletions apps/loopover-ui/src/routeTree.gen.ts
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,7 @@ import { Route as DocsGithubAppRouteImport } from './routes/docs.github-app'
import { Route as DocsFumadocsSpikeApiReferenceRouteImport } from './routes/docs.fumadocs-spike-api-reference'
import { Route as DocsBranchAnalysisRouteImport } from './routes/docs.branch-analysis'
import { Route as DocsBetaOnboardingRouteImport } from './routes/docs.beta-onboarding'
import { Route as DocsAmsUnattendedSchedulingRouteImport } from './routes/docs.ams-unattended-scheduling'
import { Route as DocsAmsOperationsRunbookRouteImport } from './routes/docs.ams-operations-runbook'
import { Route as DocsAmsObservabilityRouteImport } from './routes/docs.ams-observability'
import { Route as DocsAmsDeploymentRouteImport } from './routes/docs.ams-deployment'
Expand Down Expand Up @@ -331,6 +332,12 @@ const DocsBetaOnboardingRoute = DocsBetaOnboardingRouteImport.update({
path: '/beta-onboarding',
getParentRoute: () => DocsRoute,
} as any)
const DocsAmsUnattendedSchedulingRoute =
DocsAmsUnattendedSchedulingRouteImport.update({
id: '/ams-unattended-scheduling',
path: '/ams-unattended-scheduling',
getParentRoute: () => DocsRoute,
} as any)
const DocsAmsOperationsRunbookRoute =
DocsAmsOperationsRunbookRouteImport.update({
id: '/ams-operations-runbook',
Expand Down Expand Up @@ -457,6 +464,7 @@ export interface FileRoutesByFullPath {
'/docs/ams-deployment': typeof DocsAmsDeploymentRoute
'/docs/ams-observability': typeof DocsAmsObservabilityRoute
'/docs/ams-operations-runbook': typeof DocsAmsOperationsRunbookRoute
'/docs/ams-unattended-scheduling': typeof DocsAmsUnattendedSchedulingRoute
'/docs/beta-onboarding': typeof DocsBetaOnboardingRoute
'/docs/branch-analysis': typeof DocsBranchAnalysisRoute
'/docs/fumadocs-spike-api-reference': typeof DocsFumadocsSpikeApiReferenceRoute
Expand Down Expand Up @@ -523,6 +531,7 @@ export interface FileRoutesByTo {
'/docs/ams-deployment': typeof DocsAmsDeploymentRoute
'/docs/ams-observability': typeof DocsAmsObservabilityRoute
'/docs/ams-operations-runbook': typeof DocsAmsOperationsRunbookRoute
'/docs/ams-unattended-scheduling': typeof DocsAmsUnattendedSchedulingRoute
'/docs/beta-onboarding': typeof DocsBetaOnboardingRoute
'/docs/branch-analysis': typeof DocsBranchAnalysisRoute
'/docs/fumadocs-spike-api-reference': typeof DocsFumadocsSpikeApiReferenceRoute
Expand Down Expand Up @@ -593,6 +602,7 @@ export interface FileRoutesById {
'/docs/ams-deployment': typeof DocsAmsDeploymentRoute
'/docs/ams-observability': typeof DocsAmsObservabilityRoute
'/docs/ams-operations-runbook': typeof DocsAmsOperationsRunbookRoute
'/docs/ams-unattended-scheduling': typeof DocsAmsUnattendedSchedulingRoute
'/docs/beta-onboarding': typeof DocsBetaOnboardingRoute
'/docs/branch-analysis': typeof DocsBranchAnalysisRoute
'/docs/fumadocs-spike-api-reference': typeof DocsFumadocsSpikeApiReferenceRoute
Expand Down Expand Up @@ -664,6 +674,7 @@ export interface FileRouteTypes {
| '/docs/ams-deployment'
| '/docs/ams-observability'
| '/docs/ams-operations-runbook'
| '/docs/ams-unattended-scheduling'
| '/docs/beta-onboarding'
| '/docs/branch-analysis'
| '/docs/fumadocs-spike-api-reference'
Expand Down Expand Up @@ -730,6 +741,7 @@ export interface FileRouteTypes {
| '/docs/ams-deployment'
| '/docs/ams-observability'
| '/docs/ams-operations-runbook'
| '/docs/ams-unattended-scheduling'
| '/docs/beta-onboarding'
| '/docs/branch-analysis'
| '/docs/fumadocs-spike-api-reference'
Expand Down Expand Up @@ -799,6 +811,7 @@ export interface FileRouteTypes {
| '/docs/ams-deployment'
| '/docs/ams-observability'
| '/docs/ams-operations-runbook'
| '/docs/ams-unattended-scheduling'
| '/docs/beta-onboarding'
| '/docs/branch-analysis'
| '/docs/fumadocs-spike-api-reference'
Expand Down Expand Up @@ -1192,6 +1205,13 @@ declare module '@tanstack/react-router' {
preLoaderRoute: typeof DocsBetaOnboardingRouteImport
parentRoute: typeof DocsRoute
}
'/docs/ams-unattended-scheduling': {
id: '/docs/ams-unattended-scheduling'
path: '/ams-unattended-scheduling'
fullPath: '/docs/ams-unattended-scheduling'
preLoaderRoute: typeof DocsAmsUnattendedSchedulingRouteImport
parentRoute: typeof DocsRoute
}
'/docs/ams-operations-runbook': {
id: '/docs/ams-operations-runbook'
path: '/ams-operations-runbook'
Expand Down Expand Up @@ -1381,6 +1401,7 @@ interface DocsRouteChildren {
DocsAmsDeploymentRoute: typeof DocsAmsDeploymentRoute
DocsAmsObservabilityRoute: typeof DocsAmsObservabilityRoute
DocsAmsOperationsRunbookRoute: typeof DocsAmsOperationsRunbookRoute
DocsAmsUnattendedSchedulingRoute: typeof DocsAmsUnattendedSchedulingRoute
DocsBetaOnboardingRoute: typeof DocsBetaOnboardingRoute
DocsBranchAnalysisRoute: typeof DocsBranchAnalysisRoute
DocsFumadocsSpikeApiReferenceRoute: typeof DocsFumadocsSpikeApiReferenceRoute
Expand Down Expand Up @@ -1424,6 +1445,7 @@ const DocsRouteChildren: DocsRouteChildren = {
DocsAmsDeploymentRoute: DocsAmsDeploymentRoute,
DocsAmsObservabilityRoute: DocsAmsObservabilityRoute,
DocsAmsOperationsRunbookRoute: DocsAmsOperationsRunbookRoute,
DocsAmsUnattendedSchedulingRoute: DocsAmsUnattendedSchedulingRoute,
DocsBetaOnboardingRoute: DocsBetaOnboardingRoute,
DocsBranchAnalysisRoute: DocsBranchAnalysisRoute,
DocsFumadocsSpikeApiReferenceRoute: DocsFumadocsSpikeApiReferenceRoute,
Expand Down
49 changes: 49 additions & 0 deletions apps/loopover-ui/src/routes/docs.ams-unattended-scheduling.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
import { createFileRoute, notFound } from "@tanstack/react-router";
import { Suspense } from "react";

import { DocsPage } from "@/components/site/docs-page";
import { docsClientLoader } from "@/lib/docs-client-loader";

// Rendered from content/docs/ams-unattended-scheduling.mdx via fumadocs-mdx's browser entry
// (docsClientLoader), through the existing DocsPage/Callout/CodeBlock/FeatureRow
// primitives -- not fumadocs-ui's bundled components. See docs-source.ts's comment
// for why the loader below resolves only a plain, serializable path string.
export const Route = createFileRoute("/docs/ams-unattended-scheduling")({
loader: async () => {
const { docsSource } = await import("@/lib/docs-source");
const page = docsSource.getPage(["ams-unattended-scheduling"]);
if (!page) throw notFound();
return { path: page.path, title: page.data.title, description: page.data.description };
},
head: () => ({
meta: [
{ title: "Unattended scheduling & failure alerting — LoopOver docs" },
{
name: "description",
content:
"Run the miner's scheduled commands -- manage poll and discover -- unattended on cron or systemd, and alert reliably when a run fails.",
},
{ property: "og:title", content: "Unattended scheduling & failure alerting — LoopOver docs" },
{
property: "og:description",
content:
"Run the miner's scheduled commands -- manage poll and discover -- unattended on cron or systemd, and alert reliably when a run fails.",
},
{ property: "og:url", content: "/docs/ams-unattended-scheduling" },
],
links: [{ rel: "canonical", href: "/docs/ams-unattended-scheduling" }],
}),
component: AmsUnattendedScheduling,
});

function AmsUnattendedScheduling() {
const { path, title, description } = Route.useLoaderData();
const Content = docsClientLoader.getComponent(path);
return (
<DocsPage eyebrow="Maintainers" title={title} description={description}>
<Suspense fallback={<p className="text-token-sm text-muted-foreground">Loading…</p>}>
<Content />
</Suspense>
</DocsPage>
);
}
1 change: 1 addition & 0 deletions apps/loopover-ui/src/routes/docs.index.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,7 @@ const AUDIENCES: Audience[] = [
{ to: "/docs/ams-deployment", label: "AMS deployment guide" },
{ to: "/docs/ams-operations-runbook", label: "AMS operations runbook" },
{ to: "/docs/ams-observability", label: "Observing your miner" },
{ to: "/docs/ams-unattended-scheduling", label: "Unattended scheduling" },
{ to: "/docs/self-hosting-docs-audit", label: "Self-host docs audit" },
{ to: "/docs/maintainer-install-trust", label: "Install & trust guide" },
{ to: "/docs/github-app", label: "GitHub App configuration" },
Expand Down
5 changes: 5 additions & 0 deletions packages/loopover-miner/docs/unattended-scheduling.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,10 @@
# loopover-miner — unattended scheduling & failure alerting

> Also published on the docs website: [Unattended scheduling & failure
> alerting](https://loopover.ai/docs/ams-unattended-scheduling) (same content, rendered with
> search and the rest of the maintainer docs nav). This file remains the canonical source and
> ships inside the published `@loopover/miner` package.

Operational guidance for running the miner's scheduled commands — `manage poll` and `discover` —
unattended on a timer (cron or systemd), and for alerting when a run fails. These are the two commands
most likely to run on a schedule; everything they need is local and they make no interactive prompts.
Expand Down
Loading