Backfill

Create and monitor historical backfill jobs. See the Backfill guide for background.

const TOKEN = "hv_..."; // your API key
const headers = { Authorization: `Bearer ${TOKEN}` };

Create a backfill job

POST /admin/backfill

interface BackfillJob {
  id: string;
  status: string;
}

const response = await fetch("http://127.0.0.1:3000/admin/backfill", {
  method: "POST",
  headers: {
    ...headers,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    collection: "xyz.statusphere.status",
  }),
});
const data: BackfillJob = await response.json();

Response: 201 Created

{
  "id": "550e8400-e29b-41d4-a716-446655440000",
  "status": "running"
}

Cancel a backfill job

POST /admin/backfill/{id}/cancel

Requests cancellation of a running backfill job. The job status transitions to cancelling immediately; the background worker will stop at its next checkpoint and set the final status to cancelled. See the Backfill guide for details on the two-phase process.

const response = await fetch(
  `http://127.0.0.1:3000/admin/backfill/${jobId}/cancel`,
  { method: "POST", headers },
);
const data = await response.json();

Response: 200 OK

{
  "id": "550e8400-e29b-41d4-a716-446655440000",
  "status": "cancelling"
}

Returns 400 if the job is not currently running, or 404 if the job ID is not found.

Pause a backfill job

POST /admin/backfill/{id}/pause

Requests a running backfill job to pause. The job status transitions to pausing immediately; the background worker will stop at its next checkpoint and set the status to paused. Paused jobs retain all progress and can be resumed later.

curl -X POST "http://127.0.0.1:3000/admin/backfill/$JOB_ID/pause" -H "$AUTH"

Response: 200 OK

{
  "id": "550e8400-e29b-41d4-a716-446655440000",
  "status": "pausing"
}

Returns 400 if the job is not currently running, or 404 if the job ID is not found.

Resume a backfill job

POST /admin/backfill/{id}/resume

Resume a paused backfill job. The job status transitions back to running and processing continues from where it left off.

curl -X POST "http://127.0.0.1:3000/admin/backfill/$JOB_ID/resume" -H "$AUTH"

Response: 200 OK

{
  "id": "550e8400-e29b-41d4-a716-446655440000",
  "status": "running"
}

Returns 400 if the job is not currently paused, or 404 if the job ID is not found.

List backfill jobs

GET /admin/backfill/status

interface BackfillJob {
  id: string;
  collection: string | null;
  did: string | null;
  status: string;
  stage: string;
  total_repos: number | null;
  resolved_repos: number | null;
  processed_repos: number | null;
  total_records: number | null;
  error: string | null;
  started_at: string | null;
  completed_at: string | null;
  created_at: string;
}

const response = await fetch("http://127.0.0.1:3000/admin/backfill/status", {
  headers,
});
const data: BackfillJob[] = await response.json();

Response: 200 OK

[
  {
    "id": "550e8400-e29b-41d4-a716-446655440000",
    "collection": "xyz.statusphere.status",
    "did": null,
    "status": "completed",
    "stage": "completed",
    "total_repos": 42,
    "resolved_repos": 42,
    "processed_repos": 42,
    "total_records": 1000,
    "error": null,
    "started_at": "2025-01-01T00:01:00Z",
    "completed_at": "2025-01-01T00:05:00Z",
    "created_at": "2025-01-01T00:00:00Z"
  }
]

The status field tracks the overall job state (running, pausing, paused, cancelling, cancelled, completed, failed). The stage field tracks the current processing phase (pending, discovering_repos, resolving_and_fetching, completed, failed, cancelled). The resolved_repos counter tracks PDS resolution progress during the pipelined phase, while processed_repos tracks record fetching progress.

List repos for a job

GET /admin/backfill/{id}/repos

Paginated list of per-DID tracking rows for a backfill job. Requires BackfillRead.

curl "http://127.0.0.1:3000/admin/backfill/$JOB_ID/repos?phase=fetched&limit=10" -H "$AUTH"

Response: 200 OK

{
  "repos": [
    { "did": "did:plc:abc", "pds_endpoint": "https://pds.example.com", "status": "completed", "records_fetched": 42 }
  ],
  "cursor": "did:plc:def"
}

cursor is null when there are no more results.

PDS summary for a job

GET /admin/backfill/{id}/pds-summary

Aggregated PDS breakdown for a backfill job. Requires BackfillRead. No pagination — returns all PDS endpoints in one response, sorted by repo count descending.

curl "http://127.0.0.1:3000/admin/backfill/$JOB_ID/pds-summary" -H "$AUTH"

Response: 200 OK

{
  "pds_endpoints": [
    { "pds_endpoint": "https://morel.us-east.host.bsky.network", "total_repos": 1200, "completed_repos": 800, "total_records": 5000 }
  ]
}

Stream backfill events (SSE)

GET /admin/backfill/{id}/events

Server-Sent Events stream of real-time backfill progress. Requires BackfillRead. The connection stays open until the job completes or the client disconnects. A keepalive comment is sent periodically to prevent timeouts.

Events are sent with event: event and a JSON data payload. Each event has a type field:

Flush job details

DELETE /admin/backfill/{id}/details

Delete all per-repo tracking rows for a single backfill job. Requires BackfillCreate.

curl -X DELETE "http://127.0.0.1:3000/admin/backfill/$JOB_ID/details" -H "$AUTH"

Response: 204 No Content

Flush all job details

DELETE /admin/backfill/details

Delete per-repo tracking rows for all completed, cancelled, and failed backfill jobs. Requires BackfillCreate.

curl -X DELETE "http://127.0.0.1:3000/admin/backfill/details" -H "$AUTH"

Response: 204 No Content