Knowledge documents are indexed by two components: the Pinter backend (management, metadata, RBAC, search) and the parser (a batch Cloud Run Job that does the heavy lifting). They replace the retired Knowledge Hub service. The parser lives in the knowledge-hub-parser repository; the backend side lives in pinter-be-refactor behind the KNOWLEDGE_BACKEND=hub|parser flag (default hub during migration).

Component responsibilities

Upload lifecycle

  • Job input: PARSER_REQUEST JSON (asset_id, asset_version_id, asset_name, version, source_uri, filename, result_uri). No secrets in the request.
  • Job output: result.json — status, error {code,message}, chunk_count, page_count, warnings, layout_uri. Exit codes: 0 ok, 1 classified failure, 2 invalid request. Retries are safe (replace_version is idempotent).
  • Backend status sync: lazy on read (throttled) plus a scheduled sweeper (POST /api/internal/jobs/knowledge-sweep) that resubmits stale processing versions, starts queued pending ones (bounded by KNOWLEDGE_PARSER_MAX_PARALLEL), and purges orphan vectors. A crash without result.json is detected from the Cloud Run execution state.
  • Error codes surfaced to the UI: encrypted_pdf, unreadable_pdf, too_many_pages, no_text, conversion_failed, office_conversion_failed, office_conversion_timeout, office_unavailable, embedding_failed, index_failed, source_unavailable, internal_error. Failed documents can be reprocessed via POST /hub-documents/{id}/reprocess.

Search path (all inside BE)

  1. Resolve the caller’s accessible, processed asset_version_ids from SQL (owner, grants, public, expiry, view-only).
  2. Embed the query with the same model (RETRIEVAL_QUERY).
  3. Qdrant search filtered by MatchAny(asset_version_id) (capped at 1000 versions per query, with a truncated flag).
  4. Results carry document{...}, similarity, and metadata{locator, page_start, page_end, kind, source_ids}.
Agent runtime resolves the scope once per agent build and caches it (60 s); share/delete changes invalidate the cache, so revoked access stops applying without redeploying.

Layout artifact and viewer

On success the job also writes layout.json: per-block {source_id, kind, page_number, bbox (0..1, top-left origin), text}. The backend serves it via GET /hub-documents/{id}/layout (cached 60 s). The frontend renders the PDF with pdfjs-dist and draws an SVG overlay per page; clicks map blocks to chunks through meta.source_ids.

Data migration

Legacy hub documents are migrated with scripts/migrate_hub_documents.py (backend repo): dry-run by default, idempotent, resumable, with rollback. Chunks are re-parsed from original files (manual chunk edits from the old hub are not carried over). See scripts/migrate_hub_documents.md and docs/knowledge-native-cutover.md in the backend repo.