knowledge-hub-parser repository; the backend side lives in pinter-be-refactor behind the KNOWLEDGE_BACKEND=hub|parser flag (default hub during migration).
Component responsibilities
Upload lifecycle
- Job input:
PARSER_REQUESTJSON (asset_id,asset_version_id,asset_name,version,source_uri,filename,result_uri). No secrets in the request. - Job output:
result.json—status,error {code,message},chunk_count,page_count,warnings,layout_uri. Exit codes: 0 ok, 1 classified failure, 2 invalid request. Retries are safe (replace_versionis idempotent). - Backend status sync: lazy on read (throttled) plus a scheduled sweeper (
POST /api/internal/jobs/knowledge-sweep) that resubmits staleprocessingversions, starts queuedpendingones (bounded byKNOWLEDGE_PARSER_MAX_PARALLEL), and purges orphan vectors. A crash withoutresult.jsonis detected from the Cloud Run execution state. - Error codes surfaced to the UI:
encrypted_pdf,unreadable_pdf,too_many_pages,no_text,conversion_failed,office_conversion_failed,office_conversion_timeout,office_unavailable,embedding_failed,index_failed,source_unavailable,internal_error. Failed documents can be reprocessed viaPOST /hub-documents/{id}/reprocess.
Search path (all inside BE)
- Resolve the caller’s accessible,
processedasset_version_ids from SQL (owner, grants, public, expiry, view-only). - Embed the query with the same model (
RETRIEVAL_QUERY). - Qdrant search filtered by
MatchAny(asset_version_id)(capped at 1000 versions per query, with atruncatedflag). - Results carry
document{...},similarity, andmetadata{locator, page_start, page_end, kind, source_ids}.
Layout artifact and viewer
On success the job also writeslayout.json: per-block {source_id, kind, page_number, bbox (0..1, top-left origin), text}. The backend serves it via GET /hub-documents/{id}/layout (cached 60 s). The frontend renders the PDF with pdfjs-dist and draws an SVG overlay per page; clicks map blocks to chunks through meta.source_ids.
Data migration
Legacy hub documents are migrated withscripts/migrate_hub_documents.py (backend repo): dry-run by default, idempotent, resumable, with rollback. Chunks are re-parsed from original files (manual chunk edits from the old hub are not carried over). See scripts/migrate_hub_documents.md and docs/knowledge-native-cutover.md in the backend repo.