glean-core-workflow-b

# Glean Core Workflow B: Indexing & Connectors ## Overview Build custom Glean connectors: set up datasources, bulk index documents, manage content lifecycle, and configure permissions. ## Instructions ### Step 1: Create Custom Datasource ```typescript await fetch(`${GLEAN}/index/v1/adddatasource`, { method: 'POST', headers: idxHeaders, body: JSON.stringify({ name: 'internal_docs', displayName: 'Internal Documentation', datasourceCategory: 'PUBLISHED_CONTENT', urlRegex: 'https://docs.internal.company.com/.*', isOnPrem: false, }), }); ``` ### Step 2: Bulk Index Documents ```typescript // Bulk indexing replaces ALL documents in the datasource const uploadId = `upload-${Date.now()}`; // Send documents in batches of 100 for (let i = 0; i < allDocs.length; i += 100) { const batch = allDocs.slice(i, i + 100); const isFirst = i === 0; const isLast = i + 100 >= allDocs.length; await fetch(`${GLEAN}/index/v1/bulkindexdocuments`, { method: 'POST', headers: idxHeaders, body: JSON.stringify({ datasource: 'internal_docs', uploadId, isFirstPage: isFirst, isLastPage: isLast, documents: batch.map(doc => ({ id: doc.id, title: doc.title, url: doc.url, body: { mimeType: 'text/html', textContent: doc.content }, author: { email: doc.authorEmail }, updatedAt: doc.updatedAt, permissions: { allowAnonymousAccess: true }, })), }), }); console.log(`Indexe...