Integrate SharePoint connector with Weaviate vector database

[NicolasLMP]

Why:
As a developer, I need to integrate the SharePoint connector with Weaviate to enable SharePoint documents to be indexed and searchable through the Isschat RAG system.

DoD:

• Create SharePoint document processor for ingestion pipeline
• Implement document chunking for SharePoint content
• Add SharePoint metadata handling and storage
• Integrate SharePoint connector with existing ingestion framework
• Test document embedding and indexing in Weaviate
• Add SharePoint source identification in search results
• Implement incremental updates for SharePoint content
• Test end-to-end integration with RAG pipeline

How:

• Extend existing ingestion pipeline to support SharePoint documents
• Create SharePoint-specific document processor with metadata handling
• Implement chunking strategy for SharePoint document types
• Add SharePoint source tagging for result identification
• Integrate with existing Weaviate indexing process
• Test document retrieval and semantic search functionality
• Add incremental update mechanism for changed documents
• Validate SharePoint content appears correctly in chat responses

[onestardao]

neat setup — but just a heads-up, once SharePoint ingestion is live, you might hit semantic drift or retrieval collapse if your chunking logic and source metadata don’t integrate well with Weaviate’s vector semantics.

we’ve seen this play out repeatedly in hybrid doc pipelines, especially when combining hierarchical enterprise formats (like SharePoint) with vector-first RAG systems.
usually shows up as hallucinated sources, duplicated results, or collapsed queries during incremental updates.

we've documented this as ProblemMap No.4 “Document Structure Misfit” + No.11 “Incremental Ingestion Drift”.

if you’re curious, we’ve open-sourced the full test suite + debug tools — happy to share.

[NicolasLMP]

neat setup — but just a heads-up, once SharePoint ingestion is live, you might hit semantic drift or retrieval collapse if your chunking logic and source metadata don’t integrate well with Weaviate’s vector semantics.

we’ve seen this play out repeatedly in hybrid doc pipelines, especially when combining hierarchical enterprise formats (like SharePoint) with vector-first RAG systems. usually shows up as hallucinated sources, duplicated results, or collapsed queries during incremental updates.

we've documented this as ProblemMap No.4 “Document Structure Misfit” + No.11 “Incremental Ingestion Drift”.

if you’re curious, we’ve open-sourced the full test suite + debug tools — happy to share.

Thank you for your comment !

[NicolasLMP]

Blocked because : To integrate a SharePoint connector into Isschat, Microsoft requires the creation of a dedicated Azure AD App Registration in order to allow access to SharePoint documents. To authorize this application to read the SharePoint site and its documents, administrator consent is required (which I cannot grant myself).

[onestardao]

Great to hear back — thanks for sharing the Azure AD registration roadblock too.

Since you’re clearly deep in this, I’ll go ahead and share what we’ve put together:

Full test suite for hybrid source ingestion (esp. SharePoint + vector DBs like Weaviate)

Symbolic fix modules for [No.4] Structure Misfit and [No.11] Incremental Drift

Bonus: end-to-end pipeline validator + hallucination suppressor (open source)

📎 GitHub (problem map + fixes): onestardao/WFGY

Let me know if you'd like the CLI test runner or the vectorstore metrics page — happy to send it.