LCORE-1496: Inference providers e2e tests failing (Watson, Vertex, Azure, RHAIIS, RHELAI) by are-ces · Pull Request #1358 · lightspeed-core/lightspeed-stack

are-ces · 2026-03-19T11:08:44Z

Description

Several issues are addressed in this PR:

configuration of providers had breaking changes upstream (WatsonX, Azure)
constants module was not visible in the enrichment script: added src to the PYTHONPATH
Azure URL changed

Type of change

Tools used to create PR

Identify any AI code assistants used in this PR (for transparency and review context)

Assisted-by: Claude

Related Tickets & Documents

Related Issue # LCORE-1496
Closes # LCORE-1496

Checklist before requesting a review

I have performed a self-review of my code.
PR has passed all pre-merge test jobs.
If it is a core feature, I have added thorough tests.

Testing

e2e tests passing for the providers.

Summary by CodeRabbit

New Features
- Added Model Context Protocol (MCP) support for enhanced tool capabilities.
- Integrated new embedding model support (all-mpnet-base-v2) with improved vector storage.
Chores
- Updated provider configurations for Azure, Vertex AI, and Watsonx services.
- Modified CI/CD workflows to run end-to-end tests on every code push.
- Updated vector storage backend configuration and paths.

coderabbitai · 2026-03-19T11:09:05Z

Walkthrough

This PR updates GitHub Actions workflow triggers to run on push events instead of scheduled cron jobs, adds Model Context Protocol (MCP) provider support across multiple configuration files, updates provider endpoint configurations (renaming api_base/url to base_url), and refactors vector storage backend selections with new FAISS vector store definitions and embedding model updates.

Changes

Cohort / File(s)	Summary
GitHub Actions Workflow Triggers `.github/workflows/e2e_tests_*.yaml`	Changed workflow execution from scheduled cron and manual `workflow_dispatch` triggers to event-driven `push` triggers; all job logic remains unchanged.
Example Configuration Files `examples/azure-run.yaml`, `examples/vertexai-run.yaml`, `examples/watsonx-run.yaml`	Added `model-context-protocol` remote provider for MCP support; updated provider endpoints (`api_base`/`url` → `base_url`); introduced `kv_rag` storage backend for RAG vector I/O; added `all-mpnet-base-v2` embedding model and FAISS vector store resources; updated default embedding model and removed telemetry section (vertexai only).
Test Configuration Files `tests/e2e/configs/run-*.yaml`	Added `model-context-protocol` remote provider for MCP support; updated Azure provider endpoint (`api_base` → `base_url`); updated Watsonx provider configuration key (`url` → `base_url`); added inline documentation comments to RAG runtime entries.
Container Configuration `test.containerfile`	Added `PYTHONPATH=/opt/app-root/src` environment variable alongside existing `PATH` configuration for Python module resolution.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

🚥 Pre-merge checks | ✅ 3

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title directly references the ticket (LCORE-1496) and explicitly lists all five providers mentioned in the file changes, accurately summarizing the main fix objective.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

📝 Coding Plan

Generate coding plan for human review comments

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

Tip

You can customize the tone of the review comments and chat replies.

Configure the tone_instructions setting to customize the tone of the review comments and chat replies. For example, you can set the tone to Act like a strict teacher, Act like a pirate and more.

coderabbitai

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

tests/e2e/configs/run-watsonx.yaml (1)

157-162: ⚠️ Potential issue | 🔴 Critical

Duplicate vector_stores key will break vector store registration.

Same issue as in examples/watsonx-run.yaml - the vector_stores: [] at line 162 overwrites the FAISS configuration above it.

🐛 Proposed fix - remove the duplicate key

   vector_stores: 
   - embedding_dimension: 768
     embedding_model: sentence-transformers/all-mpnet-base-v2
     provider_id: faiss
     vector_store_id: ${env.FAISS_VECTOR_STORE_ID}
-  vector_stores: []
   datasets: []

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@tests/e2e/configs/run-watsonx.yaml` around lines 157 - 162, The YAML contains
a duplicate top-level key "vector_stores" which causes the configured FAISS
entry (the block with embedding_dimension, embedding_model, provider_id, and
vector_store_id) to be overwritten by the later empty `vector_stores: []`;
remove the trailing duplicate `vector_stores: []` so the FAISS configuration
block remains the sole "vector_stores" entry (same fix as in
examples/watsonx-run.yaml).

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@examples/watsonx-run.yaml`:
- Around line 157-162: The YAML contains a duplicate top-level key
"vector_stores" which causes the configured FAISS block (embedding_dimension,
embedding_model, provider_id, vector_store_id with ${env.FAISS_VECTOR_STORE_ID})
to be overwritten by the later "vector_stores: []"; remove the redundant
"vector_stores: []" entry so the FAISS vector store configuration remains (keep
the block containing embedding_dimension, embedding_model, provider_id,
vector_store_id).

---

Outside diff comments:
In `@tests/e2e/configs/run-watsonx.yaml`:
- Around line 157-162: The YAML contains a duplicate top-level key
"vector_stores" which causes the configured FAISS entry (the block with
embedding_dimension, embedding_model, provider_id, and vector_store_id) to be
overwritten by the later empty `vector_stores: []`; remove the trailing
duplicate `vector_stores: []` so the FAISS configuration block remains the sole
"vector_stores" entry (same fix as in examples/watsonx-run.yaml).

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 761aed89-4a09-46d4-b1c2-c76b08c57a02

📥 Commits

Reviewing files that changed from the base of the PR and between f814439 and e29492c.

📒 Files selected for processing (10)

.github/workflows/e2e_tests_providers.yaml
.github/workflows/e2e_tests_rhaiis.yaml
.github/workflows/e2e_tests_rhelai.yaml
examples/azure-run.yaml
examples/vertexai-run.yaml
examples/watsonx-run.yaml
test.containerfile
tests/e2e/configs/run-azure.yaml
tests/e2e/configs/run-vertexai.yaml
tests/e2e/configs/run-watsonx.yaml

coderabbitai · 2026-03-19T11:13:27Z

examples/watsonx-run.yaml

+  vector_stores: 
+  - embedding_dimension: 768
+    embedding_model: sentence-transformers/all-mpnet-base-v2
+    provider_id: faiss
+    vector_store_id: ${env.FAISS_VECTOR_STORE_ID}
  vector_stores: []


⚠️ Potential issue | 🔴 Critical

Duplicate vector_stores key will cause the FAISS configuration to be silently overwritten.

In YAML, duplicate keys result in the later value overwriting the earlier one. The vector_stores: [] at line 162 will override the FAISS vector store configuration defined at lines 157-161, effectively removing the vector store registration and breaking RAG functionality.

🐛 Proposed fix - remove the duplicate key

vector_stores: - embedding_dimension: 768 embedding_model: sentence-transformers/all-mpnet-base-v2 provider_id: faiss vector_store_id: ${env.FAISS_VECTOR_STORE_ID} - vector_stores: [] datasets: []

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

vector_stores:

- embedding_dimension: 768

embedding_model: sentence-transformers/all-mpnet-base-v2

provider_id: faiss

vector_store_id: ${env.FAISS_VECTOR_STORE_ID}

vector_stores: []

vector_stores:

- embedding_dimension: 768

embedding_model: sentence-transformers/all-mpnet-base-v2

provider_id: faiss

vector_store_id: ${env.FAISS_VECTOR_STORE_ID}

🧰 Tools

🪛 YAMLlint (1.38.0)

[error] 162-162: duplication of key "vector_stores" in mapping

(key-duplicates)

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed. In `@examples/watsonx-run.yaml` around lines 157 - 162, The YAML contains a duplicate top-level key "vector_stores" which causes the configured FAISS block (embedding_dimension, embedding_model, provider_id, vector_store_id with ${env.FAISS_VECTOR_STORE_ID}) to be overwritten by the later "vector_stores: []"; remove the redundant "vector_stores: []" entry so the FAISS vector store configuration remains (keep the block containing embedding_dimension, embedding_model, provider_id, vector_store_id).

are-ces added 6 commits March 19, 2026 09:56

Run tests on push

91b1099

Fix vertex ai

cd2429d

Fix constants module not found

1cb8cbd

Fix watsonx, azure config change

00d15fe

Fix watson and vertex

7fe5a4b

Fix Azure

e29492c

are-ces marked this pull request as draft March 19, 2026 11:10

coderabbitai bot reviewed Mar 19, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

LCORE-1496: Inference providers e2e tests failing (Watson, Vertex, Azure, RHAIIS, RHELAI)#1358

LCORE-1496: Inference providers e2e tests failing (Watson, Vertex, Azure, RHAIIS, RHELAI)#1358
are-ces wants to merge 6 commits intolightspeed-core:mainfrom
are-ces:fix-providers-tests

are-ces commented Mar 19, 2026 •

edited by coderabbitai bot

Loading

Uh oh!

coderabbitai bot commented Mar 19, 2026 •

edited

Loading

Uh oh!

coderabbitai bot left a comment

Uh oh!

coderabbitai bot Mar 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

are-ces commented Mar 19, 2026 • edited by coderabbitai bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

Tools used to create PR

Related Tickets & Documents

Checklist before requesting a review

Testing

Summary by CodeRabbit

Uh oh!

coderabbitai bot commented Mar 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Estimated code review effort

Uh oh!

coderabbitai bot left a comment

Choose a reason for hiding this comment

Uh oh!

coderabbitai bot Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

are-ces commented Mar 19, 2026 •

edited by coderabbitai bot

Loading

coderabbitai bot commented Mar 19, 2026 •

edited

Loading