fix: Validate the embeddings size to catch silent embeddings batch failures

[aperepel]

With a larger batch size for add_documents (e.g. 1000), the embeddings service may silently fail and return nothing for some entries. This lead to the more cryptic error:

[values_dict[key][i] for key in values_dict]
     ~~~~~~~~~~~~~~~~^^^
IndexError: list index out of range

Add additional size validation and a suggestion on how to remedy.

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

• Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
• Ensure the tests and linter pass
• Code coverage does not decrease (if any source code was changed)
• Appropriate docs were updated (if necessary)

Fixes #<issue_number_goes_here> 🦕

[averikitsch]

/gcbrun

[gauravpurohit06]

@aperepel, thank you raising the pull request. Question: In your PR you are adding a validation check, which is required for sure. On a separate note: Have you tried diving it into batches and then generating the embeddings if it's failing for large batches?

[gauravpurohit06]

Changes look good to me.

[aperepel]

@aperepel, thank you raising the pull request. Question: In your PR you are adding a validation check, which is required for sure. On a separate note: Have you tried diving it into batches and then generating the embeddings if it's failing for large batches?

Yes, that's what we've been doing. On a larger batch (usually around 1000 items), the embeddings service returned ~20-25% loss. The 100 size batch works perfectly stable, however.

We do use retry logic everywhere we run micro-batches, but if you were asking about pro-actively splitting it after a failure - that's probably too many LLM calls, as we must retry a whole split.

[averikitsch]

Tests are failing due to code coverage. Can you add a tests for your changes?

[kurtisvg]

@gauravpurohit06 @vishwarajanand FYI