feat(logging): add fluent-bit log shipping #3431
Conversation
Implements dstackai#3430. This PR is partially implemented using Cursor.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…or for malformed tokens
|
|
||
| === "Ship-only mode" | ||
|
|
||
| Logs are forwarded to Fluent-bit but cannot be read through dstack. |
There was a problem hiding this comment.
We usually write dstack in backticks in the docs. Please follow it for consistency.
|
@DragonStuff, FluentBit+Elasticsearch seems like a good first option to offer for "bring your own" logging stack. Surely there can be many other log collector (e.g. fluentd, filebit, logstash) + log storage (e.g. loki) combinations that we should support in the future. I suppose we could separate log collector + log storage implementations and allow configuring them independently. What do you think? Currently fluentbit and elasticseach are coupled but they could be separated later I suppose. |
@DragonStuff, would you like the PR to be merged before testing it in the pilot environment or should we wait before you test it first and then merge? So far you tested it locally? |
|
I haven't tested the PR but overall it looks good. |
2 participants
Implements #3430.
This PR is partially implemented using Cursor.
This PR has NOT been run in our (TableCheck's) pilot environment yet (we are planning to run it in early January).