Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .claude/skills/update-source-references/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ For all mdx/md files in @docs/canary-checker/ and @docs/mission-control/ that ha
- Document ALL fields from that struct, not just the ones currently in docs
- Follow type references across packages to get complete field lists

4. For _canary-spec.mdx, ensure all check types from CanarySpec are listed with correct field names matching the json tags
4. For \_canary-spec.mdx, ensure all check types from CanarySpec are listed with correct field names matching the json tags

Pay attention to:

Expand Down
2 changes: 1 addition & 1 deletion canary-checker/docs/concepts/distributed-canaries.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ sidebar_position: 6
Distributed canaries allow you to define a check once and have it automatically run on multiple agents. This is useful for monitoring services from different locations, clusters, or network segments.

:::info
This feature is only available in [Mission Control](https://flanksource.com/docs) since Canary Checker does not support agents
This feature is only available in [Mission Control](https://flanksource.com/docs) since Canary Checker does not support agents
:::

## How It Works
Expand Down
Binary file not shown.
75 changes: 49 additions & 26 deletions mission-control/docs/guide/notifications/concepts/silences.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -4,52 +4,75 @@ sidebar_custom_props:
icon: mage:notification-bell-muted
---

import Silence from "@site/docs/reference/notifications/_silence.mdx"

A silence is a way to temporarily suppress notifications. Each silence has:

- A set duration - like an expiration timer
- A scope - it can be applied to either:
- Individual resources (a specific catalog, health check, or component)
- Multiple matching resources using filters
- A description - explaining why the silence was created
- A duration - specified by `from` and `until` timestamps in RFC3339 format, datetime or datemath expressions (e.g., `now`, `now+2d`)
- A scope - defined through selectors and filters to target specific resources
- A recursive flag - to apply silence to child resources

:::note
Notifications that aren't sent due to silence are still visible in the notification history for auditing purposes.
:::


```yaml title="" file=<rootDir>/modules/mission-control/fixtures/silences/silence-test-env.yaml
```

## Use cases

- Planned maintenance or deployments. Eg: You can silence a namespace or a helm release and automatically silence notifications from all their children.
- Non-critical resources: Notifications from resources that routinely trigger alerts but are expected and harmless can be silenced.
- Known issues: If there's a known issue that can't be immediately resolved (e.g., due to dependencies or resource constraints), you might silence related alerts until a fix can be implemented.
- Planned maintenance or deployments - Silence notifications from a namespace or helm release and optionally all their children
- Non-critical resources - Suppress notifications from resources that routinely trigger expected and harmless alerts
- Known issues - Temporarily silence alerts for known issues that can't be immediately resolved due to dependencies or resource constraints

## Creating Silences

Silences can be created in multiple ways:
1. Through the notification page UI
2. Using the silence button on Slack notifications (when using default templates)
3. By applying a NotificationSilence custom resource

<Silence />

## Add Silence
### Resource Selection

Silences can be added from the notification page. Alternatively, if you're using the default slack notification templates, you get a silence button
on each notification.
Silences can target resources using selectors, filters, or a specific resource. At least one targeting method must be specified.

![Silence Notification form](./silence-notification-form.png)
1. **Selectors**: Direct resource matching using types, names, namespaces, labels, and tags
2. **Filters**: Complex matching using CEL expressions
3. **Specific Resource**: Selecting a specific config, check, canary, or component (via UI)

### Filters
:::info
When both `filter` and `selectors` are specified, the filter is evaluated first. If the filter matches, the silence is applied. Selectors are only evaluated if the filter does not match.
:::

#### Selectors

Selectors use [Resource Selectors](/docs/reference/resource-selector) to target specific resources. Multiple selectors can be specified to target different resources.

A silence can optionally contain a filter. A filter is a cel-expression that results in a boolean value.
If a filter return true, the notification is silenced.
#### Filters

#### Examples
A filter is a CEL expression that evaluates to a boolean value. The notification is silenced when the filter returns true. Filters provide powerful, flexible matching capabilities.

| Filter | description |
##### Filter Examples

| Filter | Description |
|--------|-------------|
| `check.type == 'http'` | silences only HTTP check related notifications. |
| `regexp.Match("^check-[0-9]+", check.name)` | matches any check with the prefix `check-` |
| `config.name == "postgresql" && config.type == "Kubernetes::StatefulSet"` | silences notifications from a stateful set named "postgresql" |
| `config.type == "Kubernetes::Pod" && catalog.traverse(config.id, "Kubernetes::Namespace", "incoming").size > 0 && catalog.traverse(config.id, "Kubernetes::Namespace", "incoming")[0].tags.?env.orValue("") == "prod"` | matches kubernetes pod in a prod namespace |
| `check.type == 'http'` | Silences HTTP check notifications |
| `regexp.Match("^check-[0-9]+", check.name)` | Matches checks with prefix `check-` |
| `config.name == "postgresql" && config.type == "Kubernetes::StatefulSet"` | Silences notifications from a specific StatefulSet |
| `config.type == "Kubernetes::Pod" && catalog.traverse(config.id, "Kubernetes::Namespace", "incoming").size > 0 && catalog.traverse(config.id, "Kubernetes::Namespace", "incoming")[0].tags.?env.orValue("") == "prod"` | Matches pods in production namespaces |

#### Template variables
##### Available Template Variables

Filters can reference these variables:
- [CheckEvents](/docs/reference/notifications/template_vars/checks)
- [ConfigEvents](/docs/reference/notifications/template_vars/config)

### Recursive mode

When a silence is recursively applied, it applies to its all children.
So silencing a namespace would silence all deployments, statefulsets, pods, etc in that namespace.

### Recursive Mode

When `recursive: true` is set, the silence applies to all child resources of the matched resources. For example:
- Silencing a namespace affects all deployments, statefulsets, pods, etc. within it
- Silencing a statefulset affects all its pods
49 changes: 42 additions & 7 deletions mission-control/docs/reference/notifications/_env_vars.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -19,12 +19,27 @@ export function CheckHealthEnv() {
{
"field": "permalink",
"scheme": "string",
"description": "Link to the Catalog in mission control"
"description": "A link to the resource in Mission Control"
},
{
"field": "silenceURL",
"scheme": "string",
"description": "A link to silence this notification"
},
{
"field": "agent",
"scheme": "[`Agent`](#agent)",
"description": "Details of the agent that created the config."
"description": "Details of the agent"
},
{
"field": "channel",
"scheme": "string",
"description": "The notification channel, e.g. `slack`, `email`"
},
{
"field": "groupedResources",
"scheme": "`[]string`",
"description": "A list of grouped resource names. Only available when notifications are grouped using `waitFor`."
}
]}
/>
Expand All @@ -35,23 +50,43 @@ export function ConfigEventsTemplateVars() {
rows={[
{
"field": "config",
"scheme": "[`Config`](#config)",
"scheme": "[`Config`](#config-item)",
"description": "Corresponding config item."
},
{
"field": "new_state",
"scheme": "`created`, `updated`, `deleted`",
"description": "The db state change that caused this event.\n\n*Only available on state events.*"
"scheme": "`created` | `updated` | `deleted`",
"description": "The state change that triggered this event.\n\n*Only available on config state events (e.g. `config.created`, `config.updated`, `config.deleted`).*"
},
{
"field": "permalink",
"scheme": "string",
"description": "Link to the Catalog in mission control"
"description": "A link to the resource in Mission Control"
},
{
"field": "silenceURL",
"scheme": "string",
"description": "A link to silence this notification"
},
{
"field": "agent",
"scheme": "[`Agent`](#agent)",
"description": "Details of the agent that created the config."
"description": "Details of the agent"
},
{
"field": "recent_events",
"scheme": "`[]string`",
"description": "Recent config changes from the last hour"
},
{
"field": "channel",
"scheme": "string",
"description": "The notification channel, e.g. `slack`, `email`"
},
{
"field": "groupedResources",
"scheme": "`[]string`",
"description": "A list of grouped resource names. Only available when notifications are grouped using `waitFor`."
}
]}
/>
Expand Down
40 changes: 40 additions & 0 deletions mission-control/docs/reference/notifications/_silence.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
<Fields withTemplates="true" rows={[
{
field: "description",
description: "A description explaining the purpose of the silence",
scheme: "string"
},
{
field: "from",
description: "Start time of the silence period in RFC3339 format, datetime or datemath expression (e.g., \"now\", \"now+2h\", \"2025-01-01\")",
scheme: "string"
},
{
field: "until",
description: "End time of the silence period in RFC3339 format, datetime or datemath expression (e.g., \"now\", \"now+2d\", \"2025-02-01\")",
scheme: "string"
},
{
field: "recursive",
description: "When true, the silence applies to all child resources of the matched resources. For example, silencing a namespace would silence all deployments, statefulsets, pods, etc. within it.",
scheme: "boolean"
},
{
field: "filter",
description: "A CEL expression that determines whether to apply the silence. The silence is only applied if the filter evaluates to true.",
scheme: "CEL",
templateEnv: [
{"name": "CheckEvents", "url": "/docs/reference/notifications/template_vars/checks"},
{"name": "ConfigEvents", "url": "/docs/reference/notifications/template_vars/config"},
],
},
{
field: "selectors",
description: "List of resource selectors to target specific resources",
scheme: "[[]ResourceSelector](/docs/reference/resource-selector)"
}
]}/>

:::info
At least one of `filter` or `selectors` must be specified to target resources for the silence. When both are specified, `filter` takes precedence.
:::