fix(encapsulation): route Cilium IPIP traffic through VxLAN overlay#6
fix(encapsulation): route Cilium IPIP traffic through VxLAN overlay#6
Conversation
Rewrite Cilium encapsulator to create IPIP tunnels instead of using cilium_host interface directly. Each node autodiscovers its cilium_host IP and advertises it via kilo.squat.ai/cilium-internal-ip annotation, allowing other nodes to route IPIP outer packets through Cilium's VxLAN overlay and preventing routing loops. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
📝 WalkthroughWalkthroughThe PR adds support for tracking and utilizing Cilium internal IPs throughout the mesh networking system. Changes include updating the Encapsulator interface signature to accept a ciliumInternalIP parameter, adding a LocalIP() method, refactoring Cilium encapsulation to use IPIP tunnels, propagating Cilium internal IP through mesh nodes and segments, extending Kubernetes backend to handle cilium internal IP annotations, and updating route gw calculations to include the new internal IP parameter. Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Poem
🚥 Pre-merge checks | ✅ 3 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Summary of ChangesHello @kvaps, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly refactors the Cilium encapsulation mechanism to resolve routing loop issues when IPIP tunnels are used in conjunction with Cilium's networking. By introducing a dedicated IPIP tunnel for Cilium traffic and leveraging Cilium's VxLAN overlay for routing, it ensures seamless cross-node pod communication. The changes also enable nodes to advertise their internal Cilium IP, allowing for more intelligent gateway selection and robust network topology management. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Changelog
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request refactors the Cilium encapsulator to use IPIP tunnels routed through Cilium's VxLAN overlay, which is a significant improvement for compatibility. It introduces a mechanism for nodes to discover and advertise their Cilium internal IP via Kubernetes annotations. The changes are well-distributed across the codebase and appear to correctly implement the intended logic. I have identified one critical issue that could lead to a panic and one medium-severity issue related to error handling that should be addressed.
| return a.Key.String() == b.Key.String() && | ||
| ipNetsEqual(a.WireGuardIP, b.WireGuardIP) && | ||
| ipNetsEqual(a.InternalIP, b.InternalIP) && | ||
| a.CiliumInternalIP.Equal(b.CiliumInternalIP) && |
There was a problem hiding this comment.
Calling .Equal() on a nil net.IP slice will cause a panic. The CiliumInternalIP field can be nil, for example, if a node does not have the Cilium internal IP annotation. To prevent a panic, you should use a nil-safe comparison method, such as bytes.Equal.
| a.CiliumInternalIP.Equal(b.CiliumInternalIP) && | |
| bytes.Equal(a.CiliumInternalIP, b.CiliumInternalIP) && |
| if err := iproute.DeleteAddresses(c.iface); err != nil { | ||
| return nil | ||
| } |
There was a problem hiding this comment.
The error handling in this function is inconsistent. If iproute.DeleteAddresses fails, the error is swallowed, and iproute.RemoveInterface is not called. However, if iproute.RemoveInterface fails, the error is returned. Swallowing the error from DeleteAddresses can hide underlying issues during cleanup. It's better to return the error to the caller for logging and diagnostics.
| if err := iproute.DeleteAddresses(c.iface); err != nil { | |
| return nil | |
| } | |
| if err := iproute.DeleteAddresses(c.iface); err != nil { | |
| return err | |
| } |
Align constant block formatting for gofmt, add ciliumInternalIPs to expected topology test segments, use bytes.Equal for nil-safe CiliumInternalIP comparison, and return error from CleanUp. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
staticcheck SA1021 requires net.IP.Equal for IP comparison. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
pkg/k8s/backend.go (1)
132-146:⚠️ Potential issue | 🟠 MajorMissing cleanup of
ciliumInternalIPAnnotationKeyannotation.The
Setmethod writesciliumInternalIPAnnotationKey(line 246), butCleanUpdoes not remove it. This leaves a stale annotation on the Kubernetes node object when the agent shuts down.Proposed fix
patch := []byte("[" + strings.Join([]string{ fmt.Sprintf(jsonRemovePatch, path.Join("/metadata", "annotations", strings.Replace(endpointAnnotationKey, "/", jsonPatchSlash, 1))), fmt.Sprintf(jsonRemovePatch, path.Join("/metadata", "annotations", strings.Replace(internalIPAnnotationKey, "/", jsonPatchSlash, 1))), fmt.Sprintf(jsonRemovePatch, path.Join("/metadata", "annotations", strings.Replace(keyAnnotationKey, "/", jsonPatchSlash, 1))), fmt.Sprintf(jsonRemovePatch, path.Join("/metadata", "annotations", strings.Replace(lastSeenAnnotationKey, "/", jsonPatchSlash, 1))), fmt.Sprintf(jsonRemovePatch, path.Join("/metadata", "annotations", strings.Replace(wireGuardIPAnnotationKey, "/", jsonPatchSlash, 1))), fmt.Sprintf(jsonRemovePatch, path.Join("/metadata", "annotations", strings.Replace(discoveredEndpointsKey, "/", jsonPatchSlash, 1))), fmt.Sprintf(jsonRemovePatch, path.Join("/metadata", "annotations", strings.Replace(granularityKey, "/", jsonPatchSlash, 1))), + fmt.Sprintf(jsonRemovePatch, path.Join("/metadata", "annotations", strings.Replace(ciliumInternalIPAnnotationKey, "/", jsonPatchSlash, 1))), }, ",") + "]")pkg/encapsulation/ipip.go (1)
36-41:⚠️ Potential issue | 🔴 CriticalBug:
CleanUpswallowsDeleteAddresseserror and skipsRemoveInterfaceon failure.When
DeleteAddressesfails, the method returnsnil(line 38) instead of the error. This both hides the failure and skips theRemoveInterfacecall. Conversely, whenDeleteAddressessucceeds, execution falls through toRemoveInterface— which is the correct happy path, but the error branch is wrong.🐛 Proposed fix
func (i *ipip) CleanUp() error { if err := iproute.DeleteAddresses(i.iface); err != nil { - return nil + return err } return iproute.RemoveInterface(i.iface) }
🤖 Fix all issues with AI agents
In `@pkg/mesh/routes.go`:
- Line 43: The three slices on segment (privateIPs, cidrs, ciliumInternalIPs)
can get out of sync because privateIPs is only appended when node.InternalIP !=
nil; fix the population so all three slices keep the same length (append a
nil/empty placeholder to privateIPs when InternalIP is absent) or change
iteration to use the minimum length of the three slices to avoid OOB; also guard
the use of segment.leader when indexing privateIPs in the enc.Gw(...) call
(involving updateEndpoint and segment.leader) by ensuring segment.leader <
len(privateIPs) or by selecting the corresponding placeholder element so the
enc.Gw invocation cannot index past privateIPs.
🧹 Nitpick comments (2)
pkg/encapsulation/cilium.go (1)
38-44: Early return inCleanUpskips interface removal if address deletion fails.If
iproute.DeleteAddressesfails, the IPIP interface itself is never removed. Consider cleaning up the interface regardless of address deletion errors.Proposed fix
func (c *cilium) CleanUp() error { - if err := iproute.DeleteAddresses(c.iface); err != nil { - return err - } - return iproute.RemoveInterface(c.iface) + var errs []error + if err := iproute.DeleteAddresses(c.iface); err != nil { + errs = append(errs, err) + } + if err := iproute.RemoveInterface(c.iface); err != nil { + errs = append(errs, err) + } + return errors.Join(errs...) }pkg/encapsulation/encapsulation.go (1)
47-58: Consider adding parameter names to theGwsignature for clarity.The
Gwmethod now takes four parameters of similar types (net.IP, net.IP, net.IP, *net.IPNet), making it hard to understand each parameter's purpose from the interface alone. Named parameters improve self-documentation.Suggested improvement
- Gw(net.IP, net.IP, net.IP, *net.IPNet) net.IP + Gw(src, private, ciliumIP net.IP, subnet *net.IPNet) net.IP
| for _, segment := range t.segments { | ||
| if segment.location == t.location { | ||
| gw = enc.Gw(t.updateEndpoint(segment.endpoint, segment.key, &segment.persistentKeepalive).IP(), segment.privateIPs[segment.leader], segment.cidrs[segment.leader]) | ||
| gw = enc.Gw(t.updateEndpoint(segment.endpoint, segment.key, &segment.persistentKeepalive).IP(), segment.privateIPs[segment.leader], segment.ciliumInternalIPs[segment.leader], segment.cidrs[segment.leader]) |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
rg -n "ciliumInternalIPs" --type go -C 3Repository: cozystack/kilo
Length of output: 15502
🏁 Script executed:
sed -n '157,185p' pkg/mesh/topology.goRepository: cozystack/kilo
Length of output: 1160
🏁 Script executed:
sed -n '40,75p' pkg/mesh/routes.goRepository: cozystack/kilo
Length of output: 1701
🏁 Script executed:
sed -n '150,175p' pkg/mesh/routes.goRepository: cozystack/kilo
Length of output: 1264
Fix slice length mismatch between privateIPs and cidrs/ciliumInternalIPs in segment population.
The privateIPs slice is populated conditionally (only when node.InternalIP != nil), but cidrs and ciliumInternalIPs are always populated. This causes privateIPs to have fewer elements than the other slices, leading to index out of bounds panics when routes.go loops over segment.cidrs and accesses segment.privateIPs[i] at indices where privateIPs was never appended (lines 61 and 156). Additionally, segment.leader indexing on line 43 could exceed privateIPs bounds. Ensure all three slices maintain the same length, or adjust the loop bounds to match the shortest slice.
🤖 Prompt for AI Agents
In `@pkg/mesh/routes.go` at line 43, The three slices on segment (privateIPs,
cidrs, ciliumInternalIPs) can get out of sync because privateIPs is only
appended when node.InternalIP != nil; fix the population so all three slices
keep the same length (append a nil/empty placeholder to privateIPs when
InternalIP is absent) or change iteration to use the minimum length of the three
slices to avoid OOB; also guard the use of segment.leader when indexing
privateIPs in the enc.Gw(...) call (involving updateEndpoint and segment.leader)
by ensuring segment.leader < len(privateIPs) or by selecting the corresponding
placeholder element so the enc.Gw invocation cannot index past privateIPs.
1 participant
Summary
Test plan
Summary by CodeRabbit
Release Notes
Refactor
Enhancement