Debugging Spinnaker Connectivity Issues with EKS Clusters Across VPCs: A Journey Through AWS Networking
In a recent debugging journey, we encountered a puzzling connectivity issue: our deployment tool, Spinnaker, which operates in a backend EKS cluster, suddenly lost access to resources in another cluster used for our public gateway. With recent EKS upgrades underway, we initially suspected a security group misconfiguration. However, this wasn’t a typical connectivity issue as no recent changes could clearly explain it.
As we investigated, we found ourselves navigating the complexities of AWS cross cluster routing and subnet configurations in EKS. Each clue pointed us deeper into the architecture, leading to unexpected discoveries about ENI placement and routing behavior within AWS. This blog details our journey through these layers of troubleshooting, the insights we gained, and the lessons learned along the way.