KASTERVO Cloud BlogBlog

 

Overview

 

We opted to use BGP instead of Layer 2 announcements for scalability reasons in our Kubernetes setup with Cilium and pfSense. Layer 2 announcements work by advertising an IP address using the same network interface and MAC address of the node. This creates an ARP entry, allowing access to the service. While effective for smaller deployments, this approach is limited by the available IP addresses within the cluster’s subnet.

As the deployment size grows, relying on Layer 2 becomes less efficient. BGP removes these limitations by enabling dynamic routing and allowing IPs to be advertised across multiple subnets. By integrating pfSense as the BGP router, we can dynamically propagate routes across the network, ensuring seamless communication between distributed systems.

Using BGP with pfSense allows us to scale without the subnet limitations of Layer 2. This means new Kubernetes services can be deployed without worrying about running out of IP addresses. Cilium’s advanced networking and LoadBalancer integration with BGP ensures that newly assigned LoadBalancer IPs are routed efficiently, making services instantly accessible.

This approach significantly enhances our deployment’s scalability and flexibility, enabling expansion without the overhead of managing static routes or IP conflicts.

 

Step #1: Environment

 

Assumptions & Prerequisites.

To follow this guide, ensure the following prerequisites are met.

• Configured and functional pfSense: pfSense should be fully operational to facilitate BGP peering with Cilium, ensuring proper routing between your Kubernetes cluster and external networks.
• Dedicated VLAN for BGP Peering: For security and isolation, it’s recommended to create a dedicated VLAN for BGP communication between your Kubernetes cluster and pfSense.
• Kubernetes Cluster with Cilium Installed: Ensure your Kubernetes cluster is set up with Cilium in a “green state” (i.e., without errors) and is fully functional within the designated BGP VLAN.
• Private ASN Allocation for BGP: You will need two Autonomous System Numbers (ASNs) for BGP peering—one for pfSense and one for Cilium.
• Understanding of the BGP Protocol and Kubernetes Architecture: Basic knowledge of BGP and Kubernetes will be helpful in understanding how the routing and peering are set up.

 

Network Information:

Below are the network configurations used in this guide. Feel free to adjust them to fit your environment.

• pfSense ASN: 64520
• Cilium ASN: 64521
• LAN: 10.145.10.0/24 (Local network where your cluster resides)
• BGP/Cluster Network: 10.146.10.0/24 (Network used for BGP peering between pfSense and Kubernetes)
• LoadBalancer Network: 10.147.10.0/24 (Network for LoadBalancer IPs to be routed)

 

Diagram:

 

 

 

 

Step #2: pfSense Setup

In this step, we configure pfSense as a BGP router using FRR (Free Range Routing). FRR allows dynamic route exchanges between pfSense and Kubernetes via Cilium. We will install and enable FRR, set up BGP peering, and configure route maps to ensure proper routing.

 

1. Install FRR on pfSense

To enable BGP functionality, install FRR (Free Range Routing) on pfSense:

• System → Package Manager → Available Packages

 

2. Enable FRR

Once installed, enable FRR Global/Zebra:

• Services → FRR Global/Zebra → Global Settings

  • Enable: Checked

  • Master Password: <insert-a-master-password>

  • Syslog Logging: Checked

 

3. Create a Route Map

Route maps control which routes are accepted or advertised by BGP. We will create a simple ALLOW-ALL route map to permit all routes between pfSense and Cilium.

• Services → FRR Global/Zebra → Route Maps → Add

  • Name: ALLOW-ALL

  • Description: Match any route

  • Action: Permit

  • Sequence: 10

 

4. Enable BGP Routing

Now, enable BGP (Border Gateway Protocol) and set the Local ASN (Autonomous System Number):

• Services → FRR BGP → BGP

  • Enable: Checked

  • Log Adjacency Changes: Checked

  • Local AS: 64520

  • Router ID: 10.146.10.254

 

5. Create BGP Neighbors

Next, configure BGP neighbors for the Kubernetes control plane and worker nodes.

➤ Add the Control Plane Node

• Services → FRR BGP → Neighbors → Add

  • General Options

    • Name/Address: 10.146.10.20

    • Description: control-plane

  • Basic Options:

    • Remote AS: 64521

  • Peer Filtering → Route Map Filters:

    • Inbound Route Map Filter: ALLOW-ALL - Match any route

    • Outbound Route Map Filter: ALLOW-ALL - Match any route

➤ Add the Worker Node

• Services → FRR BGP → Neighbors → Add

  • General Options

    • Name/Address: 10.146.10.30

    • Description: worker-node

  • Basic Options:

    • Remote AS: 64521

  • Peer Filtering → Route Map Filters:

    • Inbound Route Map Filter: ALLOW-ALL - Match any route

    • Outbound Route Map Filter: ALLOW-ALL - Match any route

 

Step #3: Cilium Setup

 

1. Enable the BGP Control Plane

Run the following command to enable Cilium’s BGP control plane:

 

2. Create the Required Manifests

These manifests configure BGP peering between Cilium and pfSense.

 

3. Deploy the BGP Configuration

Apply all the manifests to the cluster:

 

4. Restart Cilium Services

After deploying the configurations, restart Cilium to ensure changes take effect:

 

Step #4: Validation & Testing

Once the configuration is complete, follow these steps to validate that BGP peering and route propagation are working correctly.

 

1. Verify pfSense BGP Neighbor Status

Check if pfSense has successfully established BGP sessions with the Cilium nodes.

Navigate to:

• Status → FRR → BGP Summary

Expected Result:
Both Cilium nodes (control-plane and worker-node) should be listed as established neighbors.

 

2. Verify Received Routes on pfSense

Ensure pfSense is correctly receiving routes from the Cilium nodes.

Navigate to:

• Status → FRR → BGP Routes

Expected Result:
You should see the routes received from the Cilium nodes, corresponding to the LoadBalancer IPs.

 

3. Check Cilium BGP Peers

Run the following command to verify BGP peer connections from Cilium.

Expected Output:

Key Indicators:

• Session State: established → Peering is successful.
• Received & Advertised Routes: Ensure there are valid entries.

 

4. Check Cilium Advertised Routes

Run the following command to verify which routes Cilium is distributing to pfSense.

Expected Output:

This confirms that the LoadBalancer IPs are being properly advertised by Cilium and received by pfSense.

 

Conclusion

By integrating Cilium’s BGP control plane with pfSense, we’ve established a scalable and efficient way to announce Kubernetes LoadBalancer IPs dynamically. Unlike traditional Layer 2 advertisements, BGP provides greater flexibility, eliminates subnet limitations, and ensures seamless route propagation across the network.

With this setup, new Kubernetes services can be deployed without manual IP management, and pfSense dynamically receives and routes these addresses, ensuring high availability and smooth network operations. This approach is ideal for growing infrastructures where scalability, automation, and reliability are key priorities.

 

About Us

KASTERVO is a cloud service provider specializing in security, high availability, and cost-efficient infrastructure. Founded by a team of passionate I.T. engineers, we help businesses reduce capital and operational expenses through our premium Private Cloud solutions.

As a strategic partner of Veeam and Microsoft, we enable seamless integration of Private Cloud with Microsoft 365 and Veeam backup solutions, ensuring robust data protection and business continuity.

Beyond cloud hosting, our team has developed industry-leading standards and procedures for Managed Cloud services, including Systems Hardening, proactive monitoring, maintenance, and seamless cloud integration tailored to each client’s needs.

Accelerate your cloud journey with KASTERVO—contact us today at: https://kastervo.com/contact