KASTERVO Cloud BlogBlog

Your source for technology related articles, KASTERVO press releases and more!

post-thumb

BY KASTERVO / ON February 8, 2025

Scaling Kubernetes with BGP: Integrating Cilium and pfSense for Dynamic Routing

 

Overview

 

We opted to use BGP instead of Layer 2 announcements for scalability reasons in our Kubernetes setup with Cilium and pfSense. Layer 2 announcements work by advertising an IP address using the same network interface and MAC address of the node. This creates an ARP entry, allowing access to the service. While effective for smaller deployments, this approach is limited by the available IP addresses within the cluster’s subnet.

As the deployment size grows, relying on Layer 2 becomes less efficient. BGP removes these limitations by enabling dynamic routing and allowing IPs to be advertised across multiple subnets. By integrating pfSense as the BGP router, we can dynamically propagate routes across the network, ensuring seamless communication between distributed systems.

Using BGP with pfSense allows us to scale without the subnet limitations of Layer 2. This means new Kubernetes services can be deployed without worrying about running out of IP addresses. Cilium’s advanced networking and LoadBalancer integration with BGP ensures that newly assigned LoadBalancer IPs are routed efficiently, making services instantly accessible.

This approach significantly enhances our deployment’s scalability and flexibility, enabling expansion without the overhead of managing static routes or IP conflicts.

 

Step #1: Environment

 

Assumptions & Prerequisites.

To follow this guide, ensure the following prerequisites are met.

  • Configured and functional pfSense: pfSense should be fully operational to facilitate BGP peering with Cilium, ensuring proper routing between your Kubernetes cluster and external networks.
  • Dedicated VLAN for BGP Peering: For security and isolation, it’s recommended to create a dedicated VLAN for BGP communication between your Kubernetes cluster and pfSense.
  • Kubernetes Cluster with Cilium Installed: Ensure your Kubernetes cluster is set up with Cilium in a “green state” (i.e., without errors) and is fully functional within the designated BGP VLAN.
  • Private ASN Allocation for BGP: You will need two Autonomous System Numbers (ASNs) for BGP peering—one for pfSense and one for Cilium.
  • Understanding of the BGP Protocol and Kubernetes Architecture: Basic knowledge of BGP and Kubernetes will be helpful in understanding how the routing and peering are set up.

 

Network Information:

Below are the network configurations used in this guide. Feel free to adjust them to fit your environment.

  • pfSense ASN: 64520
  • Cilium ASN: 64521
  • LAN: 10.145.10.0/24 (Local network where your cluster resides)
  • BGP/Cluster Network: 10.146.10.0/24 (Network used for BGP peering between pfSense and Kubernetes)
  • LoadBalancer Network: 10.147.10.0/24 (Network for LoadBalancer IPs to be routed)

 

Diagram:

 

 

 

 

Step #2: pfSense Setup

In this step, we configure pfSense as a BGP router using FRR (Free Range Routing). FRR allows dynamic route exchanges between pfSense and Kubernetes via Cilium. We will install and enable FRR, set up BGP peering, and configure route maps to ensure proper routing.

 

1. Install FRR on pfSense

To enable BGP functionality, install FRR (Free Range Routing) on pfSense:

  • System → Package Manager → Available Packages

 

2. Enable FRR

Once installed, enable FRR Global/Zebra:

  • Services → FRR Global/Zebra → Global Settings

    • Enable: Checked

    • Master Password: <insert-a-master-password>

    • Syslog Logging: Checked

 

3. Create a Route Map

Route maps control which routes are accepted or advertised by BGP. We will create a simple ALLOW-ALL route map to permit all routes between pfSense and Cilium.

  • Services → FRR Global/Zebra → Route Maps → Add

    • Name: ALLOW-ALL

    • Description: Match any route

    • Action: Permit

    • Sequence: 10

 

4. Enable BGP Routing

Now, enable BGP (Border Gateway Protocol) and set the Local ASN (Autonomous System Number):

  • Services → FRR BGP → BGP

    • Enable: Checked

    • Log Adjacency Changes: Checked

    • Local AS: 64520

    • Router ID: 10.146.10.254

📌 Note: Cilium does not install received BGP routes. It only advertises routes, meaning pfSense (or another upstream router) must handle route propagation.

 

5. Create BGP Neighbors

Next, configure BGP neighbors for the Kubernetes control plane and worker nodes.

➤ Add the Control Plane Node

  • Services → FRR BGP → Neighbors → Add

    • General Options

      • Name/Address: 10.146.10.20

      • Description: control-plane

    • Basic Options:

      • Remote AS: 64521

    • Peer Filtering → Route Map Filters:

      • Inbound Route Map Filter: ALLOW-ALL - Match any route

      • Outbound Route Map Filter: ALLOW-ALL - Match any route

➤ Add the Worker Node

  • Services → FRR BGP → Neighbors → Add

    • General Options

      • Name/Address: 10.146.10.30

      • Description: worker-node

    • Basic Options:

      • Remote AS: 64521

    • Peer Filtering → Route Map Filters:

      • Inbound Route Map Filter: ALLOW-ALL - Match any route

      • Outbound Route Map Filter: ALLOW-ALL - Match any route

 

Step #3: Cilium Setup

 

1. Enable the BGP Control Plane

Run the following command to enable Cilium’s BGP control plane:

 

2. Create the Required Manifests

These manifests configure BGP peering between Cilium and pfSense.

 

3. Deploy the BGP Configuration

Apply all the manifests to the cluster:

 

4. Restart Cilium Services

After deploying the configurations, restart Cilium to ensure changes take effect:

 

Step #4: Validation & Testing

Once the configuration is complete, follow these steps to validate that BGP peering and route propagation are working correctly.

 

1. Verify pfSense BGP Neighbor Status

Check if pfSense has successfully established BGP sessions with the Cilium nodes.

Navigate to:

  • Status → FRR → BGP Summary

Expected Result:
Both Cilium nodes (control-plane and worker-node) should be listed as established neighbors.

 

2. Verify Received Routes on pfSense

Ensure pfSense is correctly receiving routes from the Cilium nodes.

Navigate to:

  • Status → FRR → BGP Routes

Expected Result:
You should see the routes received from the Cilium nodes, corresponding to the LoadBalancer IPs.

 

3. Check Cilium BGP Peers

Run the following command to verify BGP peer connections from Cilium.

Expected Output:

Key Indicators:

  • Session State: established → Peering is successful.
  • Received & Advertised Routes: Ensure there are valid entries.

 

4. Check Cilium Advertised Routes

Run the following command to verify which routes Cilium is distributing to pfSense.

Expected Output:

This confirms that the LoadBalancer IPs are being properly advertised by Cilium and received by pfSense.

 

Conclusion

By integrating Cilium’s BGP control plane with pfSense, we’ve established a scalable and efficient way to announce Kubernetes LoadBalancer IPs dynamically. Unlike traditional Layer 2 advertisements, BGP provides greater flexibility, eliminates subnet limitations, and ensures seamless route propagation across the network.

With this setup, new Kubernetes services can be deployed without manual IP management, and pfSense dynamically receives and routes these addresses, ensuring high availability and smooth network operations. This approach is ideal for growing infrastructures where scalability, automation, and reliability are key priorities.

 

About Us

KASTERVO is a cloud service provider specializing in security, high availability, and cost-efficient infrastructure. Founded by a team of passionate I.T. engineers, we help businesses reduce capital and operational expenses through our premium Private Cloud solutions.

As a strategic partner of Veeam and Microsoft, we enable seamless integration of Private Cloud with Microsoft 365 and Veeam backup solutions, ensuring robust data protection and business continuity.

Beyond cloud hosting, our team has developed industry-leading standards and procedures for Managed Cloud services, including Systems Hardening, proactive monitoring, maintenance, and seamless cloud integration tailored to each client’s needs.

Accelerate your cloud journey with KASTERVO—contact us today at: https://kastervo.com/contact

Search

Category

Latest Articles

Tags

post-thumb

BY KASTERVO / ON January 10, 2024

Navigating Information Security

In the realm of information security, safeguarding organizational assets against potential threats is imperative. While the conventional wisdom advises executives to champion security policies and systems, it is equally pertinent to pose the critical question: “How much security is truly necessary?” The answer, inevitably, rests on the unique dynamics of the business and the nature of the information requiring protection. This discourse aims to delve into this essential consideration.

Introduction

Before delving into the intricate nuances of security measures, establishing a baseline for essential practices for anyone utilizing an internet-connected device is crucial.

Foundational Practices:

  • Automatic and/or frequent Software Updates: The vulnerabilities inherent in the software we employ necessitate continuous patching to eliminate discovered vulnerabilities.
  • Antivirus Software: A foundational layer of defense, investing in a reputable antivirus solution is imperative for the protection of digital assets.

Levels of Security

For the purpose of clarity, we categorize security requirements into six distinct levels, facilitating a systematic understanding and implementation approach.

Level 0: Fundamental Protection

  • Automatic Updates: Regularly update software to patch vulnerabilities.
  • Antivirus Software: Invest in a reliable antivirus solution as a baseline defense.

Level 1: Data Integrity Assurance

  • Automatic Encrypted Backup: Implement automated, encrypted backup solutions to ensure data integrity.

Level 2: Enhanced Resilience

  • Multiple Backup Locations: Diversify backup locations for redundancy.
  • Strong Password Policies: Enforce robust password practices.
  • MFA on Critical Accounts: Implement Multi-Factor Authentication (MFA) for crucial accounts.

Level 3: Comprehensive Security Framework

  • Documented Security Policies & Procedures: Establish clear security documentation.
  • Strict Access Control: Rigorously manage and monitor user access.
  • DR Planning: Develop detailed Disaster Recovery (DR) plans.
  • Central Logging: Consolidate logs for comprehensive monitoring.
  • Central MFA Solution: Implement a centralized MFA solution.
  • Vulnerability Management: Regularly assess and manage system vulnerabilities.

Level 4: Advanced Safeguards

  • Managed Security Service Providers (MSSP): Consider external experts for specialized security services.
  • Frequent Penetration Testing: Regularly conduct thorough penetration testing.

Level 5: In-House Fortification

  • Security Operations Center (SOC): Establish an internal SOC for continuous monitoring and response.

These levels serve as a guideline, each building upon the last to create a comprehensive security posture. Tailoring these practices to your business’s unique requirements is crucial for effective information security.

Key Considerations

The pivotal point lies in asking strategic questions to determine the requisite level of security:

  • Does my business heavily rely on vital documents for its operations?
  • How critical are these documents to the core of my business?
  • Can the information be reconstructed if lost?
  • Do I handle confidential information for clients, such as Personally Identifiable Information (PII), Trade Secrets, or Health Information?
  • Does my business operate in high-risk industries like IT, Finance, Health, or Law?
  • Is my organization subject to specific compliance requirements (e.g. GDRP, PCI-DSS, ISO, NIST, etc.)?
  • What is the significance of continuous security monitoring for my organization?
  • Is there a budgetary allowance for establishing a Security Operations Center (SOC) and hiring internal security professionals?

Desired Outcome

The ultimate goal of information security is to shield the business, its employees and customers. Implementing security practices requires a meticulous determination of what needs protection and its corresponding value. If the cost of security measures surpasses the value of the assets under protection, a reassessment of the security strategy may be warranted.

Conclusion

The topic of information security demands a nuanced approach. This article serves as a general guide on initiating organizational security practices. Recognizing the uniqueness of each business is paramount in crafting tailored security strategies.

About Us

At KASTERVO, we redefine possibilities in the digital realm, steering businesses towards unparalleled success through innovative cloud solutions. As a leading cloud service provider, we bring a wealth of experience and expertise to the table, empowering organizations to thrive in an ever-evolving technological landscape.