Part 6: Ingress and Certificate Management

Exposing services to the internet seems simple until you're debugging SSL certificate failures at midnight. After wrestling with nginx-ingress, cert-manager misconfigurations, and DNS-01 challenges, I've settled on a rock-solid Traefik setup. This article covers ingress routing with detailed explanations, automatic TLS certificates, OAuth middleware, and the gotchas that will save you hours.

Prerequisites

Required DNS Records:

Create A (or AAAA if using IPv6) records pointing to your LB IP. If you use split-horizon DNS, internal A/AAAA can point to 192.168.0.203; otherwise use your public IP (or your proxy/WAF IP).

A: homelab.example      → 192.168.0.203  # or public IP
A: *.homelab.example    → 192.168.0.203  # wildcard for subdomains
Optional CAA: issue "letsencrypt.org"
Suggested TTL: 60–300s during setup, raise after stable

Understanding Kubernetes Ingress

What is Ingress?

Ingress is how external traffic reaches services inside your Kubernetes cluster. Think of it as a smart reverse proxy that sits at the edge of your cluster and routes incoming requests to the right services based on hostnames and paths.

Key Ingress Concepts:

Ingress Controller: The actual proxy software (like Traefik, nginx, HAProxy)
Ingress Resource: Kubernetes objects that define routing rules
External IP: The IP address that external clients connect to
TLS Termination: Where HTTPS encryption/decryption happens
Load Balancing: Distributing traffic across multiple backend pods

Why Traefik Over Other Ingress Controllers

I've run nginx-ingress, HAProxy, and Traefik in production. Here's why Traefik wins for homelabs:

Native Kubernetes Integration

CRDs (Custom Resource Definitions): Configure everything using Kubernetes YAML instead of complex config files
Automatic service discovery: Traefik watches Kubernetes for new services and automatically routes to them
Real-time configuration updates: Changes apply instantly without restarts or config reloads

Built-in Middleware

OAuth/OIDC authentication: Protect services with forward auth (like PocketID)
Rate limiting: Prevent abuse by limiting requests per minute/hour
Circuit breakers: Automatically stop sending traffic to failing backends
Header manipulation: Add security headers, modify requests/responses

Superior Observability

Built-in dashboard: Web UI showing real-time traffic, routes, and health
Prometheus metrics: Ready-to-use metrics for monitoring
Detailed access logs: Customizable JSON logs for debugging

Let's Encrypt Integration

Multiple ACME providers: Use different providers for different domains
Automatic certificate renewal: Certificates renew by default ~30 days before expiry (configurable)
Wildcard certificate support: One cert for *.example.com (requires DNS-01 challenges)

Cilium vs Traefik Ingress

Understanding Your Options:
Cilium provides both LoadBalancer services (L4 load balancing) and Gateway API ingress (L7 routing), while Traefik focuses on feature-rich L7 ingress with middleware. Here's when to use each:

Cilium Ingress (Gateway API)

Best for:

Simple HTTP/HTTPS routing without complex middleware
Standards-based configuration (Gateway API)
Maximum performance with eBPF data plane
Unified networking (CNI + ingress in one component)

Cilium Ingress Features:

Gateway API compliance: Use standard Gateway and HTTPRoute resources
eBPF acceleration: in-kernel eBPF datapath (kube-proxy bypass) for ultra-low latency
L4/L7 load balancing: TCP and HTTP load balancing
TLS termination: References certificate Secrets (cert-manager populates them)
Path-based routing: Host and path matching

Traefik Ingress

Best for:

Complex routing scenarios with middleware chains
OAuth/OIDC authentication integration
Advanced features (rate limiting, circuit breakers, etc.)
Mature ecosystem with extensive documentation

Why We Choose Traefik:
For a homelab requiring authentication, security headers, rate limiting, and complex routing, Traefik's middleware ecosystem is unmatched. Cilium excels at high-performance simple routing, but Traefik provides the feature depth needed for production services.

Setting Up Cilium Gateway API (Production Example)

Real-world Cilium Gateway setup for those preferring standards-based configuration with maximum performance:

Prerequisites

# Ensure Cilium is installed with Gateway API support
cilium install --version 1.16.0 --set gatewayAPI.enabled=true

# Install Gateway API CRDs
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.0.0/standard-install.yaml

Version Pinning Note: Always pin Gateway API CRD versions in production. Cilium's controller must be compatible with the installed CRD version. Check Cilium's Gateway API compatibility matrix before upgrading.

GatewayClass Configuration

# cilium-gatewayclass.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: cilium
spec:
  controllerName: io.cilium/gateway-controller
  description: "Cilium Gateway Controller using eBPF data plane"

Main Gateway Configuration

# cilium-gateway.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: cilium-gateway
  namespace: gateway-system
  annotations:
    # Use Cilium's L2 announcements for external IP
    lbipam.cilium.io/ips: "192.168.0.200"
spec:
  gatewayClassName: cilium
  listeners:
  # HTTP listener (with automatic HTTPS redirect)
  - name: web
    port: 80
    protocol: HTTP
    allowedRoutes:
      namespaces:
        from: All  # Allow routes from any namespace
  # HTTPS listener with wildcard certificate
  - name: websecure
    port: 443
    protocol: HTTPS
    allowedRoutes:
      namespaces:
        from: All
    tls:
      mode: Terminate
      certificateRefs:
      - name: wildcard-homelab-tls  # cert-manager populates this Secret
        namespace: traefik-system    # Cross-namespace cert reference

# Note: The Gateway creates a backing LoadBalancer Service.
# Even with lbipam.cilium.io/ips on the Gateway, some versions place the annotation
# on the generated Service. After creating the Gateway, verify and patch if needed:
# kubectl -n gateway-system get svc --show-labels
# kubectl -n gateway-system annotate svc cilium-gateway-cilium-gateway \
#   lbipam.cilium.io/ips="192.168.0.200" --overwrite
# kubectl -n gateway-system patch svc cilium-gateway-cilium-gateway \
#   --type merge -p '{"spec":{"loadBalancerClass":"io.cilium/l2-announcer"}}'

# Verify the resulting Service configuration:
# kubectl -n gateway-system get svc cilium-gateway-cilium-gateway -o yaml | yq '.spec.loadBalancerClass, .metadata.annotations."lbipam.cilium.io/ips"'
---
# Allow Gateway to reference TLS secret from traefik-system namespace
apiVersion: gateway.networking.k8s.io/v1beta1
kind: ReferenceGrant
metadata:
  name: allow-wildcard-cert
  namespace: traefik-system   # namespace where the Secret lives
spec:
  from:
  - group: gateway.networking.k8s.io
    kind: Gateway
    namespace: gateway-system # namespace where the Gateway lives
  to:
  - group: ""                 # core API group for Secret
    kind: Secret
    name: wildcard-homelab-tls

Apply and Wait for Readiness:

# Apply the configurations
kubectl apply -f cilium-gatewayclass.yaml
kubectl apply -f cilium-gateway.yaml
# ReferenceGrant is included in cilium-gateway.yaml above

# Wait for Gateway to be ready
kubectl wait --for=condition=Accepted gateway/cilium-gateway -n gateway-system --timeout=300s
kubectl wait --for=condition=Programmed gateway/cilium-gateway -n gateway-system --timeout=300s

# Wait for wildcard certificate (if using cert-manager)
kubectl wait --for=condition=Ready certificate/wildcard-homelab -n traefik-system --timeout=300s

HTTP Routes for Applications

# app-httproute.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: myapp-route
  namespace: apps
spec:
  parentRefs:
  - name: cilium-gateway
    namespace: gateway-system
    sectionName: websecure  # Use HTTPS listener
  hostnames:
  - myapp.homelab.example
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - name: myapp-service
      port: 80
      weight: 100
---
# Multiple backend example (load balancing)
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: api-route
  namespace: apps
spec:
  parentRefs:
  - name: cilium-gateway
    namespace: gateway-system
    sectionName: websecure
  hostnames:
  - api.homelab.example
  rules:
  # API v1 routes
  - matches:
    - path:
        type: PathPrefix
        value: /v1
    backendRefs:
    - name: api-v1-service
      port: 8080
      weight: 100
  # API v2 routes with load balancing
  - matches:
    - path:
        type: PathPrefix
        value: /v2
    backendRefs:
    - name: api-v2-primary
      port: 8080
      weight: 80
    - name: api-v2-canary
      port: 8080
      weight: 20
---
# HTTP to HTTPS redirect
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: redirect-https
  namespace: gateway-system
spec:
  parentRefs:
  - name: cilium-gateway
    sectionName: web  # HTTP listener
  hostnames:
  - "*.homelab.example"
  rules:
  - filters:
    - type: RequestRedirect
      requestRedirect:
        scheme: https
        statusCode: 301

Test HTTPRoute Connectivity:

# Smoke test your HTTPRoute
curl -H "Host: myapp.homelab.example" -k https://192.168.0.200/ -I
# Should return HTTP 200 with your app's headers

# Test HTTP → HTTPS redirect
curl -I http://192.168.0.200 -H "Host: myapp.homelab.example"
# Expect 301 Location: https://myapp.homelab.example/

Advanced Routing Examples

# advanced-httproute.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: advanced-routing
  namespace: apps
spec:
  parentRefs:
  - name: cilium-gateway
    namespace: gateway-system
    sectionName: websecure
  hostnames:
  - app.homelab.example
  rules:
  # Header-based routing
  - matches:
    - headers:
      - name: X-API-Version
        value: v2
    - path:
        type: PathPrefix
        value: /api
    backendRefs:
    - name: api-v2-service
      port: 8080
  # Query parameter routing
  - matches:
    - queryParams:
      - name: version
        value: beta
    - path:
        type: PathPrefix
        value: /features
    backendRefs:
    - name: beta-features-service
      port: 8080
  # Method-based routing
  - matches:
    - method: POST
    - path:
        type: PathPrefix
        value: /webhook
    backendRefs:
    - name: webhook-service
      port: 9000
  # Default route
  - matches:
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - name: frontend-service
      port: 80

Request/Response Modification

# header-modification.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: header-modification
  namespace: apps
spec:
  parentRefs:
  - name: cilium-gateway
    namespace: gateway-system
    sectionName: websecure
  hostnames:
  - api.homelab.example
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /api
    filters:
    # Security headers (add and remove in single filter)
    - type: ResponseHeaderModifier
      responseHeaderModifier:
        add:
        - name: X-Content-Type-Options
          value: nosniff
        - name: X-Frame-Options
          value: DENY
        - name: Strict-Transport-Security
          value: max-age=31536000; includeSubDomains
        remove:
        - Server
        - X-Powered-By
    # Add request headers for backend
    - type: RequestHeaderModifier
      requestHeaderModifier:
        add:
        - name: X-Forwarded-Proto
          value: https
        # Note: Gateway API doesn't support variable expansion
        # Headers must use literal values, not templates
    backendRefs:
    - name: api-service
      port: 8080

Multi-Cluster Gateway (Advanced)

# multi-cluster-routing.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: multi-cluster-route
  namespace: apps
spec:
  parentRefs:
  - name: cilium-gateway
    namespace: gateway-system
    sectionName: websecure
  hostnames:
  - global.homelab.example
  rules:
  # Route based on geographic headers
  - matches:
    - headers:
      - name: CF-IPCountry
        value: US
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - name: us-cluster-service
      port: 80
  - matches:
    - headers:
      - name: CF-IPCountry
        value: EU
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - name: eu-cluster-service
      port: 80
  # Default to closest cluster
  - matches:
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - name: primary-cluster-service
      port: 80

Monitoring Cilium Gateway

# cilium-gateway-servicemonitor.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: cilium-gateway
  namespace: gateway-system
spec:
  selector:
    matchLabels:
      # Match the Service created by the Gateway
      io.cilium.gateway/owning-gateway: cilium-gateway
  endpoints:
  - port: metrics
    interval: 30s
    path: /metrics

# Tip: Verify the Service labels match your selector:
# kubectl get svc -n gateway-system --show-labels

Key Cilium Gateway Advantages:

eBPF Data Plane: In-kernel eBPF datapath (kube-proxy bypass) for ultra-low latency
Standards Compliance: Pure Gateway API without vendor lock-in
Simplified Stack: CNI + ingress in one component
High Performance: Minimal overhead for simple routing scenarios
Multi-Cluster Ready: Built-in support for cluster mesh routing

When to Choose Cilium Gateway:

Maximum performance is critical
Standardized Gateway API is preferred
Simple L7 routing without complex middleware needs
Multi-cluster service mesh requirements
Unified CNI + ingress management

Cilium Gateway Limitations:

Fewer built-in L7 features compared to dedicated proxies
No authentication middleware: Requires external auth services
Limited rate limiting: Basic features only
No circuit breakers: Must implement at application level
Newer ecosystem: Fewer community examples and integrations

Performance Comparison (illustrative):

Cilium: Lower latency with in-kernel eBPF data plane (kube-proxy bypass)
Traefik: Higher latency but with rich L7 features and middleware

For maximum performance with simple routing, use Cilium Gateway API. For feature-rich ingress with authentication and middleware, Traefik is the better choice.

Installing Traefik

Deploy Traefik with Helm

What is Helm?
Helm is a package manager for Kubernetes, like apt for Ubuntu or brew for macOS. It uses "charts" (pre-packaged applications) and "values" (customization) to deploy complex applications easily.

Important: Do not configure Traefik ACME if cert-manager manages certificates—this will cause conflicts.

# traefik-values.yaml
# Helm chart version: traefik-30.0.2 (app version 3.2.0)
# For GitOps reproducibility: helm repo add traefik https://helm.traefik.io/traefik
deployment:
  enabled: true
  replicas: 2  # Run 2 copies for high availability (HA)
  podAnnotations: {}
  podLabels: {}
  # High availability scheduling (prevents externalTrafficPolicy: Local blackholes)
  topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: kubernetes.io/hostname
      whenUnsatisfiable: ScheduleAnyway
      labelSelector:
        matchLabels:
          app.kubernetes.io/name: traefik

# Use Cilium's LoadBalancer + L2 announcements
# This uses Cilium's native LB-IPAM with L2 announcements instead of MetalLB
# Prerequisites: Requires IPAddressPool (LB-IPAM) and L2AnnouncementPolicy configured in Cilium
# See Part 3 for Cilium L2 announcements setup or: https://docs.cilium.io/en/latest/network/l2-announcements/
# The annotation and loadBalancerClass below are the official way to request specific IPs with Cilium
service:
  enabled: true
  type: LoadBalancer
  annotations:
    lbipam.cilium.io/ips: "192.168.0.203"   # Official Cilium LB-IPAM annotation
  spec:
    loadBalancerClass: io.cilium/l2-announcer # Official Cilium L2 announcer class
    externalTrafficPolicy: Local              # Preserve client source IPs (no SNAT)
                                                # Ensure at least one Traefik pod on each node receiving traffic
                                                # (use DaemonSet or topology-aware scheduling)

# Important: If using externalTrafficPolicy: Local, run Traefik as a DaemonSet or ensure
# a pod on every node that may receive traffic to avoid traffic blackholes.
# (If some nodes won't ever receive LB traffic, taint or cordon them for Traefik pods)

# Enable dashboard (we'll secure it later)
ingressRoute:
  dashboard:
    enabled: false  # We'll create custom IngressRoute

# Configure ports that Traefik listens on
# Container ports >1024 allow running as non-root user for security
ports:
  web:  # HTTP port
    port: 8000            # Internal port (inside container, >1024 for non-root)
    exposedPort: 80       # External port (on LoadBalancer IP)
    protocol: TCP
  websecure:  # HTTPS port
    port: 8443            # Internal port (>1024 for non-root compatibility)
    exposedPort: 443      # External port (on LoadBalancer IP)
    protocol: TCP
    tls:
      enabled: true       # Enable TLS termination
  metrics:  # Prometheus metrics port (internal only)
    port: 9100            # Standard Traefik metrics port
    # exposedPort: 9100   # Omit to keep metrics internal
    protocol: TCP

# Enable Kubernetes Custom Resource Definitions
ingressClass:
  enabled: true
  isDefaultClass: true  # Make Traefik the default ingress controller

# Resource limits
resources:
  requests:
    cpu: 100m
    memory: 128Mi
  limits:
    cpu: 1000m
    memory: 512Mi

# Pod disruption budget
podDisruptionBudget:
  enabled: true
  minAvailable: 1

# Access logs
logs:
  general:
    level: INFO
  access:
    enabled: true
    format: json
    fields:
      defaultMode: keep
      headers:
        defaultMode: drop
        names:
          User-Agent: keep
          Authorization: drop
          Cookie: drop

# Prometheus metrics
metrics:
  prometheus:
    entryPoint: metrics
    buckets:  # Custom buckets (Traefik defaults: ≈0.1,0.3,1.2,5.0) - tune for your SLOs
      - 0.005
      - 0.01
      - 0.025
      - 0.05
      - 0.1
      - 0.25
      - 0.5
      - 1.0
      - 2.5
      - 5.0
      - 10.0

# Enable pilot (telemetry)
pilot:
  enabled: false  # Disable telemetry

# Additional command-line arguments for Traefik
additionalArguments:
  - "--global.checknewversion=false"     # Don't check for updates (privacy)
  - "--global.sendanonymoususage=false"  # Don't send telemetry
  # Note: Use ServersTransport CRD instead of global insecureSkipVerify for better security
  - "--api.dashboard=true"               # Enable web dashboard
  - "--api.debug=false"                  # Disable debug API
  - "--ping=true"                        # Enable ping endpoint
  - "--ping.entrypoint=websecure"         # Health check endpoint (avoid redirect issues)
  # Test: curl -sS -k https://traefik.homelab.example/ping  (Should return: OK)
  - "--providers.kubernetesingress.allowexternalnameservices=true"  # Allow external services
  - "--providers.kubernetescrd.allowexternalnameservices=true"      # Allow in CRDs too
  - "--providers.kubernetescrd.allowCrossNamespace=true"             # Allow cross-namespace middleware refs (increases blast radius - scope with RBAC)
  # RBAC Check: kubectl auth can-i get secrets --all-namespaces --as=system:serviceaccount:traefik-system:traefik
  # This should return "no" unless you intentionally grant cluster-wide Secret access
  # Best Practice: Grant Traefik read on Secrets only in namespaces where you terminate TLS or use mTLS
  # Example RBAC to limit Secret access to traefik-system namespace only:
  # apiVersion: rbac.authorization.k8s.io/v1
  # kind: Role
  # metadata:
  #   name: traefik-secrets-reader
  #   namespace: traefik-system
  # rules:
  # - apiGroups: [""]
  #   resources: ["secrets"]
  #   verbs: ["get", "list", "watch"]
  # ---
  # apiVersion: rbac.authorization.k8s.io/v1
  # kind: RoleBinding
  # metadata:
  #   name: traefik-secrets-reader
  #   namespace: traefik-system
  # roleRef:
  #   apiGroup: rbac.authorization.k8s.io
  #   kind: Role
  #   name: traefik-secrets-reader
  # subjects:
  # - kind: ServiceAccount
  #   name: traefik
  #   namespace: traefik-system
  # Note: Global entrypoint redirects can interfere with cert-manager HTTP-01 challenges
  # Use per-route redirect middleware instead, or prefer DNS-01 for all certificates
  # Trust Cloudflare IP ranges for proper client IP detection behind proxy
  # Update these ranges periodically from: https://www.cloudflare.com/ips/
  - "--entrypoints.web.forwardedHeaders.trustedIPs=173.245.48.0/20,103.21.244.0/22,103.22.200.0/22,103.31.4.0/22,141.101.64.0/18,108.162.192.0/18,190.93.240.0/20,188.114.96.0/20,197.234.240.0/22,198.41.128.0/17,162.158.0.0/15,104.16.0.0/12,172.64.0.0/13,131.0.72.0/22,2400:cb00::/32,2606:4700::/32,2803:f800::/32,2405:b500::/32,2405:8100::/32,2a06:98c0::/29,2c0f:f248::/32"
  - "--entrypoints.websecure.forwardedHeaders.trustedIPs=173.245.48.0/20,103.21.244.0/22,103.22.200.0/22,103.31.4.0/22,141.101.64.0/18,108.162.192.0/18,190.93.240.0/20,188.114.96.0/20,197.234.240.0/22,198.41.128.0/17,162.158.0.0/15,104.16.0.0/12,172.64.0.0/13,131.0.72.0/22,2400:cb00::/32,2606:4700::/32,2803:f800::/32,2405:b500::/32,2405:8100::/32,2a06:98c0::/29,2c0f:f248::/32"

# Security settings
securityContext:
  runAsNonRoot: true
  runAsUser: 65532
  runAsGroup: 65532
  fsGroup: 65532
  readOnlyRootFilesystem: true
  seccompProfile:
    type: RuntimeDefault
  capabilities:
    drop:
      - ALL

# Persistence for certificates and state
persistence:
  enabled: true
  storageClass: ceph-block-fast  # Use our fast Ceph storage
  size: 1Gi                     # 1GB is plenty for certificates
  path: /data                    # Mount point inside container

Install Traefik:

# Create dedicated namespace for Traefik components
kubectl create namespace traefik-system

# Add Traefik's Helm repository
helm repo add traefik https://helm.traefik.io/traefik
helm repo update  # Refresh to get latest chart versions

# Install Traefik using our custom values
export TRAEFIK_CHART_VERSION=30.0.2  # Pin for reproducibility (app version 3.2.0)
helm install traefik traefik/traefik \
  --namespace traefik-system \    # Install in traefik-system namespace
  --version "$TRAEFIK_CHART_VERSION" \  # Pin to specific chart version
  --values traefik-values.yaml     # Use our configuration

# Wait for deployment to complete
kubectl -n traefik-system rollout status deployment/traefik

# Verify LoadBalancer got an IP
kubectl -n traefik-system get svc traefik
# Should show EXTERNAL-IP: 192.168.0.203

# Test the ping endpoint (requires --ping=true in additionalArguments)
curl -sS -k https://traefik.homelab.example/ping
# OK

Installing cert-manager

What is cert-manager?
cert-manager automates SSL/TLS certificate management in Kubernetes. It can request certificates from Let's Encrypt, automatically renew them before expiry, and store them as Kubernetes secrets. Think of it as an automated certificate authority client.

Key cert-manager Concepts:

Issuer/ClusterIssuer: Defines how to request certificates (Let's Encrypt, self-signed, etc.)
Certificate: Represents a certificate you want cert-manager to obtain and maintain
Challenge: How to prove you own a domain (HTTP-01 or DNS-01)
ACME: The protocol Let's Encrypt uses

Deploy cert-manager

# Install cert-manager from official manifests
# This includes CRDs, RBAC, and all required components
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.18.2/cert-manager.yaml

# Wait for the webhook to be ready (required for validation)
kubectl -n cert-manager rollout status deployment/cert-manager-webhook

# Verify all components are running
kubectl -n cert-manager get pods
# Should see: cert-manager, cert-manager-cainjector, cert-manager-webhook

Configure Let's Encrypt Issuers

What are Issuers?
Issuers tell cert-manager how to request certificates. ClusterIssuers work cluster-wide, while Issuers work in a single namespace.

Challenge Types:

HTTP-01: Let's Encrypt makes HTTP request to your domain to verify ownership (requires port 80 accessible)
DNS-01: You create a DNS TXT record to prove domain ownership (required for wildcard certificates)

# letsencrypt-issuers.yaml
---
# Staging issuer for testing (higher rate limits, fake certificates)
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory  # Staging API
    email: [email protected]                                    # Your email for notifications
    privateKeySecretRef:
      name: letsencrypt-staging-key  # Where to store account private key
    solvers:
    # HTTP-01 challenge for regular domains
    - http01:
        ingress:
          class: traefik  # Use Traefik for HTTP challenge
    # DNS-01 challenge for wildcard certificates
    - dns01:
        cloudflare:  # Using Cloudflare DNS provider
          apiTokenSecretRef:
            name: cloudflare-api-token  # Secret containing API token
            key: api-token
      selector:
        dnsZones:  # Only use DNS-01 for these zones
        - "homelab.example"
---
# Production issuer (real certificates, strict rate limits)
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory  # Production API
    email: [email protected]                           # Required for renewal notifications
    privateKeySecretRef:
      name: letsencrypt-prod-key  # Production account key (different from staging)
    solvers:
    - http01:  # For regular domain certificates
        ingress:
          class: traefik
    - dns01:   # For wildcard certificates
        cloudflare:
          apiTokenSecretRef:
            name: cloudflare-api-token
            key: api-token
      selector:
        dnsZones:
        - "homelab.example"
---
# Cloudflare API token secret (managed by Infisical)
apiVersion: secrets.infisical.com/v1alpha1
kind: InfisicalSecret
metadata:
  name: cloudflare-api-token
  namespace: cert-manager
spec:
  hostAPI: https://app.infisical.com/api
  authentication:
    universalAuth:
      credentialsRef:
        secretName: infisical-auth
        secretNamespace: infisical
  managedSecretReference:
    secretName: cloudflare-api-token      # K8s secret to create
    secretNamespace: cert-manager         # In cert-manager namespace
  secretsPath: /cert-manager              # Path in Infisical project
  projectSlug: homelab
  envSlug: prod

# Note: In Infisical, create a secret named "api-token" containing your
# Cloudflare API token with Zone:Read and DNS:Edit permissions

Apply the issuers:

# Apply the issuer configuration
kubectl apply -f letsencrypt-issuers.yaml

# Wait a moment for issuers to initialize
sleep 10

# Verify issuers are ready
kubectl get clusterissuers
# Expected output:
NAME                  READY   AGE
letsencrypt-staging   True    30s   # Ready means configuration is valid
letsencrypt-prod      True    30s

# If READY shows False, check the status:
kubectl describe clusterissuer letsencrypt-prod
# Look for error messages in the Status section

Wildcard Certificate

What is a Wildcard Certificate?
A wildcard certificate covers a domain and all its subdomains (*.example.com). Instead of managing separate certificates for each service, one wildcard certificate protects everything. Wildcard certificates require DNS-01 challenges since HTTP-01 cannot validate multiple subdomains.

Benefits:

One certificate for unlimited subdomains
Simpler management and renewal
Faster new service deployment
Reduced Let's Encrypt API calls

Create a wildcard certificate for all services:

# wildcard-certificate.yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: wildcard-homelab
  namespace: traefik-system  # Store in Traefik namespace for easy access
spec:
  secretName: wildcard-homelab-tls  # Kubernetes secret containing certificate
  dnsNames:
  - "homelab.example"      # Root domain
  - "*.homelab.example"    # All subdomains (app.homelab.example, etc.)
  issuerRef:
    name: letsencrypt-prod  # Use production issuer
    kind: ClusterIssuer
    group: cert-manager.io
  duration: 2160h    # Certificate valid for 90 days (Let's Encrypt standard)
  renewBefore: 720h  # Start renewal 30 days before expiry

# Request the wildcard certificate
kubectl apply -f wildcard-certificate.yaml

# Watch certificate provisioning process
kubectl -n traefik-system describe certificate wildcard-homelab
# Look for Events section showing progress:
# - Generated private key
# - Created DNS challenge
# - Waiting for DNS propagation
# - Certificate issued

# Monitor the status until ready
kubectl -n traefik-system get certificate wildcard-homelab -w
# Wait for READY column to show True (may take 2-5 minutes)

# Final verification
kubectl -n traefik-system get certificate wildcard-homelab
NAME               READY   SECRET                 AGE
wildcard-homelab   True    wildcard-homelab-tls   2m

# The certificate is now stored in the secret and ready to use
kubectl -n traefik-system get secret wildcard-homelab-tls

Traefik Middleware

What is Middleware?
Middleware is code that runs between receiving a request and sending it to your application. Traefik middleware can modify requests, add authentication, rate limiting, security headers, and more. Think of it as a pipeline of filters.

Common Middleware Types:

Authentication: Require login before accessing services
Security Headers: Add headers to protect against attacks
Rate Limiting: Prevent abuse by limiting request frequency
Compression: Reduce bandwidth usage
Path Manipulation: Modify URLs before sending to backend

OAuth Authentication with PocketID

What is Forward Auth?
Forward authentication sends each request to an auth service first. If the auth service says "OK," the request continues to your app. If not, the user gets redirected to login.

# pocketid-middleware.yaml
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: pocketid
  namespace: traefik-system
spec:
  forwardAuth:
    # URL of the auth service to check each request
    address: http://pocketid.pocketid.svc.cluster.local/auth/traefik?rd=https://auth.homelab.example
    trustForwardHeader: true  # Trust X-Forwarded-* headers
    authResponseHeaders:      # Headers to forward from auth service to app
      - X-Pocketid-User       # Username
      - X-Pocketid-Groups     # User's groups
      - X-Pocketid-Name       # Display name
      - X-Pocketid-Email      # Email address
---
# Security headers middleware
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: security-headers
  namespace: traefik-system
spec:
  headers:
    stsSeconds: 63072000        # HSTS: Force HTTPS for 2 years
    stsIncludeSubdomains: true  # Apply HSTS to all subdomains
    stsPreload: true           # Allow HSTS preload list inclusion (only if domain meets preload requirements)
    forceSTSHeader: true       # Send HSTS even on HTTPS
    contentTypeNosniff: true   # Prevent MIME type sniffing attacks
    referrerPolicy: "strict-origin-when-cross-origin"  # Control referrer info
    customFrameOptionsValue: "SAMEORIGIN"               # Prevent clickjacking
    customResponseHeaders:
      X-Robots-Tag: "noindex, nofollow"  # Tell search engines to ignore
      Server: ""                          # Hide server information
---
# Rate limiting middleware
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: rate-limit
  namespace: traefik-system
spec:
  rateLimit:
    average: 100    # Allow 100 requests per period on average
    burst: 200      # Allow bursts up to 200 requests
    period: 1m      # Rate limit period (1 minute)
    sourceCriterion:
      ipStrategy:
        depth: 0    # depth 0 + trustedIPs ensures we don't trust spoofed XFF from public internet
                    # Use ipStrategy for general abuse control; requestHeaderName for per-API key quotas
                    # With Cloudflare, real client IP comes via CF-Connecting-IP → X-Forwarded-For when trustedIPs is set
---
# Redirect HTTP to HTTPS
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: redirect-https
  namespace: traefik-system
spec:
  redirectScheme:
    scheme: https   # Redirect to HTTPS
    permanent: true # HTTP 301 (permanent redirect) for SEO
---
# Cache headers for static assets
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: cache-headers
  namespace: traefik-system
spec:
  headers:
    customResponseHeaders:
      Cache-Control: "public, max-age=31536000"  # Cache for 1 year
      # Note: ETag removed - let backend applications set appropriate ETags
---
# Chain multiple middlewares together
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: secure-chain
  namespace: traefik-system
spec:
  chain:  # Middlewares run in order
    middlewares:
    - name: redirect-https   # 1. Force HTTPS
    - name: security-headers # 2. Add security headers
    - name: rate-limit      # 3. Apply rate limiting
    # This creates a reusable security stack for all services

Exposing Services

Understanding IngressRoute vs Ingress:
Traefik supports both standard Kubernetes Ingress and its own IngressRoute CRD. IngressRoute is more powerful and supports all Traefik features. We'll use IngressRoute for full functionality.

Basic Service Exposure

# app-ingressroute.yaml
apiVersion: traefik.io/v1alpha1
kind: IngressRoute  # Traefik's enhanced ingress resource
metadata:
  name: myapp
  namespace: apps
spec:
  entryPoints:
    - websecure  # Only accept HTTPS traffic (port 443)
  routes:
    - match: Host(`myapp.homelab.example`)  # Match this hostname
      kind: Rule
      services:
        - name: myapp-service  # Backend Kubernetes service
          port: 80            # Port on the service
          # For plaintext gRPC upstreams, set scheme: h2c
      middlewares:            # Apply middleware chain
        - name: secure-chain
          namespace: traefik-system
  tls:
    options:
      name: tls-options              # TLS configuration (we'll create this)
      namespace: traefik-system
    # Note: Using default TLSStore for wildcard certificate

Advanced Routing

Routing Rules:
Traefik uses flexible routing rules that can match on host, path, headers, and more. Rules can be combined with AND/OR logic for complex routing.

# advanced-routing.yaml
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: complex-app
  namespace: apps
spec:
  entryPoints:
    - websecure
  routes:
    # API routes (higher priority = matched first)
    - match: Host(`api.homelab.example`) && PathPrefix(`/v1`)  # Match host AND path
      kind: Rule
      priority: 100  # Higher number = higher priority (100 wins over 90)
      services:
        - name: api-v1
          port: 8080
          sticky:      # Session affinity (user stays on same backend)
            cookie:
              name: api-affinity
              secure: true    # Only send over HTTPS
              httpOnly: true  # Not accessible via JavaScript
      middlewares:
        - name: api-ratelimit
          namespace: traefik-system

    # WebSocket routes (special handling for real-time connections)
    - match: Host(`app.homelab.example`) && PathPrefix(`/ws`)
      kind: Rule
      priority: 90  # Lower priority than API routes
      services:
        - name: websocket-service
          port: 8080
      # Note: Avoid compression middleware for WebSocket routes

    # Static content with caching
    - match: Host(`static.homelab.example`)
      kind: Rule
      services:
        - name: static-service
          port: 80
      middlewares:
        - name: cache-headers  # Set cache headers for static assets (defined separately)
          namespace: traefik-system

  tls: {}
    # Note: Using default TLSStore for wildcard certificate

Traefik Dashboard

Accessing the Dashboard:
Traefik includes a built-in web dashboard that shows real-time traffic, routes, services, and middleware. We'll expose it securely with authentication.

# traefik-dashboard.yaml
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: traefik-dashboard
  namespace: traefik-system
spec:
  entryPoints:
    - websecure
  routes:
    - match: Host(`traefik.homelab.example`)
      kind: Rule
      services:
        - name: api@internal  # Special service name that exposes Traefik's dashboard + API
          kind: TraefikService  # Special service type for internal Traefik services
      middlewares:
        - name: pocketid        # Require authentication (protect dashboard)
          namespace: traefik-system
        - name: security-headers
          namespace: traefik-system
  tls: {}
    # Note: Using default TLSStore for wildcard certificate

TLS Configuration

Why Configure TLS Options?
Default TLS settings might allow weak ciphers or old TLS versions. Custom TLS options ensure only secure protocols and ciphers are used, protecting against attacks.

Scoped Backend TLS Configuration

For backends that require insecure TLS verification (like self-signed certificates), use a scoped ServersTransport instead of global flags:

# serverstransport.yaml
apiVersion: traefik.io/v1alpha1
kind: ServersTransport
metadata:
  name: allow-insecure-backend
  namespace: apps
spec:
  serverName: "internal.example.local"
  insecureSkipVerify: true

Reference it from an IngressRoute service:

# In your IngressRoute spec:
routes:
  - match: Host(`myapp.homelab.example`)
    kind: Rule
    services:
      - name: myapp-service
        port: 443
        scheme: https
        serversTransport: allow-insecure-backend@kubernetescrd

This approach keeps TLS verification strict for most backends while allowing exceptions only where needed. The default@internal transport is used for all other services, maintaining secure TLS verification.

mTLS Backend Configuration

For backends requiring mutual TLS authentication:

# mtls-serverstransport.yaml
apiVersion: traefik.io/v1alpha1
kind: ServersTransport
metadata:
  name: mtls-backend
  namespace: apps
spec:
  serverName: "backend.internal"  # Must match a SAN on the backend cert
  # Client certificates for mTLS
  certificatesSecrets:
    - client-cert-secret  # Contains tls.crt and tls.key
  # Optional: Custom CA for backend verification
  rootCAsSecrets:
    - backend-ca-secret   # Contains ca.crt

Critical mTLS Requirement: serverName must match a SAN on the backend cert or verification will fail even with a custom CA.

mTLS Secret Requirements: client-cert-secret must be type kubernetes.io/tls (contains tls.crt, tls.key). backend-ca-secret may be type Opaque or kubernetes.io/tls but must contain ca.crt. Both must live in the same namespace as the ServersTransport.

Important TLS Secret Rule: If you set tls.secretName on an IngressRoute, the secret must live in the same namespace as the IngressRoute. Otherwise, use the default TLSStore pattern shown above.

RBAC Note: If using ServersTransport with mTLS secrets in other namespaces, create similar Role/RoleBinding pairs granting Traefik's ServiceAccount read access to Secrets in those specific namespaces only.

Strict TLS Options

# tls-options.yaml
apiVersion: traefik.io/v1alpha1
kind: TLSOption
metadata:
  name: tls-options
  namespace: traefik-system
spec:
  minVersion: VersionTLS12  # Minimum TLS 1.2 (TLS 1.0/1.1 are insecure)
  maxVersion: VersionTLS13  # Allow TLS 1.3 (latest and most secure)
  cipherSuites:    # TLS 1.2 only (TLS 1.3 suites are chosen by Go automatically)
    - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
    - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
    - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
    - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
    - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
    - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305
  curvePreferences:
    - CurveP256     # Most compatible
    - CurveP384
    - CurveP521
  sniStrict: true   # Require SNI (Server Name Indication) for security
---
# Default TLS options applied globally (name "default" is special)
apiVersion: traefik.io/v1alpha1
kind: TLSOption
metadata:
  name: default
  namespace: traefik-system
spec:
  minVersion: VersionTLS12
  maxVersion: VersionTLS13
  cipherSuites:    # TLS 1.2 only (TLS 1.3 suites are chosen by Go automatically)
    - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
    - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
    - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
    - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
    - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
    - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305
  curvePreferences:
    - CurveP256     # Most compatible
    - CurveP384
    - CurveP521
  sniStrict: true
---
# Set default certificate for all TLS connections
apiVersion: traefik.io/v1alpha1
kind: TLSStore
metadata:
  name: default  # Special name "default" is global - only one "default" TLSStore allowed cluster-wide
  namespace: traefik-system
spec:
  defaultCertificate:
    secretName: wildcard-homelab-tls  # Use our wildcard cert as default

Path-Based Routing

# path-routing.yaml
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: multi-service
  namespace: apps
spec:
  entryPoints:
    - websecure
  routes:
    # Strip path prefix
    - match: Host(`apps.homelab.example`) && PathPrefix(`/app1`)
      kind: Rule
      services:
        - name: app1-service
          port: 80
      middlewares:
        - name: strip-app1
          namespace: apps

    # Keep path
    - match: Host(`apps.homelab.example`) && PathPrefix(`/app2`)
      kind: Rule
      services:
        - name: app2-service
          port: 80

    # Regex matching
    - match: Host(`apps.homelab.example`) && PathRegexp(`^/api/v[0-9]+`)
      kind: Rule
      services:
        - name: api-service
          port: 8080

  tls: {}
    # Note: Using default TLSStore for wildcard certificate
---
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: strip-app1
  namespace: apps
spec:
  stripPrefix:
    prefixes:
      - /app1
    forceSlash: false

Load Balancing Strategies

# load-balancing.yaml
apiVersion: traefik.io/v1alpha1
kind: TraefikService
metadata:
  name: weighted-service
  namespace: apps
spec:
  weighted:
    services:
      - name: app-v1
        port: 80
        weight: 80  # 80% traffic
      - name: app-v2
        port: 80
        weight: 20  # 20% traffic (canary)
---
apiVersion: traefik.io/v1alpha1
kind: TraefikService
metadata:
  name: mirrored-service
  namespace: apps
spec:
  mirroring:
    name: app-primary
    port: 80
    mirrors:
      - name: app-shadow
        port: 80
        percent: 10  # Mirror 10% for testing

Using TraefikService in IngressRoutes:

# Reference TraefikService from IngressRoute
routes:
  - match: Host(`api.homelab.example`)
    kind: Rule
    services:
      - name: weighted-service
        kind: TraefikService  # Don't forget this!

Monitoring and Metrics

Prometheus ServiceMonitor

# traefik-servicemonitor.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: traefik
  namespace: traefik-system
spec:
  namespaceSelector:
    matchNames:
      - traefik-system
  selector:
    matchLabels:
      app.kubernetes.io/name: traefik
  endpoints:
    - port: metrics
      interval: 30s
      path: /metrics

# Note: Ensure your Prometheus is configured to discover ServiceMonitors
# across namespaces via serviceMonitorNamespaceSelector (kube-prometheus-stack
# enables this when configured properly)
# The Traefik Helm chart exposes a service port named "metrics" when Prometheus is enabled.
# IMPORTANT: The ServiceMonitor 'port' field must match the Service's port name exactly.
# If you rename the port in Helm values, update this selector accordingly.

Verify the metrics port is exposed after installation:

kubectl -n traefik-system get svc traefik -o jsonpath='{.spec.ports[?(@.name=="metrics")].name}{"\n"}'
# Should output: metrics

# Verify the Service port names match:
# kubectl -n traefik-system get svc traefik -o jsonpath='{.spec.ports[*].name}'

Key Metrics to Monitor

# Request rate
rate(traefik_service_requests_total[5m])

# Error ratio (5xx errors as percentage - multiply by 100 for dashboard display)
100 * rate(traefik_service_requests_total{code=~"5.."}[5m]) / rate(traefik_service_requests_total[5m])

# Request duration
histogram_quantile(0.95,
  rate(traefik_service_request_duration_seconds_bucket[5m])
)

# TLS certificate expiry
certmanager_certificate_expiration_timestamp_seconds - time()

Troubleshooting

Certificate Not Issued

# 1. Check certificate status and events
kubectl describe certificate -n traefik-system wildcard-homelab
# Look for:
# - Events section showing progress or errors
# - Status conditions for failure reasons

# 2. Check cert-manager controller logs
kubectl logs -n cert-manager deployment/cert-manager --tail=50
# Look for errors about:
# - ACME account registration
# - DNS provider authentication
# - Rate limiting

# 3. Check if challenges were created
kubectl get challenges -A
# DNS-01 challenges should appear for wildcard certs
# HTTP-01 challenges for regular certs

# 4. For DNS challenges, check if TXT record was created
# Replace with your domain:
dig TXT _acme-challenge.homelab.example

# Common issues and fixes:

# Issue 1: DNS-01 API token incorrect
# Fix: Verify Cloudflare token has Zone:Read + DNS:Edit permissions
kubectl get secret -n cert-manager cloudflare-api-token -o yaml

# Issue 2: HTTP-01 firewall blocking port 80
# Fix: Ensure port 80 is accessible from internet
curl -I http://yourdomain.com/.well-known/acme-challenge/test

# Issue 3: Let's Encrypt rate limits
# Fix: Use staging issuer first, then switch to prod
# Rate limits: 50 certificates per registered domain per week

# Issue 4: DNS propagation delays
# Fix: Wait longer or check DNS propagation
dig @8.8.8.8 TXT _acme-challenge.homelab.example

# Issue 5: CAA records blocking Let's Encrypt
# Fix: Add Let's Encrypt to your CAA records
# Example: homelab.example. CAA 0 issue "letsencrypt.org"

# Issue 6: DNS firewalls or WAF rules
# Fix: Temporarily disable bot rules for /.well-known/acme-challenge/
# Cloudflare: Ensure "orange cloud" doesn't alter the token path

Service Not Accessible

# 1. Check IngressRoute configuration
kubectl describe ingressroute -n apps myapp
# Look for:
# - Status section showing if route was accepted
# - Any warning events

# 2. Check Traefik logs for routing errors
kubectl logs -n traefik-system deployment/traefik --tail=50
# Look for:
# - "404 Not Found" = route not matching
# - "502 Bad Gateway" = backend service down
# - "503 Service Unavailable" = no healthy backends

# 3. Verify service has endpoints (pods are ready)
kubectl get endpoints -n apps myapp-service
# Should show IP addresses of ready pods
# If empty, pods aren't ready or service selector is wrong

# 4. Test service connectivity from inside cluster
kubectl run test --rm -it --image=alpine/curl -- \
  curl -v http://myapp-service.apps.svc.cluster.local
# This tests if the service itself works (bypasses ingress)

# 5. Check if LoadBalancer IP is reachable
ping 192.168.0.203
curl -H "Host: myapp.homelab.example" http://192.168.0.203

# 6. Debug DNS resolution
nslookup myapp.homelab.example
# Should point to your LoadBalancer IP

# 7. Common issues:
# - Wrong service name/namespace in IngressRoute
# - Service selector doesn't match pod labels
# - Pod failing health checks
# - DNS not pointing to LoadBalancer IP
# - Firewall blocking access

TLS Handshake Failures

# 1. Test TLS connection with verbose output
curl -vvv https://myapp.homelab.example
# Look for:
# - SSL certificate verification errors
# - Cipher suite negotiation failures
# - Protocol version mismatches

# 2. Check certificate details
openssl s_client -connect myapp.homelab.example:443 -servername myapp.homelab.example
# Verify:
# - Certificate is valid and not expired
# - Certificate matches the hostname
# - Complete certificate chain is present

# 3. Test specific TLS versions
openssl s_client -connect myapp.homelab.example:443 -tls1_2
openssl s_client -connect myapp.homelab.example:443 -tls1_3

# 4. Verify TLS configuration
kubectl get tlsoptions -n traefik-system -o yaml
# Check:
# - minVersion/maxVersion settings
# - Cipher suite restrictions
# - Curve preferences

# 5. Check if certificate secret exists and is valid
kubectl get secret -n traefik-system wildcard-homelab-tls
kubectl get secret -n traefik-system wildcard-homelab-tls -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -text -noout

# 6. Common TLS issues:
# - Certificate expired or not yet valid
# - Certificate doesn't match hostname (CN/SAN mismatch)
# - TLS options too restrictive for client
# - Incomplete certificate chain
# - Client using outdated TLS version

Security Best Practices

1. Always Use HTTPS

# Force HTTPS everywhere
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: force-https
  namespace: traefik-system
spec:
  redirectScheme:
    scheme: https
    permanent: true
    port: "443"

Important: These HTTP-01 considerations apply only to HTTP challenges—DNS-01 challenges are unaffected by WAF/HTTP proxies since they use DNS TXT records. Let's Encrypt will follow HTTP→HTTPS redirects for the token URL, but not to another host/port. Ensure port 80 is reachable and the ACME path is served as requested. The real failure modes are blocking port 80, redirecting to a different host/port, or WAF/bot rules that change the response. Note that cert-manager creates its own HTTP-01 Ingress; don't attach HTTPS-redirect middleware to that path, or prefer DNS-01 for all certificates.

cert-manager HTTP-01 Class Verification: If using HTTP-01 challenges, verify the ingress class name matches what Traefik registered:

kubectl get ingressclass
# Update cert-manager ClusterIssuer solver "class" if different from "traefik"

2. Implement Security Headers

apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: security-headers-strict
  namespace: traefik-system
spec:
  headers:
    contentSecurityPolicy: "default-src 'self'; frame-ancestors 'self'"
    permissionsPolicy: "camera=(), microphone=(), geolocation=()"
    referrerPolicy: "strict-origin-when-cross-origin"
    contentTypeNosniff: true
    customFrameOptionsValue: "SAMEORIGIN"
    stsSeconds: 63072000
    stsIncludeSubdomains: true
    stsPreload: true
    forceSTSHeader: true
    customResponseHeaders:
      X-Robots-Tag: "noindex, nofollow"
      Server: ""  # Best effort server banner hiding

# **Warning: Only set stsPreload: true if you intend to submit the domain to the preload list.**
# Otherwise users may be locked out. Ensure domain meets HSTS preload requirements first.

3. Rate Limiting by Path

apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: api-ratelimit
  namespace: traefik-system
spec:
  rateLimit:
    average: 10
    burst: 20
    period: 1m
    sourceCriterion:
      requestHeaderName: X-API-Key  # Per-API key quotas (spoofable if service is public without auth)

Performance Optimization

Enable Compression

apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: compress
  namespace: traefik-system
spec:
  compress:
    excludedContentTypes:
      - text/event-stream
    minResponseBodyBytes: 1024

Circuit Breaker

apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: circuit-breaker
  namespace: traefik-system
spec:
  circuitBreaker:
    expression: ResponseCodeRatio(500, 600, 0, 600) > 0.30
    checkPeriod: 10s
    fallbackDuration: 10s
    recoveryDuration: 10s

What's Next

With ingress and certificates automated, your services are securely accessible from anywhere. In Part 7, we'll deploy a complete observability stack with SigNoz, enabling deep insights into your cluster's behavior and performance.

Future-Proofing Note: Traefik v3 includes production-ready support for the Kubernetes Gateway API as an alternative to IngressRoute. While IngressRoute remains fully supported, Gateway API offers standardized multi-vendor configuration. You can swap IngressRoute with HTTPRoute/Gateway resources for vendor-neutral ingress configuration.

Key Takeaways

Traefik's CRDs make configuration declarative and GitOps-friendly
Wildcard certificates simplify TLS management for all subdomains
Middleware chains provide defense in depth without complexity
DNS-01 challenges enable wildcard certificates behind firewalls
Monitoring certificate expiry prevents outages from expired certs

References

Traefik Documentation: https://doc.traefik.io/traefik/
cert-manager Documentation: https://cert-manager.io/docs/
Let's Encrypt Rate Limits: https://letsencrypt.org/docs/rate-limits/
ACME Protocol: https://datatracker.ietf.org/doc/html/rfc8555
TLS Best Practices: https://wiki.mozilla.org/Security/Server_Side_TLS

Continue to Part 7: Complete Observability Stack →

Part 6: Ingress and Certificate Management

Josh Dow

Prerequisites

Understanding Kubernetes Ingress

What is Ingress?

Why Traefik Over Other Ingress Controllers

Cilium vs Traefik Ingress

Cilium Ingress (Gateway API)

Traefik Ingress

Setting Up Cilium Gateway API (Production Example)

Prerequisites

GatewayClass Configuration

Main Gateway Configuration

HTTP Routes for Applications

Advanced Routing Examples

Request/Response Modification

Multi-Cluster Gateway (Advanced)

Monitoring Cilium Gateway

Installing Traefik

Deploy Traefik with Helm

Installing cert-manager

Deploy cert-manager

Configure Let's Encrypt Issuers

Wildcard Certificate

Traefik Middleware

OAuth Authentication with PocketID

Exposing Services

Basic Service Exposure

Advanced Routing

Traefik Dashboard

TLS Configuration

Scoped Backend TLS Configuration

mTLS Backend Configuration

Strict TLS Options

Path-Based Routing

Load Balancing Strategies

Monitoring and Metrics

Prometheus ServiceMonitor

Key Metrics to Monitor

Troubleshooting

Certificate Not Issued

Service Not Accessible

TLS Handshake Failures

Security Best Practices

1. Always Use HTTPS

2. Implement Security Headers

3. Rate Limiting by Path

Performance Optimization

Enable Compression

Circuit Breaker

What's Next

Key Takeaways

References