Multi-Tenancy in Amazon EKS: Secure, Scalable Kubernetes Isolation with Quotas, Observability & DR

Multi-Tenant Models in EKS

Multi-Tenancy in Amazon EKS — Purpose, When to Use, and How

Why & When to Use ResourceQuota in Multi-Tenant EKS

Why & When to Use NetworkPolicy in Multi-Tenant EKS

What This Policy Does

AWS Config Rule Example

ConfigRule:
  name: restrict-common-ports
  source:
    owner: AWS
    sourceIdentifier: RESTRICTED_INCOMING_TRAFFIC

# Ensures restricted ports (22, 3389) are not publicly exposed

Explanation

Why & When to Use This Query in Multi-Tenant EKS

How to Use It

1. Go to CloudWatch Logs Insights in the AWS Console

2. Choose your log group, usually something like /aws/containerinsights/<cluster-name>/application

3. Paste and run the query above

4. Optionally, export results or build CloudWatch dashboards per tenant

Tip

• Combine with @logStream, @log, or level fields to refine further

• Add time range filters to analyze incidents (e.g., spikes, errors, outages)

• Can be embedded in automated alert pipelines per tenant

Configure Alertmanager with routing rules to isolate alerts by namespace/team.

Query Breakdown

What It Does

This expression checks whether Tenant A’s app is returning a high rate of 5xx HTTP errors over the last 5 minutes. It's often used to trigger alerts via Alertmanager or Slack when an app is failing health checks or under load.

Why & When to Use This in Multi-Tenant EKS

How to Use It

• Add this to Prometheus alerting rules:

  - alert: TenantAHigh5xxRate
    expr: rate(http_requests_total{namespace="tenant-a", status=~"5.."}[5m]) > 0.05
    for: 2m
    labels:
      severity: critical
    annotations:
      summary: "High 5xx error rate for tenant-a"
      description: "Tenant A is returning > 5% 5xx responses in the last 5 minutes."
  

• Route alerts in Alertmanager by namespace or tenant label

• Visualize this in Grafana for per-tenant dashboards

Multi-tenant PromQL expressions for monitoring key SLOs like latency, availability, and 4xx error rates — scoped to a specific tenant (tenant-a). These are ideal for per-tenant alerts, dashboards, and SLA reporting in EKS environments.

1. Availability (Success Rate)

(
  sum(rate(http_requests_total{namespace="tenant-a", status!~"5..|4.."}[5m]))
  /
  sum(rate(http_requests_total{namespace="tenant-a"}[5m]))
) < 0.99

# Success = all non-4xx/5xx responses; fires if success ratio < 99%

What It Does:
Triggers if successful requests fall below 99% (i.e., >1% 4xx or 5xx errors) in tenant-a.

Use For: SLA enforcement, tenant-specific uptime monitoring.

2. 4xx Error Rate (Client Errors)

rate(http_requests_total{namespace="tenant-a", status=~"4.."}[5m]) > 0.1

# Fires if client-side errors exceed 0.1/sec

 What It Does:
Alerts when client errors exceed 0.1 req/sec (e.g., bad requests, unauthorized).

 Use For: API misuse, auth/token issues, broken frontend calls.

3. Latency (p95 Response Time)

histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{namespace="tenant-a"}[5m])) by (le))
> 0.3

# Tracks 95th percentile latency over 5m; alerts if > 300ms 

What It Does:
Triggers when p95 latency for requests in tenant-a exceeds 300ms.

 Use For: Performance SLO monitoring, backend tuning, per-tenant latency SLI.

How to Use These

• Plug into Prometheus alerting rules

• Visualize in Grafana (e.g., per-tenant dashboards)

• Route via Alertmanager by tenant label

• Feed into SLA/uptime reports for customer-facing environments

4. Tenant Provisioning: vCluster + CRD Workflow

vCluster Setup Example

vcluster create tenant-a --namespace tenant-a --set sync.nodes.enabled=false

# Creates an isolated virtual Kubernetes control plane in 'tenant-a'

CRD-Based Automation (via ArgoCD/Crossplane)

apiVersion: platform.indipay.io/v1alpha1
kind: Tenant
metadata:
  name: tenant-a
spec:
  namespace: tenant-a
  quota:
    cpu: 2
    memory: 8Gi
  networkPolicy: default-deny
  vcluster: true

# Automate onboarding using ArgoCD, Crossplane, or Terraform workflows

5. Istio Service Mesh for Tenant mTLS & Routing

Use Case: mTLS Per Tenant + Ingress Policy

• Enable mTLS: encrypts intra-tenant communication

• Use Gateway + VirtualService to route based on hostname/namespace

• Control egress via ServiceEntry per tenant

Enforce mTLS + Authorization

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: tenant-a-dr
spec:
  host: tenant-a.svc.cluster.local
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL  # Enforces encrypted tenant traffic
------------

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: allow-only-tenant-a
  namespace: tenant-a
spec:
  rules:
  - from:
    - source:
        namespaces: ["tenant-a"]
# Allows only traffic from within the same tenant

6. Tenant-Aware Autoscaling with Custom Metrics

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: tenant-a-api
  namespace: tenant-a
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Pods
    pods:
      metric:
        name: custom_requests_per_second
      target:
        type: AverageValue
        averageValue: "10"

# Scales up/down based on request rate per pod

7. Compliance Mapping with AWS Native Tools

Are These Practices Recommendable?

Reminder: Things to Do Before Proceeding with This Solution

Before implementing multi-tenancy in Amazon EKS, make sure you've accounted for the following considerations to avoid rework, production outages, or compliance gaps.

We are going to set up a dedicated “tenant bootstrap pipeline” to automate namespace creation, quota application, NetworkPolicy, secrets, service account roles, and observability labels using GitOps (ArgoCD) or IaC (Terraform + Helm).

Tenant Bootstrap Pipeline Example (GitOps/IaC)

This example automates secure, consistent tenant onboarding using Helm, ArgoCD, and optionally Terraform. Every tenant gets a namespace, quota, network policy, and cost tagging — GitOps-ready.

Step 1: Helm Chart Folder Structure

tenant-bootstrap/
├── Chart.yaml             # Helm chart metadata
├── values.yaml            # Per-tenant configuration (CPU, memory, etc.)
└── templates/
    ├── namespace.yaml     # Namespace definition
    ├── resourcequota.yaml # Resource quotas
    ├── networkpolicy.yaml # Default-deny ingress policy

 namespace.yaml – Create a Namespaced Environment

apiVersion: v1
kind: Namespace
metadata:
  name: {{ .Values.tenant.name }}                     # Namespace per tenant
  labels:
    tenant: {{ .Values.tenant.name }}                 # Useful for filtering
    cost-center: {{ .Values.tenant.costCenter }}      # Tag for chargeback tracking

Why: This ensures logical and operational isolation of workloads and costs.

 resourcequota.yaml – Enforce Fair Resource Usage

apiVersion: v1
kind: ResourceQuota
metadata:
  name: {{ .Values.tenant.name }}-quota
  namespace: {{ .Values.tenant.name }}
spec:
  hard:
    cpu: {{ .Values.tenant.quota.cpu }}              # e.g., "2000m" = 2 vCPU
    memory: {{ .Values.tenant.quota.memory }}        # e.g., "8Gi"
    pods: {{ .Values.tenant.quota.pods }}            # e.g., 50 pods

 Why: Prevents a single tenant from over-consuming cluster resources.

networkpolicy.yaml – Zero-Trust by Default

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: {{ .Values.tenant.name }}
spec:
  podSelector: {}       # Applies to all pods
  ingress: []           # Blocks all ingress traffic by default

Why: Avoids lateral movement and enforces default security posture.

values.yaml – Config Per Tenant (User-Defined)

tenant:
  name: tenant-a
  costCenter: finance
  quota:
    cpu: "2000m"
    memory: "8Gi"
    pods: "50"

 Why: Allows customization per tenant for CPU, memory, labeling, and security.

Step 2: ArgoCD App (GitOps-Driven Deployment)

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: tenant-a-bootstrap
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://git.example.com/bootstrap-charts
    targetRevision: HEAD
    path: tenant-bootstrap
    helm:
      valueFiles:
        - values.yaml
  destination:
    server: https://kubernetes.default.svc
    namespace: tenant-a
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Why: Declaratively applies all tenant onboarding resources and keeps them in sync.

Step 3: Optional Terraform Trigger

resource "helm_release" "tenant_a" {
  name       = "tenant-a-bootstrap"
  chart      = "./charts/tenant-bootstrap"
  namespace  = "tenant-a"
  values     = [file("${path.module}/tenant-a-values.yaml")]
}

Why: Automate provisioning via CI/CD pipelines or Terraform cloud-based workflows.

What This Bootstrap Enables

Lessons Learned: Building Production-Grade Multi-Tenant EKS

1. Isolation is a strategic decision, not a toggle.
   Whether you use namespace isolation, vClusters, or separate EKS clusters depends on your tenant’s risk profile, compliance needs, and cost-to-scale ratio. Choose wisely based on context—not convenience.

2. Resource quotas are non-negotiable.
   Without enforced quotas and LimitRanges, a single runaway workload can degrade the entire cluster. Quotas are your first line of protection for fairness, stability, and cost control.

3. Start with a deny-all security posture.
   Always assume zero trust. Apply default-deny NetworkPolicies and build explicit exceptions. Most cross-tenant issues stem from overly permissive defaults.

4. Observability must be tenant-scoped.
   Dashboards, alerts, and logs should be isolated per tenant. Shared metrics create noise, confusion, and reduce the ability to meet SLOs and SLAs individually.

5. Automate tenant provisioning from day one.
   Manual onboarding won’t scale. GitOps-based bootstrap pipelines ensure consistency, version control, and faster rollout with fewer errors.

6. Design for compliance upfront.
   If you're in fintech, healthcare, or any regulated industry, build for auditability now—not later. Enable CloudTrail, set up audit logs, and tag everything.

7. Cost tracking starts with labels.
   Add tenant and cost-center labels to all workloads. Tools like Kubecost/OpenCost only work well when tagging is enforced consistently.

8. Disaster Recovery should be tenant-aware.
   DR plans must include namespace-level backups, targeted restores, and tenant-specific failover strategies—especially in high-SLA SaaS environments.

9. Avoid the trap of one-size-fits-all.
   Tenants may have different performance needs, policies, and compliance requirements. Treat them accordingly with custom quotas, alerts, and scaling rules.

10. Educate and empower tenant teams.
    Provide documentation, dashboards, and self-service portals explaining their environment—quotas, RBAC scopes, alerting pipelines, and limits. Transparency reduces friction and ticket load.

Multi-Tenancy in Amazon EKS — Complete Workflow

1.  Design Phase
   ├── Decide Isolation Model:
   │     ├── Soft: Namespaces + RBAC
   │     └── Hard: vClusters / Separate EKS clusters
   ├── Define tenant SLAs, SLOs, compliance boundaries
   └── Plan for resource limits, security, observability, and DR

2.  Tenant Bootstrap (GitOps/IaC)
   ├── Create namespace per tenant
   ├── Apply ResourceQuota & LimitRanges
   ├── Apply default deny-all NetworkPolicy
   ├── Tag workloads (e.g., tenant, cost-center)
   └── Assign scoped RBAC/IAM roles

3.  Observability Per Tenant
   ├── Enable CloudWatch or Loki logs
   ├── Configure Prometheus metrics + rules (e.g., 5xx alerts)
   ├── Grafana dashboards per namespace
   ├── Route alerts via Alertmanager per tenant/team
   └── Optional: tie alerts to SLOs, SLAs, Slack channels

4.  Security & Compliance
   ├── Enforce NetworkPolicies and Secrets encryption
   ├── Enable EKS Audit Logs and CloudTrail filters
   ├── Define compliance configs (PCI-DSS, HIPAA, SOC2)
   └── Apply AWS Config rules + tenant-based audit logs

5.  Cost Management
   ├── Use Kubecost/OpenCost with namespace or label tracking
   ├── Enforce quotas to cap runaway usage
   ├── Add budget alerts per tenant or team
   └── Build chargeback reports (via cost-center tagging)

6.  Automation & Scaling
   ├── Use HPA/VPA with tenant-specific metrics
   ├── Automate onboarding with ArgoCD, Terraform, Helm
   ├── Build GitOps pipelines for tenant lifecycle
   └── Integrate CI/CD for quota-aware deployments

7.  Disaster Recovery (Tenant-Aware)
   ├── Backup/restore at namespace level (e.g., Velero)
   ├── Cross-region replication for critical tenants
   └── Run automated DR tests with AWS FIS / ChaosMesh

8.  Advanced Enhancements (Optional But Powerful)
   ├──  Integrate vClusters for hard multi-tenancy isolation
   ├──  Apply Istio mTLS per tenant (strict service-to-service control)
   ├──  Enforce policies using OPA/Gatekeeper (e.g., quota, naming)
   └──  Offer GUI portals for tenant self-service provisioning and visibility

9.  Continuous Testing & Optimization
   ├── Schedule GameDays to validate isolation & DR
   ├── Monitor tenant-specific quota violations or SLA breaches
   ├── Tune HPA thresholds, alert thresholds, and policies
   └── Regularly audit IAM, RBAC, and compliance artifacts