External Secrets Operator

Panduan lengkap instalasi dan konfigurasi External Secrets Operator untuk sinkronisasi secrets dari external secret management systems ke Kubernetes.

Pengenalan

External Secrets Operator (ESO) adalah Kubernetes operator yang mengintegrasikan external secret management systems seperti AWS Secrets Manager, HashiCorp Vault, Google Secrets Manager, Azure Key Vault, dan lainnya dengan Kubernetes.

Masalah yang Dipecahkan

Tanpa External Secrets Operator:

❌ Secrets hardcoded dalam Git repository
❌ Manual sync secrets ke setiap cluster
❌ Sulit manage secrets rotation
❌ Tidak ada centralized secret management
❌ Secrets stored as plain Kubernetes Secrets

Dengan External Secrets Operator:

✅ Centralized Management: Single source of truth untuk secrets
✅ Automatic Sync: Otomatis pull secrets dari external providers
✅ Auto Rotation: Support automatic secret rotation
✅ Multiple Providers: Support 20+ secret backends
✅ Kubernetes Native: Menggunakan CRDs
✅ Security: Secrets tidak pernah di-commit ke Git

Arsitektur

┌─────────────────────────────────────────┐
│     External Secret Providers           │
│  (Vault, AWS, GCP, Azure, etc.)         │
└──────────────────┬──────────────────────┘
                   │
                   │ API Calls
                   │
┌──────────────────▼──────────────────────┐
│   External Secrets Operator             │
│  ┌────────────────────────────────┐    │
│  │  SecretStore Controller        │    │
│  └────────────────────────────────┘    │
│  ┌────────────────────────────────┐    │
│  │  ExternalSecret Controller     │    │
│  └────────────────────────────────┘    │
└──────────────────┬──────────────────────┘
                   │
                   │ Creates/Updates
                   │
┌──────────────────▼──────────────────────┐
│       Kubernetes Secrets                │
│   (Synchronized from external)          │
└─────────────────────────────────────────┘

Instalasi

Prasyarat

# Kubernetes cluster (minimal v1.19)
kubectl version --short

# Helm 3
helm version

Metode 1: Helm (Recommended)

# Add External Secrets Helm repository
helm repo add external-secrets https://charts.external-secrets.io
helm repo update

# Install External Secrets Operator
helm install external-secrets \
  external-secrets/external-secrets \
  --namespace external-secrets-system \
  --create-namespace \
  --set installCRDs=true

Metode 2: Kubectl dengan Manifest

# Install CRDs dan Operator
kubectl apply -f https://raw.githubusercontent.com/external-secrets/external-secrets/main/deploy/crds/bundle.yaml

# Install Operator
kubectl apply -f https://raw.githubusercontent.com/external-secrets/external-secrets/main/deploy/external-secrets.yaml

Verifikasi Instalasi

# Check pods
kubectl get pods -n external-secrets-system

# Output:
# NAME                                                READY   STATUS    RESTARTS   AGE
# external-secrets-7d9f7f8c8-xxxxx                   1/1     Running   0          1m
# external-secrets-cert-controller-5c6f7d8c8-xxxxx   1/1     Running   0          1m
# external-secrets-webhook-5c6f7d8c8-xxxxx          1/1     Running   0          1m

# Check CRDs
kubectl get crd | grep external-secrets

# Expected output:
# externalsecrets.external-secrets.io
# secretstores.external-secrets.io
# clustersecretstores.external-secrets.io

Custom Helm Configuration

values.yaml

# Replica count untuk HA
replicaCount: 2

# Resource limits
resources:
  requests:
    cpu: 10m
    memory: 64Mi
  limits:
    cpu: 100m
    memory: 128Mi

# Webhook configuration
webhook:
  replicaCount: 2
  resources:
    requests:
      cpu: 10m
      memory: 32Mi

# Cert controller configuration
certController:
  replicaCount: 2
  resources:
    requests:
      cpu: 10m
      memory: 32Mi

# Enable Prometheus metrics
serviceMonitor:
  enabled: true
  interval: 30s

# Security context
securityContext:
  runAsNonRoot: true
  runAsUser: 1000
  fsGroup: 1000

# Pod disruption budget
podDisruptionBudget:
  enabled: true
  minAvailable: 1

# Log level
logLevel: info  # debug, info, warn, error

Install dengan custom values:

helm install external-secrets \
  external-secrets/external-secrets \
  --namespace external-secrets-system \
  --create-namespace \
  -f values.yaml

Secret Providers

1. HashiCorp Vault

Setup Vault SecretStore

apiVersion: v1
kind: Secret
metadata:
  name: vault-token
  namespace: default
type: Opaque
stringData:
  token: "hvs.CAESIJ..."
---
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-backend
  namespace: default
spec:
  provider:
    vault:
      server: "https://vault.example.com"
      path: "secret"
      version: "v2"
      auth:
        tokenSecretRef:
          name: vault-token
          key: token

Vault dengan Kubernetes Auth

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-k8s-auth
  namespace: default
spec:
  provider:
    vault:
      server: "https://vault.example.com"
      path: "secret"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "myapp-role"
          serviceAccountRef:
            name: myapp-sa

2. AWS Secrets Manager

apiVersion: v1
kind: Secret
metadata:
  name: aws-credentials
  namespace: default
type: Opaque
stringData:
  access-key-id: "AKIAIOSFODNN7EXAMPLE"
  secret-access-key: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
---
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secrets-manager
  namespace: default
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        secretRef:
          accessKeyIDSecretRef:
            name: aws-credentials
            key: access-key-id
          secretAccessKeySecretRef:
            name: aws-credentials
            key: secret-access-key

AWS dengan IRSA (IAM Roles for Service Accounts)

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secrets-manager-irsa
  namespace: default
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets-sa

3. Google Secret Manager

apiVersion: v1
kind: Secret
metadata:
  name: gcp-credentials
  namespace: default
type: Opaque
stringData:
  credentials.json: |
    {
      "type": "service_account",
      "project_id": "my-project",
      "private_key_id": "...",
      "private_key": "...",
      "client_email": "...",
      "client_id": "...",
      "auth_uri": "...",
      "token_uri": "...",
      "auth_provider_x509_cert_url": "...",
      "client_x509_cert_url": "..."
    }
---
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: gcp-secret-manager
  namespace: default
spec:
  provider:
    gcpsm:
      projectID: "my-project"
      auth:
        secretRef:
          secretAccessKeySecretRef:
            name: gcp-credentials
            key: credentials.json

4. Azure Key Vault

apiVersion: v1
kind: Secret
metadata:
  name: azure-credentials
  namespace: default
type: Opaque
stringData:
  client-id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
  client-secret: "your-client-secret"
---
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: azure-keyvault
  namespace: default
spec:
  provider:
    azurekv:
      vaultUrl: "https://my-vault.vault.azure.net"
      tenantId: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
      authSecretRef:
        clientId:
          name: azure-credentials
          key: client-id
        clientSecret:
          name: azure-credentials
          key: client-secret

5. Doppler

apiVersion: v1
kind: Secret
metadata:
  name: doppler-token
  namespace: default
type: Opaque
stringData:
  token: "dp.st.xxxx"
---
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: doppler
  namespace: default
spec:
  provider:
    doppler:
      auth:
        secretRef:
          dopplerToken:
            name: doppler-token
            key: token

ClusterSecretStore vs SecretStore

SecretStore (Namespaced)

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: my-store
  namespace: app-namespace  # Only usable in this namespace
spec:
  provider:
    vault:
      server: "https://vault.example.com"
      path: "secret"
      # ...

ClusterSecretStore (Cluster-wide)

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: vault-backend  # Available in all namespaces
spec:
  provider:
    vault:
      server: "https://vault.example.com"
      path: "secret"
      version: "v2"
      auth:
        tokenSecretRef:
          name: vault-token
          namespace: external-secrets-system  # Must specify namespace
          key: token

ExternalSecret Resources

1. Basic ExternalSecret

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: myapp-secrets
  namespace: default
spec:
  # Refresh interval
  refreshInterval: 1h
  
  # SecretStore reference
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  
  # Target Kubernetes Secret
  target:
    name: myapp-secrets
    creationPolicy: Owner
  
  # Data to fetch
  data:
  - secretKey: database-password
    remoteRef:
      key: myapp/database
      property: password
  
  - secretKey: api-key
    remoteRef:
      key: myapp/api
      property: key

2. Multiple Keys dari Single Secret

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-secrets
  namespace: default
spec:
  refreshInterval: 30m
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: database-credentials
  data:
  - secretKey: DB_HOST
    remoteRef:
      key: database/prod
      property: host
  
  - secretKey: DB_PORT
    remoteRef:
      key: database/prod
      property: port
  
  - secretKey: DB_USER
    remoteRef:
      key: database/prod
      property: username
  
  - secretKey: DB_PASSWORD
    remoteRef:
      key: database/prod
      property: password

3. DataFrom - Import All Keys

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: all-secrets
  namespace: default
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: all-app-secrets
  dataFrom:
  - extract:
      key: myapp/config  # Import semua key dari path ini

4. Template Secrets

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: templated-secret
  namespace: default
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: app-config
    template:
      engineVersion: v2
      data:
        config.yaml: |
          database:
            host: {{ .db_host }}
            port: {{ .db_port }}
            username: {{ .db_user }}
            password: {{ .db_password }}
          api:
            key: {{ .api_key }}
            url: https://api.example.com
  data:
  - secretKey: db_host
    remoteRef:
      key: database/prod
      property: host
  - secretKey: db_port
    remoteRef:
      key: database/prod
      property: port
  - secretKey: db_user
    remoteRef:
      key: database/prod
      property: username
  - secretKey: db_password
    remoteRef:
      key: database/prod
      property: password
  - secretKey: api_key
    remoteRef:
      key: api/keys
      property: production

5. Conversion - Decode Base64

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: decoded-secret
  namespace: default
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: decoded-credentials
  data:
  - secretKey: tls.crt
    remoteRef:
      key: certificates/prod
      property: cert
      decodingStrategy: Base64  # Decode from base64
  
  - secretKey: tls.key
    remoteRef:
      key: certificates/prod
      property: key
      decodingStrategy: Base64

Contoh Lengkap End-to-End

Example: Web Application dengan Database

1. Setup SecretStore (Vault):

apiVersion: v1
kind: Secret
metadata:
  name: vault-token
  namespace: production
type: Opaque
stringData:
  token: "hvs.CAESIJ..."
---
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-production
  namespace: production
spec:
  provider:
    vault:
      server: "https://vault.example.com"
      path: "secret"
      version: "v2"
      auth:
        tokenSecretRef:
          name: vault-token
          key: token

2. Create ExternalSecret:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: webapp-secrets
  namespace: production
spec:
  refreshInterval: 15m
  secretStoreRef:
    name: vault-production
    kind: SecretStore
  target:
    name: webapp-credentials
    creationPolicy: Owner
  data:
  # Database credentials
  - secretKey: DB_HOST
    remoteRef:
      key: production/database
      property: host
  - secretKey: DB_PORT
    remoteRef:
      key: production/database
      property: port
  - secretKey: DB_NAME
    remoteRef:
      key: production/database
      property: name
  - secretKey: DB_USER
    remoteRef:
      key: production/database
      property: username
  - secretKey: DB_PASSWORD
    remoteRef:
      key: production/database
      property: password
  
  # API Keys
  - secretKey: STRIPE_API_KEY
    remoteRef:
      key: production/api-keys
      property: stripe
  - secretKey: SENDGRID_API_KEY
    remoteRef:
      key: production/api-keys
      property: sendgrid

3. Use in Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: webapp
  namespace: production
spec:
  replicas: 3
  selector:
    matchLabels:
      app: webapp
  template:
    metadata:
      labels:
        app: webapp
    spec:
      containers:
      - name: webapp
        image: myapp/webapp:2.0.0
        envFrom:
        - secretRef:
            name: webapp-credentials  # Secret created by ExternalSecret
        ports:
        - containerPort: 8080

4. Verify:

# Check ExternalSecret status
kubectl get externalsecret webapp-secrets -n production

# Check if Kubernetes Secret created
kubectl get secret webapp-credentials -n production

# Describe ExternalSecret
kubectl describe externalsecret webapp-secrets -n production

Monitoring dan Troubleshooting

Check Status

# List all ExternalSecrets
kubectl get externalsecrets --all-namespaces

# Check status
kubectl get externalsecret myapp-secrets -n default -o yaml

# Expected conditions:
# conditions:
# - type: Ready
#   status: "True"

Common Issues

Issue 1: ExternalSecret Not Syncing

# Check ExternalSecret events
kubectl describe externalsecret myapp-secrets -n default

# Check operator logs
kubectl logs -n external-secrets-system -l app.kubernetes.io/name=external-secrets

# Common causes:
# - Invalid SecretStore configuration
# - Wrong credentials
# - Network connectivity issues
# - Secret not found in backend

Issue 2: Authentication Failed

# For Vault
vault login <token>
vault kv get secret/myapp/config

# For AWS
aws secretsmanager get-secret-value --secret-id myapp/config

# Verify credentials in SecretStore Secret
kubectl get secret vault-token -n default -o yaml

Issue 3: Secret Not Found

# Verify path di backend
# For Vault:
vault kv list secret/myapp

# Check remoteRef path
kubectl get externalsecret myapp-secrets -n default -o yaml | grep remoteRef -A 3

Debug Logs

# Enable debug logging
kubectl set env deployment/external-secrets \
  -n external-secrets-system \
  LOG_LEVEL=debug

# View logs
kubectl logs -n external-secrets-system \
  -l app.kubernetes.io/name=external-secrets \
  --tail=100 -f

Monitoring dengan Prometheus

ServiceMonitor

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: external-secrets
  namespace: external-secrets-system
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: external-secrets
  endpoints:
  - port: metrics
    interval: 30s

Key Metrics

# ExternalSecret sync status
externalsecret_sync_calls_total

# ExternalSecret sync errors
externalsecret_sync_calls_error

# SecretStore connection status
externalsecret_status_condition{condition="Ready"}

# Sync duration
externalsecret_sync_duration_seconds

Grafana Dashboard

Import dashboard: 16170 (External Secrets Operator)

Best Practices

1. Use ClusterSecretStore untuk Shared Providers

✅ Recommended:

kind: ClusterSecretStore  # Reusable across all namespaces

2. Set Appropriate Refresh Interval

spec:
  refreshInterval: 15m  # Balance between freshness and API calls

Development: 5-15m
Production: 30m-1h
Rarely changing: 6h-24h

3. Use CreationPolicy Owner

target:
  creationPolicy: Owner  # ESO manages lifecycle

4. Implement RBAC

apiVersion: v1
kind: ServiceAccount
metadata:
  name: myapp-sa
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: externalsecret-reader
  namespace: default
rules:
- apiGroups: ["external-secrets.io"]
  resources: ["externalsecrets"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: myapp-externalsecret
  namespace: default
subjects:
- kind: ServiceAccount
  name: myapp-sa
roleRef:
  kind: Role
  name: externalsecret-reader
  apiGroup: rbac.authorization.k8s.io

5. Monitor Secret Expiration

# Add alerts
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: externalsecret-alerts
  namespace: external-secrets-system
spec:
  groups:
  - name: external-secrets
    interval: 30s
    rules:
    - alert: ExternalSecretSyncFailed
      expr: externalsecret_sync_calls_error > 0
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "ExternalSecret {{ $labels.name }} sync failed"

6. Use Secret Rotation

# In Vault, enable auto-rotation
spec:
  provider:
    vault:
      server: "https://vault.example.com"
      path: "database"
      version: "v2"
      auth:
        kubernetes:
          role: "myapp"

7. Namespace Isolation

# Use namespaced SecretStore untuk sensitive apps
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: production-secrets
  namespace: production  # Isolated to production namespace
spec:
  # ...

Advanced Usage

Generator - Webhook untuk Random Secrets

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: generated-password
spec:
  target:
    name: random-password
  dataFrom:
  - sourceRef:
      generatorRef:
        apiVersion: generators.external-secrets.io/v1alpha1
        kind: Password
        name: my-password
---
apiVersion: generators.external-secrets.io/v1alpha1
kind: Password
metadata:
  name: my-password
spec:
  length: 32
  digits: 10
  symbols: 10
  symbolCharacters: "-_$@"
  noUpper: false
  allowRepeat: true

PushSecret - Sync TO External Provider

apiVersion: external-secrets.io/v1alpha1
kind: PushSecret
metadata:
  name: push-to-vault
spec:
  refreshInterval: 10m
  secretStoreRefs:
  - name: vault-backend
    kind: SecretStore
  selector:
    secret:
      name: local-secret  # Kubernetes Secret to push
  data:
  - match:
      secretKey: password
    remoteRef:
      remoteKey: myapp/credentials
      property: password

Migration dari Kubernetes Secrets

Step-by-Step Migration

1. Backup existing secrets:

kubectl get secret myapp-secrets -n default -o yaml > backup.yaml

2. Upload ke external provider (contoh Vault):

vault kv put secret/myapp/config \
  DB_PASSWORD="old-password" \
  API_KEY="old-key"

3. Create ExternalSecret:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: myapp-secrets
  namespace: default
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
  target:
    name: myapp-secrets  # Same name as old secret
  dataFrom:
  - extract:
      key: myapp/config

4. Delete old secret:

kubectl delete secret myapp-secrets -n default

5. Verify:

kubectl get externalsecret myapp-secrets -n default
kubectl get secret myapp-secrets -n default

Upgrade

# Update Helm repo
helm repo update

# Check current version
helm list -n external-secrets-system

# Upgrade
helm upgrade external-secrets \
  external-secrets/external-secrets \
  --namespace external-secrets-system \
  --reuse-values

Uninstall

# Delete all ExternalSecrets
kubectl delete externalsecrets --all --all-namespaces

# Delete all SecretStores
kubectl delete secretstores --all --all-namespaces
kubectl delete clustersecretstores --all

# Uninstall Helm release
helm uninstall external-secrets -n external-secrets-system

# Delete CRDs (optional)
kubectl delete crd externalsecrets.external-secrets.io
kubectl delete crd secretstores.external-secrets.io
kubectl delete crd clustersecretstores.external-secrets.io

# Delete namespace
kubectl delete namespace external-secrets-system

Kesimpulan

External Secrets Operator adalah solusi modern untuk secret management di Kubernetes, memungkinkan Anda untuk:

Centralized secret management
Automatic synchronization
Support multiple secret backends
GitOps-friendly (no secrets in Git)

Kapan Menggunakan?

✅ Gunakan jika:

Menggunakan external secret management (Vault, AWS, etc.)
Butuh centralized secret management
Multi-cluster deployments
Compliance requirements

Pengenalan​

Masalah yang Dipecahkan​

Arsitektur​

Instalasi​

Prasyarat​

Metode 1: Helm (Recommended)​

Metode 2: Kubectl dengan Manifest​

Verifikasi Instalasi​

Custom Helm Configuration​

values.yaml​

Secret Providers​

1. HashiCorp Vault​

Setup Vault SecretStore​

Vault dengan Kubernetes Auth​

2. AWS Secrets Manager​

AWS dengan IRSA (IAM Roles for Service Accounts)​

3. Google Secret Manager​

4. Azure Key Vault​

5. Doppler​

ClusterSecretStore vs SecretStore​

SecretStore (Namespaced)​

ClusterSecretStore (Cluster-wide)​

ExternalSecret Resources​

1. Basic ExternalSecret​

2. Multiple Keys dari Single Secret​

3. DataFrom - Import All Keys​

4. Template Secrets​

5. Conversion - Decode Base64​

Contoh Lengkap End-to-End​

Example: Web Application dengan Database​

Monitoring dan Troubleshooting​

Check Status​

Common Issues​

Issue 1: ExternalSecret Not Syncing​

Issue 2: Authentication Failed​

Issue 3: Secret Not Found​

Debug Logs​

Monitoring dengan Prometheus​

ServiceMonitor​

Key Metrics​

Grafana Dashboard​

Best Practices​

1. Use ClusterSecretStore untuk Shared Providers​

2. Set Appropriate Refresh Interval​

3. Use CreationPolicy Owner​

4. Implement RBAC​

5. Monitor Secret Expiration​

6. Use Secret Rotation​

7. Namespace Isolation​

Advanced Usage​

Generator - Webhook untuk Random Secrets​

PushSecret - Sync TO External Provider​

Migration dari Kubernetes Secrets​

Step-by-Step Migration​

Upgrade​

Uninstall​

Kesimpulan​

Kapan Menggunakan?​

Resources​