Migrating to SeaweedFS

Overview

This guide moves your on-premises object storage backend from MinIO to SeaweedFS, FOSSA's S3-compatible storage option. FOSSA uses two SeaweedFS instances (one for Core analysis artifacts and one for Hubble compliance data) and the migration copies your existing MinIO data into them before cutting the deployment over.

The migration has two phases: prepare the SeaweedFS deployments, then execute the cutover during a maintenance window.

Preparation

Create SeaweedFS deployments for Core and Hubble sized to your storage needs. FOSSA requires two instances:

• core-seaweedfs: stores analysis artifacts for FOSSA Core
• hubble-seaweedfs: stores compliance data for FOSSA Hubble

Both values files set the S3 admin key pair under s3.credentials.admin. The migration Job and fossa-core authenticate with that pair, so it must match rclone.conf and the cutover values. Without it, the chart generates random keys that exist only in the release's S3 secret. The example values are seaweedfs and seaweedfs123, so replace them with your own.

The template below documents installation, upgrade, and the sizing guide:

# =============================================================================# SeaweedFS Storage Values Template for FOSSA Deployment# =============================================================================## FOSSA requires two SeaweedFS instances for S3-compatible object storage:#   1. core-seaweedfs   - Stores analysis artifacts for FOSSA Core#   2. hubble-seaweedfs - Stores compliance data for FOSSA Hubble## Adjust storage sizes based on your expected workload. See the "Sizing Guide"# at the bottom of this file for recommendations.## Prerequisites:#   - A Kubernetes cluster with a default StorageClass configured#   - Helm 3.x## Installation:#   helm repo add fossa https://charts.fossa.com#   helm repo update##   helm install core-seaweedfs fossa/seaweedfs \#     --namespace <namespace> \#     --version 4.17.0 \#     -f core-seaweedfs-values.yaml##   helm install hubble-seaweedfs fossa/seaweedfs \#     --namespace <namespace> \#     --version 4.17.0 \#     -f hubble-seaweedfs-values.yaml## Upgrade:#   helm upgrade core-seaweedfs fossa/seaweedfs \#     --namespace <namespace> \#     --version 4.17.0 \#     -f core-seaweedfs-values.yaml##   helm upgrade hubble-seaweedfs fossa/seaweedfs \#     --namespace <namespace> \#     --version 4.17.0 \#     -f hubble-seaweedfs-values.yaml## =============================================================================# Sizing Guide# =============================================================================## The main values to adjust are volume.dataDirs[].size fields. These control# how much raw data each SeaweedFS instance can store.## Recommended volume sizes by deployment scale:##   Deployment Size         | Core volume.size | Hubble volume.size | Notes#   ------------------------|------------------|--------------------|-------------------#   Small  (<100 projects)  |  50Gi            |  25Gi              | Dev/eval environments#   Medium (<500 projects)  | 100Gi            |  50Gi              | Standard production#   Large  (<2000 projects) | 250Gi            | 100Gi              | Large organizations#   XLarge (2000+ projects) | 500Gi+           | 250Gi+             | Enterprise scale## Other parameters:##   master.data.size   - Volume metadata. 1Gi is sufficient for all but the#                        largest deployments (10k+ volumes).##   filer.data.size    - File index. 1Gi handles millions of objects. Increase#                        to 5-10Gi only if you expect tens of millions of files.##   maxVolumes         - Upper limit on the number of logical volumes SeaweedFS#                        can create. The defaults (10000 for core, 6000 for#                        hubble) are generous and rarely need adjustment.##   storageClass       - Uncomment and set if you need a specific Kubernetes#                        StorageClass (e.g., for SSD-backed storage, specific#                        provisioners, or cloud-specific volume types). If left#                        commented out, the cluster default StorageClass is used.## Storage can be expanded after initial deployment by increasing the PVC size,# provided your StorageClass supports volume expansion# (allowVolumeExpansion: true).# =============================================================================

Core: core-seaweedfs-values.yaml

Inside the cluster, the Core S3 endpoint is http://core-seaweedfs-s3:8333. FOSSA also sends the Core storage endpoint to users' browsers and the CLI, so it must be reachable from outside the cluster. The s3.ingress block below exposes it, and its hostname becomes storage.endpoint at cutover.

nameOverride: core-seaweedfs global:  serviceAccountName: core-seaweedfs  imagePullSecrets: test-fossa-core-quay.io # <-- existing pull secret created by fossa-core when release name is `test`; list them with `kubectl get secrets --field-selector type=kubernetes.io/dockerconfigjson` # -- S3 API Configurations3:  enabled: true  enableAuth: true  credentials:    admin:      accessKey: seaweedfs                 # <-- ADJUST: Key pair used by rclone.conf and fossa-core storage.auth      secretKey: seaweedfs123  createBuckets:    - name: fossa.test      anonymousRead: false  # -- S3 Ingress (required for Core; end users must reach this endpoint)  ingress:    enabled: true    className: nginx                     # <-- ADJUST: Your ingress class. Leaving it empty renders an empty ingressClassName, which most controllers ignore    host: seaweedfs-core.fossa.local     # <-- ADJUST: Hostname used in storage.endpoint; needs a DNS record pointing at your ingress    annotations:                         # <-- ADJUST: ingress-nginx settings for large uploads; replace for other controllers      nginx.ingress.kubernetes.io/proxy-body-size: 1000m      nginx.ingress.kubernetes.io/proxy-request-buffering: "off"    tls: []                              # <-- TLS configuration. See your ingress controller for documentation # -- Master Server Storage (volume metadata and cluster topology; lightweight)master:  data:    type: persistentVolumeClaim    size: 1Gi                            # <-- Adjust if needed (rarely necessary)    # storageClass: ""                   # <-- Uncomment to use a specific StorageClass # -- Filer Storage (file-to-chunk mapping index)filer:  data:    type: persistentVolumeClaim    size: 1Gi                            # <-- Adjust if needed    # storageClass: ""                   # <-- Uncomment to use a specific StorageClass  logs:    type: emptyDir # -- Volume Server Storage (primary data storage)volume:  dataDirs:    - name: data1      type: persistentVolumeClaim      size: 100Gi                        # <-- ADJUST: Set based on expected data volume      maxVolumes: 10000      # storageClass: ""                 # <-- Uncomment to use a specific StorageClass

Hubble: hubble-seaweedfs-values.yaml

After installation, the Hubble S3 endpoint is available at http://hubble-seaweedfs-s3:8333. Only in-cluster services use it, so Hubble does not need an ingress.

nameOverride: hubble-seaweedfs global:  serviceAccountName: hubble-seaweedfs  imagePullSecrets: test-hubble-quay.io # <-- existing pull secret created by fossa-core when release name is `test`; list them with `kubectl get secrets --field-selector type=kubernetes.io/dockerconfigjson` # -- S3 API Configurations3:  enabled: true  enableAuth: true  credentials:    admin:      accessKey: seaweedfs                 # <-- ADJUST: Key pair used by rclone.conf and fossa-core hubble.storage.auth      secretKey: seaweedfs123  createBuckets:    - name: hubble.fossa.test      anonymousRead: false # -- Master Server Storage (volume metadata and cluster topology; lightweight)master:  data:    type: persistentVolumeClaim    size: 1Gi                            # <-- Adjust if needed (rarely necessary)    # storageClass: ""                   # <-- Uncomment to use a specific StorageClass # -- Filer Storage (file-to-chunk mapping index)filer:  data:    type: persistentVolumeClaim    size: 1Gi                            # <-- Adjust if needed    # storageClass: ""                   # <-- Uncomment to use a specific StorageClass  logs:    type: emptyDir # -- Volume Server Storage (primary data storage)volume:  dataDirs:    - name: data1      type: persistentVolumeClaim      size: 100Gi                         # <-- ADJUST: Set based on expected data volume      maxVolumes: 6000      # storageClass: ""                 # <-- Uncomment to use a specific StorageClass

Execution

1. 1

   Schedule a maintenance window

   Pick a window long enough to transfer your object data and verify the cutover.

2. 2

   Set the application to maintenance mode

   Assuming the namespace is fossa:

   ShellCopy

   kubectl config set-context --current --namespace=fossahelm upgrade -i fossa fossa/fossa-core --values fossa-core-config.yml --set global.maintenanceMode.enabled=true --version "^4.0.0"

3. 3

   Deploy SeaweedFS for Core and Hubble

   ShellCopy

   helm upgrade -i core-seaweedfs fossa/seaweedfs --values core-seaweedfs-values.yamlhelm upgrade -i hubble-seaweedfs fossa/seaweedfs --values hubble-seaweedfs-values.yaml

   Confirm the Core S3 ingress exists and answers from outside the cluster (use http if you did not configure TLS):

   ShellCopy

   kubectl get ingress ingress-core-seaweedfs-s3curl -sSf https://seaweedfs-core.fossa.local/status

4. 4

   Transfer data from MinIO to SeaweedFS

   The Job below uses rclone to copy all data from Core and Hubble MinIO into SeaweedFS. Update the rclone.conf endpoints and credentials to match your deployment. The example assumes:

   • Core: bucket fossa.test; MinIO minio / minio123 at http://fossa-core-minio:80; SeaweedFS seaweedfs / seaweedfs123 at http://core-seaweedfs-s3:8333
   • Hubble: bucket hubble.fossa.test; MinIO minio / minio123 at http://fossa-hubble-minio:80; SeaweedFS seaweedfs / seaweedfs123 at http://hubble-seaweedfs-s3:8333

   YAMLCopy

   # rclone-migration-minio-seaweedfs.yamlapiVersion: v1kind: ConfigMapmetadata:  name: rclone-migration-configdata:  rclone.conf: |    [minio-core]    type = s3    provider = Minio    access_key_id = minio    secret_access_key = minio123    endpoint = http://fossa-core-minio:80    region = minio     [minio-hubble]    type = s3    provider = Minio    access_key_id = minio    secret_access_key = minio123    endpoint = http://fossa-hubble-minio:80    region = minio     [seaweedfs-core]    type = s3    provider = Other    access_key_id = seaweedfs    secret_access_key = seaweedfs123    # In-cluster service endpoint. The Job runs inside the cluster, so keep this    # rather than the ingress hostname that fossa-core's storage.endpoint uses.    endpoint = http://core-seaweedfs-s3:8333    region = us-east-1    force_path_style = true     [seaweedfs-hubble]    type = s3    provider = Other    access_key_id = seaweedfs    secret_access_key = seaweedfs123    endpoint = http://hubble-seaweedfs-s3:8333    region = us-east-1    force_path_style = true   core-bucket: "fossa.test"  hubble-bucket: "hubble.fossa.test"  migrate.sh: |    #!/bin/sh    set -e     echo "============================================"    echo "  MinIO -> SeaweedFS Migration"    echo "  Started: $(date -u +%Y-%m-%dT%H:%M:%SZ)"    echo "============================================"     # ---------- tuning knobs (adjust for production) ----------    # TRANSFERS:  number of file transfers in parallel    # CHECKERS:   number of parallel hash-checking threads    # S3_CONCURRENCY: multipart upload concurrency per file    # S3_CHUNK:   multipart chunk size (bigger = fewer API calls for large files)    # BUFFER:     in-memory buffer per transfer    TRANSFERS="${RCLONE_TRANSFERS:-16}"    CHECKERS="${RCLONE_CHECKERS:-8}"    S3_CONCURRENCY="${RCLONE_S3_UPLOAD_CONCURRENCY:-4}"    S3_CHUNK="${RCLONE_S3_CHUNK_SIZE:-64M}"    BUFFER="${RCLONE_BUFFER_SIZE:-32M}"     COMMON_FLAGS="--transfers=$TRANSFERS \      --checkers=$CHECKERS \      --s3-upload-concurrency=$S3_CONCURRENCY \      --s3-chunk-size=$S3_CHUNK \      --buffer-size=$BUFFER \      --stats=10s \      --stats-one-line \      --log-level=INFO \      --checksum \      --retries=3 \      --retries-sleep=5s \      --low-level-retries=10"     # ---------- pre-flight checks ----------    echo ""    echo ">>> Pre-flight: listing source buckets"    echo "--- minio-core ---"    rclone lsd minio-core: 2>&1    echo "--- minio-hubble ---"    rclone lsd minio-hubble: 2>&1    echo "--- seaweedfs-core ---"    rclone lsd seaweedfs-core: 2>&1    echo "--- seaweedfs-hubble ---"    rclone lsd seaweedfs-hubble: 2>&1     echo ""    echo ">>> Source object counts"    CORE_SRC=$(rclone size minio-core:$CORE_BUCKET --json 2>/dev/null)    HUBBLE_SRC=$(rclone size minio-hubble:$HUBBLE_BUCKET --json 2>/dev/null)    echo "  core:   $CORE_SRC"    echo "  hubble: $HUBBLE_SRC"     # ---------- migration: core ----------    echo ""    echo "============================================"    echo "  [1/2] Migrating core: minio -> seaweedfs"    echo "============================================"    eval rclone sync minio-core:$CORE_BUCKET seaweedfs-core:$CORE_BUCKET $COMMON_FLAGS 2>&1    CORE_RC=$?    echo "  core migration exit code: $CORE_RC"     # ---------- migration: hubble ----------    echo ""    echo "============================================"    echo "  [2/2] Migrating hubble: minio -> seaweedfs"    echo "============================================"    eval rclone sync minio-hubble:$HUBBLE_BUCKET seaweedfs-hubble:$HUBBLE_BUCKET $COMMON_FLAGS 2>&1    HUBBLE_RC=$?    echo "  hubble migration exit code: $HUBBLE_RC"     # ---------- post-migration validation ----------    echo ""    echo "============================================"    echo "  Post-migration validation"    echo "============================================"     echo ""    echo ">>> Destination object counts"    CORE_DST=$(rclone size seaweedfs-core:$CORE_BUCKET --json 2>/dev/null)    HUBBLE_DST=$(rclone size seaweedfs-hubble:$HUBBLE_BUCKET --json 2>/dev/null)    echo "  core src:    $CORE_SRC"    echo "  core dst:    $CORE_DST"    echo "  hubble src:  $HUBBLE_SRC"    echo "  hubble dst:  $HUBBLE_DST"     echo ""    echo ">>> Checking for differences (core)..."    rclone check minio-core:$CORE_BUCKET seaweedfs-core:$CORE_BUCKET --one-way $COMMON_FLAGS 2>&1    CORE_CHECK=$?     echo ""    echo ">>> Checking for differences (hubble)..."    rclone check minio-hubble:$HUBBLE_BUCKET seaweedfs-hubble:$HUBBLE_BUCKET --one-way $COMMON_FLAGS 2>&1    HUBBLE_CHECK=$?     echo ""    echo "============================================"    echo "  Migration Summary"    echo "  Finished: $(date -u +%Y-%m-%dT%H:%M:%SZ)"    echo "============================================"    echo "  Core   sync=$CORE_RC   check=$CORE_CHECK"    echo "  Hubble sync=$HUBBLE_RC check=$HUBBLE_CHECK"     if [ "$CORE_RC" -ne 0 ] || [ "$HUBBLE_RC" -ne 0 ] || [ "$CORE_CHECK" -ne 0 ] || [ "$HUBBLE_CHECK" -ne 0 ]; then      echo "  STATUS: FAILED (see logs above)"      exit 1    fi     echo "  STATUS: SUCCESS"---apiVersion: batch/v1kind: Jobmetadata:  name: rclone-minio-to-seaweedfsspec:  backoffLimit: 2  ttlSecondsAfterFinished: 86400  template:    metadata:      labels:        app: rclone-migration    spec:      restartPolicy: OnFailure      containers:      - name: rclone        image: rclone/rclone:latest        imagePullPolicy: IfNotPresent        command: ["/bin/sh", "/scripts/migrate.sh"]        env:        - name: CORE_BUCKET          valueFrom:            configMapKeyRef:              name: rclone-migration-config              key: core-bucket        - name: HUBBLE_BUCKET          valueFrom:            configMapKeyRef:              name: rclone-migration-config              key: hubble-bucket        # --- Tune these for production (1-4TB) ---        # For a Kind test, keep conservative to avoid overwhelming the node        - name: RCLONE_TRANSFERS          value: "8"        - name: RCLONE_CHECKERS          value: "4"        - name: RCLONE_S3_UPLOAD_CONCURRENCY          value: "4"        - name: RCLONE_S3_CHUNK_SIZE          value: "16M"        - name: RCLONE_BUFFER_SIZE          value: "16M"        resources:          requests:            cpu: 500m            memory: 256Mi          limits:            cpu: "2"            memory: 1Gi        volumeMounts:        - name: rclone-config          mountPath: /config/rclone          readOnly: true        - name: scripts          mountPath: /scripts          readOnly: true      volumes:      - name: rclone-config        configMap:          name: rclone-migration-config          items:          - key: rclone.conf            path: rclone.conf      - name: scripts        configMap:          name: rclone-migration-config          items:          - key: migrate.sh            path: migrate.sh            mode: 0755

   Apply the Job:

   ShellCopy

   kubectl config set-context --current --namespace=fossakubectl apply -f rclone-migration-minio-seaweedfs.yaml

5. 5

   Follow the migration progress

   ShellCopy

   kubectl logs jobs/rclone-minio-to-seaweedfs --follow

6. 6

   Point your values file at SeaweedFS

   Update fossa-core-config.yml with the SeaweedFS endpoints, iam.region, and iam.kind. Core's storage.endpoint is the ingress hostname from s3.ingress.host, with the scheme your ingress serves, because browsers and the CLI use it directly. Hubble's endpoint stays in-cluster.

   YAMLCopy

   iam:  region: seaweedfs  kind: AccessKey storage:  forcePathStyle: true  endpoint: https://seaweedfs-core.fossa.local # <-- Ingress hostname, not the in-cluster service; use http:// if the ingress has no TLS  bucket: fossa.test  auth:    type: AccessKey    accessKey: seaweedfs    secretKey: seaweedfs123 hubble:  iam:    region: seaweedfs    kind: AccessKey   storage:    endpoint: http://hubble-seaweedfs-s3:8333    bucket: hubble.fossa.test    auth:      type: AccessKey      accessKey: seaweedfs      secretKey: seaweedfs123

   If the ingress serves a certificate from a private CA, add that CA to trustedCertificates in the fossa-core values so Core itself can reach the endpoint.

7. 7

   Cut over and disable maintenance mode

   Once the data migration completes, upgrade the deployment to use SeaweedFS and turn off maintenance mode:

   ShellCopy

   helm upgrade -i fossa fossa/fossa-core --values fossa-core-config.yml --set global.maintenanceMode.enabled=false --version "^5.0.0"

8. 8

   Verify your data

   Log in to the console and confirm your data is still available.

9. 9

   Decommission MinIO

   After a week or two of healthy operation, delete the MinIO deployments:

   ShellCopy

   helm delete fossa-core-miniohelm delete fossa-hubble-minio

   If you're certain the MinIO data is no longer needed, delete its PVCs:

   ShellCopy

   kubectl delete pvc fossa-core-miniokubectl delete pvc fossa-hubble-minio