Since 2.6.0, the FOSSA chart has supported generating debug bundles. Debug bundles are ZIP files containing information about a FOSSA application and the cluster it's installed in.
Cluster operators can generate debug bundles in a few easy steps:
- Enable the debug bundle in the Helm values.
- Upgrade the Helm release with the new values.
- Wait for the debug bundle pod to collect information.
- Extract the resulting debug bundle from the pod, and email it to FOSSA support.
Alternatively, the debug bundle can be generated by scaling the replica count for
RELEASE_NAME-fossa-core-debug-bundle to 1 and following steps 3 - 4 above.
To enable the debug bundle, set
deployment.debugBundle.enabled: true in the
deployment: debugBundle: enabled: true
Using kubectl to scale replicas:
$ kubectl scale RELEASE_NAME-fossa-core-debug-bundle --replicas=1
If you have cluster role creation permissions, you can also set
deployment.debugBundle.createClusterRole: true. This will allow FOSSA to collect debug information about your cluster nodes, which helps us diagnose issues around hardware provisioning and performance:
deployment: debugBundle: enabled: true createClusterRole: true
Apply your new Helm values the same way you would apply any set of updated values. For most users, you will likely want to use
$ helm upgrade RELEASE_NAME fossa/fossa-core --values NEW_VALUES_FILE
Once the upgrade has completed, you should see a new deployment named
RELEASE_NAME-fossa-core-debug-bundle with a single pod:
$ kubectl get deployments NAME READY UP-TO-DATE AVAILABLE AGE # ... RELEASE_NAME-fossa-core-api 15/15 15 15 50d RELEASE_NAME-fossa-core-debug-bundle 1/1 1 1 24h RELEASE_NAME-fossa-core-workers-autobuilds 3/3 3 3 50d # ... $ kubectl get pods NAME READY STATUS RESTARTS AGE # ... RELEASE_NAME-fossa-core-api-d4bb59577-z2dmf 1/1 Running 0 71m RELEASE_NAME-fossa-core-api-d4bb59577-zllkv 1/1 Running 0 71m RELEASE_NAME-fossa-core-debug-bundle-85bdd985d9-c8hm4 1/1 Running 1 (10m ago) 71m # ...
Once the debug bundle pod is running, check its logs:
$ kubectl logs -f -l app.kubernetes.io/component=debug-bundle [INFO] [2023-03-24T23:08:00.519Z] Starting collection of data. environment="staging" [INFO] [2023-03-24T23:08:00.525Z] All files will be saved to: /tmp/debug_167969928 environment="staging" [INFO] [2023-03-24T23:08:00.526Z] Validating connections... environment="staging" [INFO] [2023-03-24T23:08:00.526Z] Initiating database connection & loading models... environment="staging" [INFO] [2023-03-24T23:08:04.277Z] Connecting to Redis at staging-core-redis:6379... environment="staging" [INFO] [2023-03-24T23:08:04.333Z] Successfully connected to Redis. environment="staging" [INFO] [2023-03-24T23:08:04.337Z] Successfully connected to Faktory environment="staging" [INFO] [2023-03-24T23:08:04.376Z] Database connected & models loaded. environment="staging" [INFO] [2023-03-24T23:08:04.377Z] Collecting app configuration data: environment="staging" [INFO] [2023-03-24T23:08:04.383Z] - ✅ saved FOSSAConfigs data to: /tmp/debug_167969928/app-config/FOSSAConfigs.csv environment="staging" [INFO] [2023-03-24T23:08:05.001Z] - ✅ saved FeatureFlags data to: /tmp/debug_167969928/app-config/FeatureFlags.csv environment="staging" [INFO] [2023-03-24T23:08:05.006Z] - ✅ saved SequelizeMeta data to: /tmp/debug_167969928/app-config/SequelizeMeta.csv environment="staging" [INFO] [2023-03-24T23:08:05.006Z] Collecting build data: environment="staging" [INFO] [2023-03-24T23:08:05.819Z] - ✅ saved Builds data to: /tmp/debug_167969928/builds/Builds.csv environment="staging" [INFO] [2023-03-24T23:08:05.826Z] - ✅ saved Tasks data to: /tmp/debug_167969928/builds/Tasks.csv environment="staging" [INFO] [2023-03-24T23:08:05.835Z] - ✅ saved Revisions data to: /tmp/debug_167969928/builds/Revisions.csv environment="staging" [INFO] [2023-03-24T23:08:06.292Z] - ✅ saved DependencyLocks data to: /tmp/debug_167969928/builds/DependencyLocks.csv environment="staging" [INFO] [2023-03-24T23:08:06.397Z] - ✅ saved pods data to /tmp/debug_167969928/kube-data/pods.json environment="staging" [INFO] [2023-03-24T23:08:06.421Z] - ✅ saved pod (mailcatcher-deployment-64d9c94985-gczq2) container (mailcatcher) logs to /tmp/debug_167969928/kube-data/logs/mailcatcher-deployment-64d9c94985-gczq2_mailcatcher.json environment="staging" # ... # ... # ... [INFO] [2023-03-24T23:08:11.343Z] - ✅ saved latest helm release (sh.helm.release.v1.staging.v3925) manifest to /tmp/debug_167969928/kube-data/manifest_sh.helm.release.v1.staging.v3925.yaml environment="staging" [INFO] [2023-03-24T23:08:11.346Z] Collection of data is complete, this process will exit in an hour and data will be removed. environment="staging" [INFO] [2023-03-24T23:08:11.346Z] Use kubectl to copy the data: kubectl cp staging-fossa-core-debug-bundle-85bdd985d9-c8hm4:/tmp/debug_167969928 LOCAL_DEBUG_BUNDLE_LOCATION environment="staging" [WARN] [2023-03-24T23:08:11.346Z] Information collected may contain secrets, please parse output and remove secrets. environment="staging"
If you see errors about being unable to retrieve information about nodes and node leases, this is likely because you did not set
deployment.debugBundle.createClusterRole: true. This means the debug bundle will not contain diagnostic information about your cluster's underlying compute nodes.
The debug bundle is still useful even without this information.
Once you see
Collection of data is complete, the debug bundle is ready.
Once the debug bundle is complete, you can use
kubectl cp to copy it to your local machine. The debug bundle's log output will provide a pre-formatted command for you similar to this:
$ kubectl cp RELEASE_NAME-fossa-core-debug-bundle-85bdd985d9-c8hm4:/tmp/debug_167969928 LOCAL_DEBUG_BUNDLE_LOCATION
Make sure to replace
LOCAL_DEBUG_BUNDLE_LOCATION with your own path.
This folder's contents are the debug bundle.
Note that the debug bundle is an exact reproduction of your cluster's state, and may contain any secrets that are present in Kubernetes manifests. You may want to redact some of this information. In particular:
- Passwords in the saved Helm values (
- Passwords in the rendered Kubernetes manifests (
- Passwords in the
Depending on your cluster's configuration, this may not be an exhaustive list of all possible secrets.
Once you're ready, ZIP up the folder and attach it to an email or Zendesk ticket and send it to FOSSA Support.
Updated 5 months ago