Using the debug bundle

Since 2.6.0, the FOSSA chart has supported generating debug bundles. Debug bundles are ZIP files containing information about a FOSSA application and the cluster it's installed in.

Generating a debug bundle

Cluster operators can generate debug bundles in a few easy steps:

Enable the debug bundle in the Helm values.
Upgrade the Helm release with the new values.
Wait for the debug bundle pod to collect information.
Extract the resulting debug bundle from the pod, and email it to FOSSA support.

Alternatively, the debug bundle can be generated by scaling the replica count for RELEASE_NAME-fossa-core-debug-bundle to 1 and following steps 3 - 4 above.

Enabling the debug bundle

To enable the debug bundle, set deployment.debugBundle.enabled: true in the fossa/fossa-core chart:

deployment:
  debugBundle:
    enabled: true

Using kubectl to scale replicas:

$ kubectl scale RELEASE_NAME-fossa-core-debug-bundle --replicas=1

If you have cluster role creation permissions, you can also set deployment.debugBundle.createClusterRole: true. This will allow FOSSA to collect debug information about your cluster nodes, which helps us diagnose issues around hardware provisioning and performance:

deployment:
  debugBundle:
    enabled: true
    createClusterRole: true

Applying the new Helm values

Apply your new Helm values the same way you would apply any set of updated values. For most users, you will likely want to use helm upgrade:

$ helm upgrade RELEASE_NAME fossa/fossa-core --values NEW_VALUES_FILE

Once the upgrade has completed, you should see a new deployment named RELEASE_NAME-fossa-core-debug-bundle with a single pod:

$ kubectl get deployments
NAME                                         READY   UP-TO-DATE   AVAILABLE   AGE
# ...
RELEASE_NAME-fossa-core-api                  15/15   15           15          50d
RELEASE_NAME-fossa-core-debug-bundle         1/1     1            1           24h
RELEASE_NAME-fossa-core-workers-autobuilds   3/3     3            3           50d
# ...

$ kubectl get pods
NAME                                                    READY   STATUS      RESTARTS      AGE
# ...
RELEASE_NAME-fossa-core-api-d4bb59577-z2dmf             1/1     Running     0             71m
RELEASE_NAME-fossa-core-api-d4bb59577-zllkv             1/1     Running     0             71m
RELEASE_NAME-fossa-core-debug-bundle-85bdd985d9-c8hm4   1/1     Running     1 (10m ago)   71m
# ...

Generating the debug bundle

Once the debug bundle pod is running, check its logs:

$ kubectl logs -f -l app.kubernetes.io/component=debug-bundle
[INFO] [2023-03-24T23:08:00.519Z] Starting collection of data.
	environment="staging"
[INFO] [2023-03-24T23:08:00.525Z] All files will be saved to: /tmp/debug_167969928
	environment="staging"
[INFO] [2023-03-24T23:08:00.526Z] Validating connections...
	environment="staging"
[INFO] [2023-03-24T23:08:00.526Z] Initiating database connection & loading models...
	environment="staging"
[INFO] [2023-03-24T23:08:04.277Z] Connecting to Redis at staging-core-redis:6379...
	environment="staging"
[INFO] [2023-03-24T23:08:04.333Z] Successfully connected to Redis.
	environment="staging"
[INFO] [2023-03-24T23:08:04.337Z] Successfully connected to Faktory
	environment="staging"
[INFO] [2023-03-24T23:08:04.376Z] Database connected & models loaded.
	environment="staging"
[INFO] [2023-03-24T23:08:04.377Z] Collecting app configuration data:
	environment="staging"
[INFO] [2023-03-24T23:08:04.383Z] - ✅ saved FOSSAConfigs data to: /tmp/debug_167969928/app-config/FOSSAConfigs.csv
	environment="staging"
[INFO] [2023-03-24T23:08:05.001Z] - ✅ saved FeatureFlags data to: /tmp/debug_167969928/app-config/FeatureFlags.csv
	environment="staging"
[INFO] [2023-03-24T23:08:05.006Z] - ✅ saved SequelizeMeta data to: /tmp/debug_167969928/app-config/SequelizeMeta.csv
	environment="staging"
[INFO] [2023-03-24T23:08:05.006Z] Collecting build data:
	environment="staging"
[INFO] [2023-03-24T23:08:05.819Z] - ✅ saved Builds data to: /tmp/debug_167969928/builds/Builds.csv
	environment="staging"
[INFO] [2023-03-24T23:08:05.826Z] - ✅ saved Tasks data to: /tmp/debug_167969928/builds/Tasks.csv
	environment="staging"
[INFO] [2023-03-24T23:08:05.835Z] - ✅ saved Revisions data to: /tmp/debug_167969928/builds/Revisions.csv
	environment="staging"
[INFO] [2023-03-24T23:08:06.292Z] - ✅ saved DependencyLocks data to: /tmp/debug_167969928/builds/DependencyLocks.csv
	environment="staging"
[INFO] [2023-03-24T23:08:06.397Z] - ✅ saved pods data to /tmp/debug_167969928/kube-data/pods.json
	environment="staging"
[INFO] [2023-03-24T23:08:06.421Z] - ✅ saved pod (mailcatcher-deployment-64d9c94985-gczq2) container (mailcatcher) logs to /tmp/debug_167969928/kube-data/logs/mailcatcher-deployment-64d9c94985-gczq2_mailcatcher.json
	environment="staging"
# ...
# ...
# ...
[INFO] [2023-03-24T23:08:11.343Z] - ✅ saved latest helm release (sh.helm.release.v1.staging.v3925) manifest to /tmp/debug_167969928/kube-data/manifest_sh.helm.release.v1.staging.v3925.yaml
	environment="staging"
[INFO] [2023-03-24T23:08:11.346Z] Collection of data is complete, this process will exit in an hour and data will be removed.
	environment="staging"
[INFO] [2023-03-24T23:08:11.346Z] Use kubectl to copy the data: kubectl cp staging-fossa-core-debug-bundle-85bdd985d9-c8hm4:/tmp/debug_167969928 LOCAL_DEBUG_BUNDLE_LOCATION
	environment="staging"
[WARN] [2023-03-24T23:08:11.346Z] Information collected may contain secrets, please parse output and remove secrets.
	environment="staging"

⚠️ If you see errors about being unable to retrieve information about nodes and node leases, this is likely because you did not set deployment.debugBundle.createClusterRole: true. This means the debug bundle will not contain diagnostic information about your cluster's underlying compute nodes.

The debug bundle is still useful even without this information.

Once you see Collection of data is complete, the debug bundle is ready.

Collecting the generated debug bundle

Once the debug bundle is complete, you can use kubectl cp to copy it to your local machine. The debug bundle's log output will provide a pre-formatted command for you similar to this:

$ kubectl cp RELEASE_NAME-fossa-core-debug-bundle-85bdd985d9-c8hm4:/tmp/debug_167969928 LOCAL_DEBUG_BUNDLE_LOCATION

Make sure to replace LOCAL_DEBUG_BUNDLE_LOCATION with your own path.

This folder's contents are the debug bundle.

Note that the debug bundle is an exact reproduction of your cluster's state, and may contain any secrets that are present in Kubernetes manifests. You may want to redact some of this information. In particular:

Passwords in the saved Helm values (values_sh.*.json).
Passwords in the rendered Kubernetes manifests (manifest_sh.*.yaml).
Passwords in the ConfigMap dump (config_maps.json).

⚠️ Depending on your cluster's configuration, this may not be an exhaustive list of all possible secrets.

Once you're ready, ZIP up the folder and attach it to an email or Zendesk ticket and send it to FOSSA Support.