How to work with diffs in CI/CD

To help you protect your software development lifecycle and deliver trusted software to end-users, the Spectra Assure platform integrates with popular CI/CD services to scan your build artifacts for security issues.

Whether you're setting up a new pipeline or working with an already established build system, Spectra Assure is a suitable solution for the security testing stage of your CI/CD process.

After a build artifact is produced in the build stage, the Spectra Assure integration picks it up, scans it, and produces an analysis report that can be stored for later use. The final scan result (pass or fail) can be used to control whether the artifact is safe to deploy.

Benefits of using diffs in CI/CD

In addition to the basic security scanning workflow, the Spectra Assure platform supports the diff workflow. A diff is a comparison of two versions of the same software that highlights the changes between them.

Setting up automated diff workflows in CI/CD provides a real-time feedback loop that empowers developers to recognize and address security issues between different software versions. By automating security checks in the CI/CD pipeline, your organization can ensure that your software is continuously resilient against evolving threats.

To enable the diff workflow in CI/CD, your build system must be configured to preserve build artifacts of multiple software versions. You can then instruct the Spectra Assure integration to compare the build artifact of a new software version to a previous version. This comparison is also known as the diff scan.

The results of artifact comparison are included in the SAFE analysis report produced by Spectra Assure after the scan. The diff information in the report is useful for:

• detecting potential tampering and preventing malicious code from being introduced into new software versions,
• detecting issues early in the development process and identifying potentially unexpected differences between software versions,
• improving compliance with security standards and regulations.

About this guide

This best practices guide contains advice for efficiently implementing diff scans with the Spectra Assure CLI in CI/CD. It assumes you're using the rl-scanner Docker image.

The guide is intended for DevSecOps practitioners who want to start using Spectra Assure CLI integrations, or who want to upgrade their existing workflows to leverage the diff functionality.

Diff workflow requirements

To successfully set up and and use the diff workflow with CLI integrations, you're going to need:

• A valid rl-secure license with site key. This is a site-wide type of license that can be deployed on multiple machines. When you register for the CLI, ReversingLabs Support will send you the license as a single file, and the site key as a plaintext string. The site key needs to be used for each ephemeral run, and your DevOps teams will need access to it. To use the license with CI/CD integrations, convert the contents of your license file into a Base64-encoded string and save it for later use.

• A persistent package store. A package store is a special directory where rl-secure keeps your analyzed build artifacts and their scan results. When created, a package store is automatically organized into a predefined structure where every analyzed artifact is registered as a package version and assigned to a project and package. To enable the diff workflow, the package store must be created on a shared storage that is external to the Docker container used by the integrations.

• Package URLs of artifacts (software versions) you want to compare, in the format [pkg:type/]<project></package><@version>. The package URL is a unique identifier that registers your scanned artifacts in the package store. To compare artifacts, you will specify the package URL of a previous version to diff against when scanning a new version. Both versions must be in the same project and package.

Workflow visualization

The following diagram provides a general overview of a typical CI/CD setup and shows how the CLI integration fits into it.

In this workflow, the process of creating a diff has two main steps:

• scan only - Scan a package version (V1)
• diff scan - Scan another package version (V2) and diff it against a previous one (V1)

It is expected that these steps are performed independently, in separate runs (only one package is scanned at a time).

Diff support in CI/CD services

ReversingLabs provides a number of officially supported integrations with popular CI/CD services that you can use to scan your build artifacts with the CLI.

The integrations are implemented to accommodate the features of each CI/CD service. Because of this, some services may require a specific approach for the diff workflow. The following table lists the limitations you should be aware of before integrating with these services.

Service	Diffing limitations
Jenkins	MultiBranch pipeline or a custom solution is required. Diffs can't be done for PRs or Tags.
GitHub Actions	Package store configuration is supported only when using self-hosted runners.
GitLab CI	Package store configuration is supported only when using self-hosted runners.
Azure DevOps	Diffing requires a self-hosted agent. Azure CLI is used on the runner to retrieve the latest successful commit.

Secrets management

You must obtain a license with site key to use any of the CLI integrations. When you have the license and site key, manage them as secrets according to your organization's best practices using the features of your CI/CD tools. Depending on your use-case, you can configure them as organization-level or project-level secrets.

Environment variable names for secrets are fixed and consistent across all CLI integrations. In your configuration files, the license and site key should always map to the variables RLSECURE_ENCODED_LICENSE and RLSECURE_SITE_KEY.

Docker considerations

All CLI integrations rely on the rl-scanner Docker image, so your CI/CD tools must be configured to use Docker. You need to be able to start a Docker container in the test stage of your pipeline, because Spectra Assure scans happen inside the container.

Docker volumes

The rl-scanner Docker image uses volumes for working with input (artifacts to scan) and output files (analysis reports). Your configuration must allow Docker to access the locations that need to be mounted. Input and output paths for Docker volumes are user-configurable for every CLI integration.

Pipeline design

The frequency and scope of your security scans must be aligned with your Spectra Assure license and monthly analysis capacity. It's recommended to start scanning nightly builds, then merge requests, then every commit. Keep in mind that scans will not run if your analysis capacity is exceeded.

Scanning individual commits

To set up scanning for every commit, the build pipeline needs to check out changes from your version control system. Commit hashes can then be used to identify artifact versions and compare them in a diff scan. For this approach to work, your CI/CD tools need to be able to retrieve the branch history, where they can get the information on previous successful commits.

If we use Jenkins as an example, we can set up the configuration so that a Jenkins job is started when a pull request is merged on GitHub. The GitHub merge event starts the Jenkins job with a push context, which allows saving commit hashes to environment variables:

• GIT_COMMIT - the most recent pushed commit
• GIT_PREVIOUS_SUCCESSFUL_COMMIT - last successful commit before the most recent one
• GIT_PREVIOUS_COMMIT - last failed commit before the most recent one

The diff scan will run if the GIT_PREVIOUS_SUCCESSFUL_COMMIT is set and not equal to GIT_COMMIT.

You will also need to get the BRANCH_NAME environment variable from Jenkins. In a merge event, the BRANCH_NAME is equal to the target branch.

With information from Jenkins saved to environment variables, you can construct package URLs for artifacts you want to diff in the following way:

PURL="${Project}/${Package}@${Version}"

where Project uses your GitHub organization or username stored in the GIT_URL variable in the format https://<your-github-server>/<organisation-or-user>/<repository name>:

Project=$( echo "${GIT_URL}" | awk -F/ '{ print $4 }' )

and Package uses the repository name and the branch name:

Package=$( echo "${GIT_URL}" | awk -F/ '{ print $5 }' )
Package=$( basename "${Package}" ".git" )"-${BRANCH_NAME}"

and Version is set to GIT_COMMIT.

You can then modify the Docker command for the CLI integration to use the custom package URL when scanning artifacts produced by individual commits. In the following example, we're setting the package URL with --purl="${Purl}".

docker run --rm \
    -u $(id -u):$(id -g) \
    -e RLSECURE_ENCODED_LICENSE="${YOUR_ENCODED_LICENSE}" \
    -e RLSECURE_SITE_KEY="${YOUR_SITE_KEY}" \
    -v "${BUILD_PATH}:/packages:ro" \
    -v "./report:/report" \
    -v "${RL_STORE}:/rlstore" \
    reversinglabs/rl-scanner \
        rl-scan \
            --replace \
            --rl-store=/rlstore \
            --purl="${Purl}" \
            --package-path=/packages/"${MY_ARTIFACT_TO_SCAN}" \
            --report-path=/report \
            --report-format=all \
            --diff-with=${GIT_PREVIOUS_SUCCESSFUL_COMMIT}

Including --diff-with enables the CLI integration to automatically check for a previous artifact version to compare against. If there is no previous successful commit available, CLI integrations will not perform a diff scan, but they will scan the most recently pushed commit (GIT_COMMIT).

Reusable parameters

If there are any reusable parameters in your configuration (for example, all your pipelines use the same package store), you can store those reusable values in global environment variables:

BUILD_PATH="./dist"
RL_STORE="/mount/nfs/rl-store"

Artifact management

Artifacts must be in one of the supported file formats so that the CLI can scan them. By default, the CLI supports specifying only one artifact parameter per each run. In other words, just one artifact can be scanned or diffed at a time.

Analysis reports

Analysis reports from Spectra Assure scans are artifacts of the test stage. They are always saved to the package store, but you can also specify a custom report directory to store them elsewhere. The report directory must be mounted to the Docker container with write permissions, and must be created empty before starting the container.

The SAFE report (rl-html) is the only report format that can display the diff information. When saving analysis reports, you must specify the report format as either rl-html or all with the --report-format parameter to include the results of your diff scan.

Package store

When using CLI integrations, a package store is automatically initialized by the Docker container. To support diff scans, you need to make the store permanent outside of the container to allow saving the analysis metadata for multiple package versions.

Shared storage

If you're using only a single agent/runner, creating the package store in an external shared location is not required. In that case, the package store can be created locally, directly on the agent/runner.

When using multiple runners, the package store must be shared between all runners. To do this, you can set up an external NFS or CIFS storage location to host the package store. The storage size depends on your organization's needs and use-cases. It's recommended to start small and set it to a few GB for testing purposes, then gradually increase the storage size as you get more familiar with the CLI.

Maintenance

When using a package store, it's recommended to clean it up regularly. This can be done in an automated way with the rl-prune command included in the rl-scanner Docker image. It's recommended to clean up everything older than 14 days.