AWS IoT Core: A Compromised Device Perspective

TL;DR

I recently spent some time exploring the potential capabilities that an evil IoT device might have within an AWS IoT Core environment. This culminated in the thoughts I've transcribed in this blog post, as well as a command-line tool to help with enumeration and data harvesting during security assessments of products that interact with AWS IoT Core.

 Disclaimer

The tools, tactics, techniques, and procedures described in this blog post were developed for educational purposes with the goal of securing product implementations. I do not endorse the use of these resources against real-world production environments without prior permission from the owner of the target instance(s). Additionally, the unauthorized use of these TTPs against real-world implementations may trigger detection/alert mechanisms in IoT Device Defender (or equivalent services), resulting in client key disablement/revocation and/or further repercussions (legal or otherwise).

Scenario

Hypothetical: during a security assessment of an IoT device, you've compromised the client certificate and private key that the device uses for mTLS connections to an AWS IoT Core back-end. Now what?

I recently spent some time exploring this exact scenario, and what follows is the culmination of my thoughts on the topic.

Overview - AWS IoT Core

This section consists of an abstract, overly-simplified summary of a typical AWS IoT Core setup from a client device perspective. If you're not interested or already know the basics, you can skip to the next section.

Client IoT devices are registered with the AWS IoT back-end service. Clients receive a key pair (a private key and a certificate containing a public key) that they use for mTLS authentication with the back-end (note that custom/layered authentication mechanisms such as username/password might also be required).

The devices primarily communicate with the server via MQTT* (UDP port 8883); clients "subscribe" to "topics" by opening a connection to the server and waiting for incoming messages that were "published" to those topics (for the web-oriented people: topics are basically analogous to URL paths). Clients can also publish messages - to induce on-demand messages from the server (similar to an HTTP GET request), or to modify some stateful data (like an HTTP PUT/POST/DELETE), or for other purposes. There is a list of AWS "reserved" topics; some subset of these are used in virtually all IoT Core implementations. 

* Many of the MQTT-related concepts described in this post (e.g., topic injection) can be applied to MQTT services in general, not just AWS IoT Core.

Within IoT Core is the concept of a device "shadow" - this is some persistent data that might include the device configuration, status information (e.g., battery level), or any other arbitrary data. Shadows exist in two states at once - "reported" and "desired." The "reported" state is essentially the "last-known" state as reported by the client device, which might not always be connected to the internet. The "desired" state is the server's authoritative state; it might contain pending configuration changes or other data that was pushed to the server by a web dashboard or mobile app. If the "reported" state doesn't match the "desired" state, a "delta" message is published to the client device the each time it connects (until the client's reported state is synchronized).

There is also an HTTP REST API (TCP port 8443) that clients can interact with for shadow-related operations (again, with mTLS authentication).

Connection Requirements

Before interacting with the back-end, you need three key pieces of information*:

• Service endpoint (i.e., the hostname of the AWS server)
• Client (IoT device) mTLS certificate
• Client mTLS private key

* There are other ways to authenticate, but in this blog post we're only focusing on the standard mTLS authentication mechanism.

AWS IoT Core endpoints are formatted like so:

${random_id}.iot.${region}.amazonaws.com
${random_id}-ats.iot.${region}.amazonaws.com

Another field you probably need to know is the unique ID of your device - or in AWS IoT terms, the thingName. This value is sometimes provided as a "client ID" when connecting to the back-end, but the client ID can vary (important note: two clients with the same client ID can't be connected simultaneously).

In our hypothetical scenario, we assume that we've already compromised the client keys somehow (e.g., via an arbitrary file-read vulnerability). The service endpoint and thingName can often be compromised in the same way as the keys, but if not, you can try to obtain them in other ways:

• Service endpoint: Dump traffic on a network interface between the device and the internet; you should see DNS queries for a domain name with one of the formats shown above.
• thingName: Try inspecting the client certificate metadata, or try using typical device identifiers like the serial number or MAC address (with and without octet separators - I'd try colons and hyphens).

In general, you can also try to reverse-engineer firmware artifacts to find the information you need.

Enumeration (Offline)

If you have access to device firmware (e.g., executable user-space binaries), static analysis can be helpful for enumeration. Reverse-engineering of client implementations can potentially disclose the following (and more):

• Implementation-specific MQTT topics
• Partial authorization policy reconstruction (e.g., if you see code that publishes to a topic, you can infer that the policy allows the iot:Publish action for your device to that topic)
• Credential provider endpoint
• Aliases for IAM roles that can be assumed by the client device
• Layered authentication mechanisms (e.g., username/password)
• Shadow names

Enumeration (Online)

The techniques in this section are described with the assumption that IoT Device Defender is not present/active in the AWS instance. In a real-world scenario, there are additional considerations that one must account for before attempting anything described here (see below - Evading AWS IoT Device Defender). Additionally, AWS IoT policies are "deny-by-default;" therefore, the following techniques are most useful when some policy rule has introduced an authorization weakness.

Data harvesting with MQTT topic filter wildcards - IoT core allows MQTT subscriptions to topics with wildcard segments:

• # - subtree wildcard. This can be used after a topic prefix to subscribe to "all [topic] strings at and below its level in the topic hierarchy."
• + - single-level wildcard. Used to subscribe to "any [topic] string in the level that contains the character."

Topic filter wildcards are extremely powerful, both for data harvesting and enumeration of implementation-specific topics. For example, what would happen if you subscribed to an MQTT topic string consisting of a single character, "#"? Assuming the policy allows it, you would automatically start receiving all MQTT messages that your device is authorized to receive (intentionally or otherwise). For this reason, I call "#" the "global topic filter."

The global topic filter is a great way to quickly discover authorization flaws pertaining to the iot:Receive action. Consider an implementation with extremely poor authorization policies - by subscribing to the global topic filter, you could potentially receive all MQTT messages intended for any client device that uses the back-end. This level of information disclosure could be devastating for the resource owners, but invaluable to an attacker.

To try this technique with mosquitto, usage is as follows:

mosquitto_sub -h $AWS_HOST -p 8883 --cert $CLIENT_CERT --key $CLIENT_PRIVKEY -t '#'

To try this technique with my AWS IoT Recon tool, usage is as follows:

java -jar aws-iot-recon.jar -a mqtt-dump -H $AWS_HOST -c $CLIENT_CERT -k $CLIENT_PRIVKEY

If you don't specify any MQTT topics with -T, the tool automatically subscribes to the global topic filter.

Using the -T flag with AWS IoT Recon (equivalent to the -t flag in mosquitto), you can also try searching for more specific authorization flaws. For example, the following command attempts to listen for cross-device shadow updates by using a single-segment wildcard in place of thingName:

java -jar aws-iot-recon.jar -a mqtt-dump -H $AWS_HOST -c $CLIENT_CERT -k $CLIENT_PRIVKEY -T '$aws/things/+/shadow/update/accepted'

Reading device shadows - If you've managed to obtain thingName values for one or more devices, you can try fetch the device shadow(s). Shadows can potentially contain sensitive information such as passwords, Wi-Fi connection parameters, and more. Fetch device shadows with AWS IoT Recon like so:

java -jar aws-iot-recon.jar -a get-device-shadow -H $AWS_HOST -c $CLIENT_CERT -k $CLIENT_PRIVKEY -t $THING_NAME

If the device uses named shadows, you can specify a name with the -s flag:

java -jar aws-iot-recon.jar -a get-device-shadow -H $AWS_HOST -c $CLIENT_CERT -k $CLIENT_PRIVKEY -t $THING_NAME -s shadowName

If you're not sure whether named shadows are in use, try listing them:

java -jar aws-iot-recon.jar -a list-named-shadows -H $AWS_HOST -c $CLIENT_CERT -k $CLIENT_PRIVKEY -t $THING_NAME

Discovering IAM Roles - Using intuition, reverse engineering, or common role word lists, you can use my tool to discover IAM role aliases that your client device is allowed to assume. AWS IoT Recon can do this with a command such as the following:

java -jar aws-iot-recon.jar -a iam-credentials -H $AWS_CRED_PROVIDER_HOST -c $CLIENT_CERT -k $CLIENT_PRIVKEY -t $THING_NAME -R $ROLE_ALIAS

Note that the endpoint/hostname for the credentials provider service is different from the base AWS IoT endpoint; it should look something like this:

${random_id}.credentials.iot.${region}.amazonaws.com

Unfortunately, the random_id value will be different from the value in the base IoT endpoint, so you'll have to find it using similar techniques to the ones discussed above.

If you successfully obtain credentials for a role, you can refer to other resources for guidance on how to proceed with traditional AWS enumeration and exploitation.

MQTT Topic Injection

If the authorization policy relies on iot:ClientId (or any other attacker-controllable data), you might be able to perform a class of authorization bypass attack which I've dubbed "topic injection." The AWS documentation has a good example of this type of attack. Let's say there's a policy such as the following:

{
    "Effect": "Allow",
    "Action": ["iot:Subscribe"],
    "Resource": [
      "arn:aws:iot:us-east-1:123456789012:topicfilter/my/${iot:ClientId}/topic"
    ]
}

You can potentially perform a topic injection using a client ID of "+" to subscribe to topics meant for other devices. Here's how you'd do this with the AWS IoT Recon tool:

java -jar aws-iot-recon.jar -a mqtt-dump -H $AWS_HOST -c $CLIENT_CERT -k $CLIENT_PRIVKEY -C '+' -T 'my/+/topic'

An equivalent mosquitto command is as follows:

mosquitto_sub -h $AWS_HOST -p 8883 --cert $CLIENT_CERT --key $CLIENT_PRIVKEY -i '+' -t 'my/+/topic'

If the injection is in the last segment of the topic, you can also use a client ID of "#" for the same effect (this could be useful if an iot:Connect rule prevents the use of "+" in client IDs).

To find potential topic injections, look for any topic segment that seems to correspond to a device identifier (e.g., the thingName).

Data Exfiltration

If you find that you can publish and subscribe to arbitrary/unused MQTT topics, you can abuse the AWS IoT data plane for data exfiltration. MQTT messages are protected with mTLS and can contain arbitrary information; additionally, if an IoT Core device exists on a network, we can probably assume that data plane communications are allowed through any firewalls and out to the wider internet. The makes the data plane an excellent data exfiltration medium if you can get away with it.

Use AWS IoT Recon to test for data exfiltration capabilities like so:

java -jar aws-iot-recon.jar -a mqtt-data-exfil -H $AWS_HOST -c $CLIENT_CERT -k $CLIENT_PRIVKEY

You can specify data exfiltration topics with -T; if no topics are provided, the tool creates a unique topic consisting of the current Unix epoch timestamp.

MQTT Scripting

If you have a complex use-case that requires multiple MQTT operations, I've developed a simple script syntax for automatically executing sequential MQTT actions with the AWS IoT Recon tool. The scripting "language" supports four types of instructions (and comments):

PUB	<topic>	<payload>
SUB	<topic>
UNSUB	<topic>
SLEEP	<milliseconds>
# Comment

Instruction arguments are separated/delimited by a tab character (\t), and empty/whitespace-only lines are ignored.

PUB payloads can be provided in-line, but this will cause problems if the payload contains certain characters (e.g., newline, tab, or non-printable byte values). For this reason, I've included support for hex-encoding and reading payloads from files:

# Hex-encoded payload ("My vision is augmented")
PUB	<topic>	hex://4d7920766973696f6e206973206175676d656e746564

# Payload stored in the file /tmp/mqtt_payload.txt
PUB	<topic>	file:///tmp/mqtt_payload.txt

The following command demonstrates how to use the MQTT scripting feature of the recon tool:

java -jar aws-iot-recon.jar -a mqtt-script -f $MQTT_SCRIPT_FILE -H $AWS_HOST -c $CLIENT_CERT -k $CLIENT_PRIVKEY

Attacking Other Targets

Again, I have to interject with a disclaimer. Please do not attack real-world targets without prior authorization from the resource owner(s). Anyway...

The ability to interact with IoT Core facilitates the potential to propagate data (potentially containing exploit payloads) to other clients within the IoT Core ecosystem. Targets might include any of the following:

Other IoT devices: If you find that you can publish to MQTT topics meant for other devices, you've likely discovered a very powerful attack surface. You can use MQTT messages to modify device shadows/configurations, start/stop/monitor "jobs," and more. Additionally, you could potentially use the data plane to deliver implementation-specific exploit payloads (e.g., command injection payloads via shadow fields).

Web interfaces: It's not uncommon for IoT devices to be managed/monitored by some form of web-based dashboard. Consider how data might propagate from your MQTT messages (e.g., topic strings or device shadow updates) to a web UI - are there any fields that might facilitate web exploitation techniques such as blind XSS?

Mobile applications: Like web interfaces, it's not uncommon to manage IoT devices from a mobile app. Again, consider how you might leverage device status/configuration payloads to induce unwanted behavior in mobile clients. It might help to download any relevant apps from the app store(s) for reverse engineering.

AWS IoT Device Defender

IoT Device Defender is a powerful service; for an overview of its capabilities, I recommend checking out Introduction to IoT Device Defender on AWS Skill Builder, which consists of a single ~20-minute video lecture. Long story short, defensive mechanisms can be triggered via scheduled audits and/or device behavior anomaly detection. Heuristics can be based on a multitude of device behaviors and metrics, including:

• Client ID collisions
• Multiple client devices using the same client certificate
• MQTT message send/receive frequency
• Message size
• Authorization errors
• Open ports (TCP/UDP)*
• Number of TCP connections*
• Source IP addresses
• Destination IP addresses*
• Outbound traffic volume*

* These are device-side metrics, which are self-reported by clients, and therefore only apply to post-exploitation activities on-device.

Defensive mechanisms can be implemented using AWS Lambda functions, and therefore may be comprised of any number of potential responsive actions. If I had to speculate, I'd guess that the vast majority of automated defense mechanisms don't do anything more than trigger some alerts and (in the worst case) automatically disable the offending client certificate(s).

Detecting AWS IoT Device Defender

To take an informed approach to attacking IoT Core, it's important to determine whether Device Defender is being used at all.

Device-side: If you have access to the device firmware and/or filesystem, try to determine if the AWS IoT Device Client is present. The default build script outputs an executable binary called aws-iot-device-client; try searching the filesystem for this. If that doesn't work, try using grep to search for strings that are unique to the Device Client binary. After a quick look at the source code, I believe this search command is sufficient to determine whether the Device Client binary exists in a directory:

grep -rl -e DeviceDefenderFeature -e 'AWS IOT DEVICE CLIENT FATAL ERROR'

If any file is found using the above command, it's probably the IoT Device Client.

If you've found the Device Client binary, your next task is to locate the Device Client configuration. According to the documentation, the configuration can be "passed to the AWS IoT Device Client through either the command line, a JSON configuration file, or both." The specific configuration option we're looking for is device-defender, which looks like this:

"device-defender": {
    "enabled": true,
    "interval-in-seconds": 300
}

If the enabled field is true, we can reasonably infer that the implementation is actively using Device Defender to some extent.

If the device-defender option is missing, the default value for device-defender.enabled is false (see line 218 of Config.h).

If enabled is false, or if the AWS IoT Device Client isn't on the system, this doesn't necessarily mean that Device Defender is inactive. It does indicate that device-side metrics won't be reported upstream, but there could still be active detection rules based on cloud-side metrics.

There might also be some potential to abuse MQTT reserved topics for device-side metrics to determine whether device-side metrics are gathered/analyzed (e.g., different error responses), but I haven't found any way to do so.

Cloud-side: Unfortunately, I haven't found a way to disclose cloud-side Device Defender configurations/policies/rules from a client device standpoint (if anyone figures out a way to do this, please let me know so I can update this post).

I imagine that, even if a technique was discovered to disclose cloud-side configuration data, it would be patched to prevent this type of information-gathering (which is primarily useful for nefarious purposes).

Evading AWS IoT Device Defender

Realistically, there's no way to guarantee that you won't trip a defensive mechanism if you stray from the standard behavior of your compromised device. This is especially true in a "black box" test scenario, where you don't have access to the underlying AWS policies and configurations. Even so, there are some precautions you can take to minimize your chance of being detected:

• Avoid client ID collisions by disconnecting the (real) compromised device before performing tests using its client ID. If you do accidentally trigger a client ID collision, you should receive an error with code 5134 (AWS_ERROR_MQTT_UNEXPECTED_HANGUP), but this error could also indicate a number of other things.
• Avoid client ID collisions by using a client ID that a real device would never have (e.g., a random string). However, this might trigger other defensive rules (e.g., certificate sharing or authorization failures).
• Avoid MQTT message size/rate heuristics by monitoring and emulating the traffic coming out of the real device. If you've achieved privileged execution on the device, you can exactly calculate safe upper-bound numbers via run-time instrumentation/debugging. If not, you can still make some educated guesses through a combination of reverse-engineering and analysis of out-bound TLS traffic to the AWS host.
• Avoid authorization errors by mapping out allowed action/topic pairs through static reverse engineering of client implementations (e.g., as explained above, if you see code that publishes to a topic, you can infer that the policy allows the iot:Publish action for your device to that topic).
• Avoid source IP address block-listing by using a VPN for any actions with a high detection potential (of course, many people would argue that you should be doing this either way).
• Prevent device-side metrics-based alerts by disabling metrics reporting in the AWS IoT Device Client configuration, or by killing/instrumenting the Device Client process on the compromised device (note that the latter is likely to disrupt other functionality).
• Acquire/compromise multiple client devices/keys as a contingency plan. Depending on how many keys you can acquire, it would probably be worth burning a few pairs to check for basic authorization policy gaps (e.g., cross-device shadow operations).
• Wait - (This one's a long shot) IoT Device Defender is a paid add-on service with a 1-month free trial. If your target is brand-new, and you've got time to burn, consider waiting until the product has been deployed for more than a month before trying anything risky.

---

Resources

Source code for the tool I described in this blog post can be found on my GitHub. Below is a list of resources that I found helpful during this adventure:

• AWS Skill Builder - Deep Dive into AWS IoT Authentication and Authorization 
• AWS Skill Builder - Introduction to IoT Device Defender 
• AWS IoT Core Developer Guide
• AWS IoT Device SDK for Java v2 API documentation
• AWS IoT Device SDK for Java v2 sample implementations
• AWS Security Blog - Eliminate the Need for Hardcoded AWS Credentials in Devices