Workshop: Malicious Kubernetes

Course Objectives:

·      Review basics of Linux Containers and images

·      Primitives of how malicious containers are created

·      Get Familiar with Kubernetes

·      Secure and Break into (or out of) Containerized Workflows

·      Primitives of Analyzing Containerized Workflows for malicious content

·      Understand some details of how Kubernetes Security works

·      Understanding how to hide and maintain persistence in containers

·      Conduct log level analysis of kubernetes clusters to find malicious behavior

Part 1: Malicious Containers creation and how to find them (1 hr 20m total)

Module 1: Welcome to the world of containers. - 30m

Cover briefly how to create your own docker containers, how to create your own malicious container, etc.

Module 1 Labs: Exploring Containers

·      Quick review of Docker and containers and how they work

·      Linux process isolation primitives (namespace, cgroups, process capabilities)

·      Privileged containers and host volume mounts

·      Example of a malicious container image with a ‘C2’ callback/exfil that avoids signature detections

Module 2: Docker IR - 50 mins (inc. outbrief)

We’ll use this offensive knowledge to then cover the fundamentals of container Incident Response and analysis by analyzing the above, including a lab where users will download an image and deploy their knowledge to discover a series of progressively more savvily hidden malwares. The importance of provenance, SBOMs, image signing options, etc. will also be covered.

A preliminary version of our CTF-style lab that we will begin during the course, and will likely also involve a take home element (some of the techniques are rather hard to find) is here: https://hub.docker.com/r/digitalshokunin/webserver

Objectives:

·      Basics of what makes a docker images

·      Determining provenance of an image and if an image is what it claims to be

·      How images are put together and how they can be explored manually

·      Basic docker commands to assist with image forensics

Module 2 Labs: Exploring Containers

·      Exploring Container History

·      Inspecting an image for suspicious content

·      Forensics on offline docker filesystems

·      Safely extracting content from an image & dockerfiles

·      Using other Docker images to perform forensic and IR tasks

==== Container Practical ==== incl. A short break.

Utilizing skills from above, pull the following docker image and and complete the challenge:

“Scenario: Hosts that run containerized services are seeing unusual activity & high resource usage, but no one is able to pin down what processes are consuming CPU. You see in the logs the issues started around the time a new container image was pulled down for a web server, several instances of it are running across the hosts with issues. You decide to investigate the image.”

Objectives:

·      Figure out what / if anything was modified

·      Get hash of modified/malicious files

·      Bonus: What does the payload do?

·      Bonus bonus: find all the modifications

Part 2: Kubernetes fundamentals for red teams and incident responders

Module 1: Basics of Kubernetes - 45m

Objectives:

·      What is Kubernetes (K8S) and why is it useful

·      Learn the components of Kubernetes

·      Learn what namespaces in Kubernetes are (not the same as in Docker/Linux)

·      Learn what Pods, Deployments, Daemon Sets, and Services in K8s are

·      Learn basics of Kubernetes Networking

·      Learn about security in Kubernetes through namespace separation, role-based access control (RBAC), network policies, authentication through tokens and certificates, alternative authentication services, etc

·      Building on their container knowledge, enter the world of container orchestration and clustering; how does Kubernetes work?

·      Additionally, we've got practical cluster creation/management labs, on usage of kubectl, pod creation, etc

Module 1 Labs: K8s Fundamentals

·      Interact with a kubernetes cluster using the client.

·      Create Namespaces

·      Run Pods

Module 2: Offensive Kubernetes - 1h

Objectives:

·      Learn how Kubernetes RBAC misconfiguration can lead to privilege escalation attacks

·      Learn about K8S Private Key Infrastructure and forging certificates

·      Learn about Admissions Controllers and debugging for additional access

·      Building on their knowledge from previous sections of the workshop, what are the security pitfalls that can leave the clusters and pods/containers exposed to attack?

·      Participants will use special lab scenarios configured in their cluster to explore pitfalls and common attacks using RBAC priv esc, key dumping, and malicious pod attacks.

Module 2 Labs: Offensive K8s

·      RBAC exercises

·      Privesc via secrets enumeration

·      Golden Tickets & Abusing CA’s.

·      Cloud Metadata Attack

Module 3: Kubernetes Incident Response - 30 min.

It is all too common that people think if their kubernetes environment is compromised, they can leverage ephemerality to blow it all away and start over, but how does that help you understand the attacker? Their goals/motives/objectives, and what they obtained? What are teams consistently getting wrong when it comes to k8s IR?

(https://www.youtube.com/watch?v=MyXROAqO7YI, https://github.com/wagoodman/dive)

In this section, we’ll explore the differences in IR vs regular systems, logging breakdowns, tactics for isolation and quarantining, and container forensics, plus what to do when things aren’t going perfectly. We’ll leverage a laboratory environment alongside tooling such as Grafana/Prometheus/Loki to help people understand the key points of differentiation. This lab, like prior labs, will remain online after the workshop for people to play with.

Module 3 Labs: K8S IR

·      Investigating with eBPF / Grafana/Loki

·      Using tooling to look for notable alert opportunities in prior exercises.

·      Detecting Enumeration

·      Detecting Priv esc attempts

·      Detecting unusual activity Remote shells/or unusual outgoing connections

About the Instructors:

Adrian Wood, aka threlfall, discovered a love for hacking from cracking and modding video games and from the encouragement of online friends. He has worked as a red team consultant for WHITEHACK, a company he founded, and later as a lead engineer for an offensive research team at a US bank, where he was very interested in appsec, container security, CI/CD security, and also founded their bug bounty program. He currently works for Dropbox, working on application security. In his free time, he enjoys playing saxophone, working on vintage cars, and fly-fishing.

David Mitchell, aka digish0, started his hacking career as a script kiddie running 7th Sphere in mIRC in high school. Later falling in with some Linux/RedHat nerds at a local 2600 group at college while studying CS, etc. He got into Linux, started an IT career, later rediscovering his hacking script kiddie roots when a local hacker space opened up and shared members with a lockpicking group that worked in infosec as penetration testers, etc where he discovered he could get paid to do the things he liked doing in high school/college. He now works professionally as a red team member and cyber security researcher at a large financial institution. The rest of the time he spends being a dad/husband, trying not to get injured in Muay Thai/BJJ or mountain biking, and listening to either very expensive or very cheap vinyl.