Red Teaming at NCCDC 2018
Intro
Welcome back for some more pro CCDC hacking tips! This was my 5th year red teaming at The National CCDC competition; This is a truly amazing event and one of the highlights of my year. I think all teams are getting better over time, from the black teams competition design, to the red team’s coordination, and even the blue teams’ ability to defend and respond to incidents. The winning teams of the 2018 National CCDC event, in order of 1st, 2nd, and 3rd, were University of Virginia, University of Central Florida, and Dakota State University. This year the competition was held in Orlando Florida at Disney World, which provided amazing accommodations for both the event and the trip as a whole. Another new twist that the black team introduced this year, the red team had to set a flag on victim systems, so that they could prove their access and their persistence was verified by an agent on the blue team endpoints. The black team is also building more reliable networks, instituting better monitoring, and increasing their ability to run a fair and balanced competition. The rest of this post is going to cover blue advice as well as red team infrastructure advancements.
Blue Team Advice
Some of my best advice to blue teams starts with playing a strong network game, then moving to host based controls. Granted you should focus on the basics first, such as changing default passwords and patching vulnerable services, but beyond that I suggest focusing on your network before moving to host based controls. If possible, reshape your network and create choke points where you can apply ingress and egress firewall rules. Limit your external exposure to your scored services and only access critical hosts from trusted systems. When you move to host based controls start with host based firewall rules and remove tools that you no longer need. Look at common persistence mechanisms to determine if a machine has been compromised and if so don’t use that to access more systems until you’ve cleaned that host. If you can’t figure out what the red team is doing on the host then look at where they are coming from on the network.
I saw some pretty great hardening techniques this year, I saw lots of host based firewalls being applied, removing unnecessary services or tools from boxes, and even the use of advanced network detection appliances. If it’s not a scored service, take it down or firewall it off. I can tell you that this dramatically slows the red team down. I also found a lot of teams auditing their users and changing passwords throughout the competition. Below is a screenshot where a blue team is finding a malicious user on their systems.
I was really impressed with all of the blue teams this year. My team did a fantastic job finding implants and removing them. These incident response skills is where I feel the competition is being driven. It’s no longer a hardening and revert competition, as we’ve started getting our persistence into the backups. Blue teams are being forced to hunt for the red team, reverse their tools, and understand the initial point of compromise. That's what makes this competition so valuable, is that you get to experience an active breach where you fight an attacker for your systems. In the screenshot below you can see a blue team identifying a malicious process. The next step would be to see how these malicious applications are persisted in the scheduled tasks and further purge the red teams foothold.
Using specific workstations to admin dedicated servers can be an effective strategy for quickly scoping incidents. In that same camp, It’s good advice not to use important boxes that the red team is likely to target, such as the domain controller, for responding to injects or accessing other machines. Another tip in that vain, look for simple signs of compromise as a canary to know when not to use a machine as a pivot! Further, if you can't tell how the red team is accessing your internal infrastructure, try switching your methods of access as a way to force the red team to adapt. Similarly, you can change passwords multiple times throughout the competition to force the red team to keep gathering credentials, as opposed to just changing default credentials at the beginning of the competition. In the following screenshot you can see a great number of implants running on a victim machine. Even if you don’t know how these implants are being persisted, it’s very clear that the box is actively compromised, so you should stop using that box to access other things until you are certain it is clean.
Red Team Advancements
We, as the red team, brought some incredible tools this year. The National CCDC Red Team is notorious for bringing custom implants and techniques, and this year was no exception with many team members iterating on past creations. I’m going to focus on the stuff my team brought, as it was mostly collaborative infrastructure used by most other CCDC Red Team members, however other members of the collective red team bring some seriously powerful custom implants and automation with them.
To start, Alex overhauled Traphouse in a big way this year, which you can read about in a series of posts on his blog. Traphouse now supports multi-tenancy, such that we can standardize red team scoring and reporting across multiple regions with a single instance. Also, Traphouse will now keep track of all individual red teamer's points and progress throughout multiple competitions and from year to year. Another new feature I like are the graphs on the homepage of Traphouse, this enables the red team to quickly assess which teams are doing well and which blue teams could use some extra attention. These charts were helpful at the Western Regional but became far more helpful at Nationals with the visualization of the implants api and the team service checks. The implants API page was a massive help in our ability to track active compromise, which lets the red team see in real time which integrated team servers still have shells available on blue team target systems. Alex also added support for a bunch of his tools as unique pages in Traphouse, directly integrating things like Borg commands, Grid reservations, the ability to access scan data. There's even a page for uploading and downloading binaries from the Grid CDN, which we leveraged heavily for delivering implants, like the genesis binaries. If you want to learn more about how Alex designed and built these awesome collaboration tools, check out his blog posts!
Lucas on our team fully automated our scanning and provided this automated scanning data to all teams this year. This was done through a rewrite of old tools, such as Mr. Wizard and Borg, to create the BorgWizard! This has been a long time goal of ours to fully automate competition scanning and it was really helpful that Lucas was able to provide this service for us in a fast and reliable way. Just like the Borg workers of the past, this scanning solution will hop IP addresses across a broad range after each scan, enabling us to avoid IP blocking and get an unobstructed view of external attack surface. These tools are designed using docker and templates such that the commands can be edited on the fly with the ability to change scanning techniques, scanning cadence, and target hosts.
This year, I focused on collecting and bundling of our team-based payloads from the various red team members, and compiling them into a single dropper binary. We used gscript to generate these “genesis” binaries, such that we could evenly apply custom kits to each team at the competition. In this way we could also deconflict techniques, file names, and persistence mechanisms in code and have a way to manage arbitrary implants across the entire red team. We ended up wrapping over 7 unique second stage payloads for windows system and 5 unique second stage payloads on Linux. Each of these payloads has various features and unique techniques that red team members would leverage for persistence and to regain access to their target machines. This allows us to leverage all of the cool custom implants that get developed just for CCDC, without having to stage each of these payloads or drop / run multiple binaries on every endpoint. Further, by integrating our team servers into the Traphouse implant api, then wrapping them in the genesis binaries we gained unparalleled insight into which teams had the genesis binary run on them. I was really happy to see every red team at nationals use the genesis binaries to drop our collective persistence. This means the dream of gscript came to light and was a success, at least in my eyes.
Tons of other red teamers brought custom kits and scripted payloads. The National CCDC Red Team is an incredible team where everyone really contributes a lot of both well known and advanced techniques and tools. Several of our classic implants were iterated on this year, such as cobalt strike implants, custom PAM modules, scriptjunkie's custom kits, Donation improvements, and both Linux and Windows Jerricho enhancements. We also had team members such as Jackson5 scripting out automated attacks, such that we could quickly spray well known exploits across the team ranges, harvest credentials, and drop our genesis binaries on victim systems. All of this automation in terms of scanning, exploitation, and post exploitation allowed us to compromise and persist in a wide swath of systems across all teams, in well under the five minute mark.
Dave’s Red Team Debrief was amazing as always. He has great points in his debrief, from many of our general TTPS, to listing a bunch of custom tools, and ultimately providing sound incident response advice. If you aren't looking over Dave's red team debrief each year as a team about to go to the National CCDC competition you are doing yourself a disservice and putting yourself at a disadvantage to teams that have seen or read these debriefs. That's all folks! I hope to see even more improvement next year, and if you feel like getting even more blue team practice in the off season, I urge you to sign up for ProsVJoes at BSidesLV! Below is a photo of less than half this year’s NCCDC Red Team, but at least I remembered to snap a photo of this many at once:
