Lessons From the Cybersecurity Trenches

Growing up in the Pacific Northwest, I was fascinated by treasure hunting. I love the idea of finding something valuable or important. I had an arsenal of tools even the most seasoned treasure hunters would envy: a metal detector, a bucket, and a plastic shovel. Yet the most valuable tool I possessed was my firm belief that I would find treasure as long as I looked hard enough.

Threat hunting is like treasure hunting in many ways. Threat hunters also have their tools of the trade.

Several years ago, I traded that plastic shovel for a data visualization tool (i.e., Kibana) and too much coffee. I still feel the excitement of the hunt for something valuable. You see, treasure hunting and threat hunting both stimulate the mind. They are both filled with hidden clues, there is no set path, and sometimes you must solve complex problems. There is unmistakable value in discovering threats, so that we can improve the security of our organizations.

At one point in my military career, I worked as a networking specialist on a cyber-protection team, where I became a network traffic analysis expert. The mission was simple — just hunt. Remember, no good treasure hunt starts without a treasure map. That's where this threat-hunting story begins.

I received a PDF version of the network map that contained hundreds of endpoints, ports, protocols, and services (PPS) to quickly identify acceptable and normal network traffic as a baseline. The posters we printed from the PDF were the size of twin-sized blankets. Yet, we hung them up on the ops floor. We had our "treasure map," and set out to analyze the hex and packet captures (PCAPs).

We found nothing that deviated from the baseline and PPS listing. But I still had the unshakeable belief from my youth that I would find something if I looked hard enough. We were parsing through millions of network events and terabytes of data. I decided to investigate the top-talking ports – even the acceptable ports outlined on the PPS.

"Manual" threat hunting, for lack of better words, relies on highly skilled people and the knowledge they have collected over years in the field. Down the list I went looking at HTTPS, HTTP, DNS, SMTP, and so on. Finally, I arrived at port 1433, or SQL, which is the primary language used for managing data. This was significant as it's often a large attack surface for hackers and adversaries. I built a query and modified data fields to quickly identify the IPs communicating with one another.

One pair of IPs looked a little unusual and didn't fit into the schema of the other IPs. It stood out to me because I understood the network (thanks to our trusty map). That's when I discovered unencrypted SQL data. I could see everything and the data in these tables made my jaw drop. This data was leaving the network unencrypted. I immediately notified my mission commander, who carried it up the chain – we discovered it was a configuration error that was quickly fixed. The goal of discovering threats was so that we could take action to remediate them.

Learning From Experience

Eight-year-old me would have been very proud of my ability to find such a significant threat – being able to improve our security was certainly of value. There were many lessons to be learned from this hunt that I have held with me over the years:

Understand the network you're working on to easily recognize patterns and behaviors that deviate from normal.
Question and review all traffic, even acceptable or normal traffic. Network traffic is guilty until proven innocent in the world of threat hunting.
Not every hunt will result in threats being found, but always listen to your instincts. If you know it's out there, it probably is.

Threat hunting is an opportunity to help a greater good. Cyberattacks are relentless. We must work together as professionals to change the playing field. Equally, be willing to accept help. I have been fortunate to find work that brings me so much joy. I love what I do because it connects me with a lifelong drive. There are many specialties within cybersecurity; finding your particular niche will make you successful in this field.

Results for “"instincts"”

Name That Edge Toon: Animal Instincts

Lessons From the Cybersecurity Trenches