The Interview
What I was expecting to be extremely nerve-racking turned out to be the smoothest interview experience of my life. A recruiter from Sysdig reached out to me the same week I applied. From applying to the position to signing an internship offer, the entire process took about three weeks. The interviews were short and focused on truly getting to know whether I was the right fit for the job. Sysdig seemed to know exactly what they were looking for, both in terms of technical knowledge and work ethic. The recruiter and the engineers I met with were transparent about the process, which helped put my mind at ease.
New Kid on the Block
The first few weeks of the internship were exciting, and it only got better. It was intimidating at first, but I was lucky enough to be assigned a co-intern who was just as nervous as me. Side-by-side in onboarding, we were given a chance to explore the tech stack we would be working with through the three-month internship. To my surprise, we were given four weeks to learn all of the little details we needed to know for the project. This included playing with Docker and Kubernetes, launching attacks with Metasploit (it is as cool as it sounds), a whole new programming language (Golang, if you’re wondering), and working with machine learning frameworks like TensorFlow and scikit-learn.
The Project
I spent most of the fourth week learning about the project requirements and expectations. This is also around the time when I realized the significance of the project, and that Sysdig was willing to trust two interns with something so special. The goal of the project was to build a data pipeline that accesses Linux system calls (using Sysdig’s open-source tool), and processes these syscalls to be used as input to our machine learning (ML) model. The model is then expected to classify each Linux command as either suspicious or normal activity, even if the hacker were to rename the command to something that looks nonthreatening. Finally, the user needed to be notified in the case of suspicious activity, complete with details surrounding the attack. At this point, we had all the tools necessary to start working on this seemingly daunting project. We had even created a design document, complete with a test plan and a timeline of weekly milestones. It was time to get to work!
I had the misconception that I had learned a lot in the first weeks of this internship; I was about to be proven wrong. As the weeks passed by, I was learning something new every hour of every day, and I loved the challenge. We had two mentors, a security genius and a ML expert, who were more than happy to answer all kinds of questions. As with most ML projects, there was heavy research that went into different model architectures and choosing the right one for the Systerns project. There were many times where we had to discard entire models and start fresh. There were also instances when my project partner and I were unsure if recognizing renamed Linux commands and classifying their behavior as malicious (or not) was even possible. However, our mentors truly believed that it could be done and pushed us out of our comfort zones, which is what I am most grateful for.
The Final Obstacle
The final weeks of the internship were monumental to the project’s success. At this point, we knew that this was not an impossible goal and immersed ourselves in improving our model’s accuracy rate. The model went from barely being able to classify commands correctly to recognizing and classifying commands with around 80% accuracy on commonly used Linux commands, and nearly 100% accuracy on crypto-mining attacks.
Our final experiment was testing the pipeline with a type of malware attack called Kinsing, an attack that runs continuously, using up all of the container’s resources. I can personally attest to the Kinsing attack’s destructive behavior after leaving the attack script running for a few hours. Needless to say, the container was unrecoverable. This incident only emphasized the need for detection and notification of suspicious behavior. Since our model had never seen this attack before (and only trained with other variants of crypto-mining attacks), we were skeptical. As soon as we saw the attack notification pop up with 99% confidence, we felt instant relief. This meant that we had managed to build something useful, all while being interns and with constant help from our mentors.
Lessons Learned
I can go on for hours about the technical skills I’ll be leaving with that are very much outside the scope of my classes. These are skills that are specific to the network security and machine learning fields. At the beginning of this internship, I knew very little about machine learning, and even less about computer network security. Now, I am able to identify and launch attacks on vulnerable containers, code comfortably in Golang, work with low-level Linux system calls, deliver software in packages using Docker, and navigate TensorFlow with ease.
More importantly, I gained major life skills from working with an extremely competent team. Our supervisor, Kaizhe, taught me how valuable it is to set reasonable goals and be prepared for changes that cannot be foreseen. Omer, the VP of Engineering at Sysdig who also oversaw the internship project, taught me to think unconventionally and that considering all solutions is just as important as recognizing the right approach to solving the problem at hand. The ML expert we went to with many tough questions, Flavio, made me realize that research and experimentation go hand in hand, and that atypical solutions that often don’t work for others may be the perfect fix for you. From my co-intern, Krishna, I learned the true meaning of an excellent work ethic and communication skills. Not only did she make me a better coder, she also made me more disciplined and looked forward to tackling the next problem with a winning sense of humor. I have never worked on a project so challenging yet gratifying, and I couldn’t have asked for a better team to figure it out with.