Evaluating Models with EvidenceForge

- 12 mins read

Series: Capstone

Benchmarking Models with EvidenceForge Alright, so previously we did some hardware benchmarking to validate that our selected models could reasonable run on my current hardware. Those tests all went well, which was good, but now we are getting into the real meat and potatoes of this project. Evaluating these models security reasoning skills. How are we going to go about doing that? Well, I’m glad you asked. Enter, EvidenceForge. EvidenceForge is an open-source project from Cisco Talos that can generate synthetic security telemetry from scenario definitions, keeps the generated evidence separate from the answer key, and gives me a much better basis for comparing local models than a few improvised chat prompts.

Building llama.cpp with CUDA

- 9 mins read

Series: Capstone

Using llama.cpp with CUDA and Testing Local Models Howdy there everyone and welcome back to the next leg of my capstone journey. Which involves actually spinning up some locally hosted models and testing them out. Over the next while, we’re going to be deploying a few models, seeing how they run on my old RTX 2060 laptop, benchmarking their security reasoning skills and their ability to call tools from a MCP server.

Deploying Shuffle as My SOAR Solution

- 5 mins read

Series: Capstone

Introduction Last time, we deployed Security Onion which will be our SIEM of choice for this project. Now we need a SOAR (Security Orchestration Automation and Response) solution. Which brings us to Shuffle, an open-source SOAR platform we can use to build playbooks containing predefined actions for responding to suspicious activity. Before we get going, I’m deploying Shuffle on a recently created Ubuntu server VM using Docker Compose. Versions listed below.

Deploying Security Onion

- 12 mins read

Series: Capstone

Introduction Alright, time to get down to some serious business. From this point forward, all of my blog posts will be in support of my master’s degree capstone. I may be a little less chatty than I normally am for a while as my main focus is just getting these rolled out in a timely fashion. These posts will still go over what we’re deploying and how we’re doing it and there’s going to be a little more information on things like version numbers and any issues we run into or anything during deployment.

Homelab: The OPNsense Gauntlet

- 16 mins read

Series: Homelab Series

Introduction I don’t usually start these off with a disclaimer, but I definitely am this time. This post is an absolute doozy as we are configuring multiple services here on OPNsense. Each of these could be a separate post, but I decided to do this whole thing in one go. Now you are totally good to just skip to the sections you are interested in reading, in fact I encourage it.

Homelab: Setting Up Ollama and Open-WebUI

- 7 mins read

Series: Homelab Series

Introduction Why hello there everyone and welcome back to what is essentially a bonus to the homelab series and also the start of a kind of a new one. Today we are going to be installing Ollama, pulling down a model from Hugging Face and then giving the model a web GUI with Open-WebUI. Now the reason this was not a part of the original homelab plan, is because this is actually also the beginning of my capstone research project and so is kind of its own thing.

Homelab: Deploying FleetMDM

- 10 mins read

Series: Homelab Series

Introduction Hello everyone and welcome back to the continuation of our homelab series. If you happen to be reading these in order I wanna say welcome back, I know it’s been a bit. Been juggling a few different projects, but hoping to kinda do a big dump of posts here and then it might be a little bit before the next batch. And that’s cool and all, but you’re here to install and deploy FleetMDM with me so let’s get to that.
Introduction Hello everyone and welcome back to the next post in our Detection Engineering series. In the last post, we started getting our feet wet with detection engineering by reading through a Cisco Talos report on Chaos ransomware’s msaRAT and creating a couple Sigma rules from the activity described in the report. We made one rule for the suspicious curl.exe command used to download the malware and another for Chrome or Edge launching in headless mode with the Chrome DevTools Protocol enabled.
Introduction Hello everyone and welcome back to another post from yours truly. Today we’re going to be doing something slightly different actually. I’ve been looking more into detection engineering recently and been wanting to get better at writing detection rules in different formats and whatnot. So today we’re going to be actually looking at a threat intelligence post from Cisco Talos covering some malware from the Chaos ransomware group. Specifically, their msaRAT malware that uses the browser for its C2 communications.

Delpoying Grafana, Prometheus and Uptime Kuma

- 7 mins read

Series: Homelab Series

Deploying Grafana, Prometheus and Uptime Kuma Introduction So this entries going to be a bit of a doozy everyone. We’re going to be deploying all of our non-security monitoring tools. By that I mean tools that monitor uptime and system resource usage, good old’ fashioned sys admin things. To do this we will be deploying three different services. First off Grafana, a very popular data visualization tool which I cannot express just how much you can do with it.