Building an SOC & Incident Response Platform

Over the past few days, I designed and implemented a custom Security Operations Center (SOC) platform integrating Elasticsearch, FastAPI, automated detection pipelines, and dynamic incident investigation workflows.

The goal of this project was to move beyond static dashboards and build a real-world security operations architecture capable of ingesting alerts, detecting threats, correlating security events, and preparing the foundation for autonomous incident response.

Core Architecture

The platform currently consists of:

• Elasticsearch for log indexing and security event storage

• Custom Python detection engine for real-time threat detection

• FastAPI backend for authenticated incident ingestion APIs

• Dynamic incident investigation dashboard

• Nginx reverse proxy infrastructure

• Secure API token authentication

• Real-time incident feed pipeline

Detection Pipeline

A custom detection engine continuously queries Elasticsearch for suspicious activity such as failed authentication attempts and brute-force behavior.

When suspicious activity is detected, the engine automatically:

1. Correlates matching events

2. Extracts attacker IPs and metadata

3. Creates structured incidents

4. Sends incidents to the FastAPI backend

5. Makes incidents immediately available to the investigation dashboard

This creates a fully automated detection-to-investigation workflow.

Challenges & Engineering Lessons

One of the most valuable parts of the project was solving real-world infrastructure and dependency issues, including:

• Elasticsearch client/server version mismatches

• Python virtual environment isolation

• API authentication design

• Reverse proxy configuration

• Incident ingestion architecture

• Dependency management and package pinning

• Secure backend communication

• Production-style debugging workflows

The project evolved from a simple dashboard into a modular security engineering platform.

Challenges that I faced :

• Installed and configured Elasticsearch on a VPS

• Debugged curl: (52) Empty reply from server

• Fixed Elasticsearch 8.x ↔ Python client 9.x incompatibility

• Rebuilt FastAPI incident ingestion routes

• Added Bearer token authentication for /incident

• Recovered broken Python virtual environments

• Fixed PM2 process crashes and backend restart issues

• Created a custom Elasticsearch detection engine in Python

• Connected live detections to a FastAPI incident API

• Synced real incidents into the frontend investigation dashboard

• Replaced mock incident data with live backend ingestion

• Generated reproducible environments using pip freeze > requirements.txt

Current pipeline:

Elasticsearch → Detection Engine → FastAPI → Incident Investigation UI

Finally, seeing live security incidents appear inside the platform in real time.

Current Capabilities

The platform currently supports:

• Real-time incident ingestion

• Token-protected alert APIs

• Dynamic incident visualization

• Elasticsearch correlation

• Security alert pipelines

• Incident investigation workflows

• Severity and category mapping

• Centralized incident feeds