Unified Threat Detection Platform with Azure Sentinel

Background

Three security tools, each doing its job. Carbon Black covering endpoints. Cisco Umbrella at the DNS layer. Defender for Servers watching the workloads. All of them generating telemetry. None of them talking to each other.

The gap wasn't in the tooling. It was in correlation. An attacker could compromise an endpoint, move through the network, and access a cloud workload — and the signals would sit in three separate dashboards, waiting for an analyst to manually connect the dots. That takes time. Dwell time grows in exactly that gap.

The objective was clear: unify all three sources into one platform, automate the correlation, and get first-response actions out of the analyst's hands so they could focus on investigation rather than log aggregation.

Anil Choudhary led the architecture, integration, and operationalization from design through production handover.

The Challenge

Fragmented Security Ecosystem

The three tools in use each addressed a different attack surface, but operated in complete isolation:

A single incident touching all three surfaces would generate alerts in three places simultaneously. Without correlation, the security team couldn't tell whether three alerts were three separate events or one coordinated attack in progress. Big difference. Hard to know without a unified view.

No Centralized Visibility

With no SIEM in place:

• There were no unified dashboards — each tool had its own portal
• Alert fatigue was high — analysts triaged each tool independently, duplicating effort
• There was no baseline for normal behaviour, making anomaly detection entirely manual
• Historical investigation required querying three separate data stores with different query languages
• Compliance reporting required manual extraction and consolidation of logs from multiple sources

Log Ingestion Complexity

Even when the decision to implement Sentinel was made, the technical path to ingestion was non-trivial:

• Carbon Black used two parallel delivery mechanisms — a REST API for real-time alerts and an S3 bucket for bulk event logs — each requiring a separate integration path
• Cisco Umbrella delivered DNS proxy and firewall logs to an S3 bucket in a different format, with authentication handled via AWS IAM roles
• Defender for Servers used the Log Analytics agent, which required onboarding validation across Azure VMs, on-premises servers, and cross-cloud workloads

Building reliable, monitored pipelines for each source — with error handling, retry logic, and alerting on ingestion failures — was a significant engineering effort before any detection value could be realized.

Data Availability Issues

During the integration phase, a recurring class of problem emerged: pipelines completing successfully but data not appearing in Sentinel. The root causes varied:

• S3 bucket prefixes misconfigured — logs delivered to incorrect paths, connector polling the wrong location
• API authentication tokens expiring silently — functions executing without error but returning empty payloads
• Log Analytics ingestion latency — data appearing correct in the function output but delayed by up to 15 minutes in the workspace
• Field mapping mismatches — structured log fields not mapping to the expected Sentinel schema, causing records to be dropped at the parser stage

Each of these required a different diagnostic and resolution approach, documented into a runbook for the operations team.

Hybrid Infrastructure Scope

The Defender for Servers integration had to cover infrastructure across three environments:

• Azure VMs across multiple subscriptions
• On-premises Windows and Linux servers connected via Azure Arc
• Workloads running on a secondary cloud provider, also onboarded via Arc

Ensuring consistent agent deployment, policy coverage, and log forwarding across all three environments — with different network paths and authentication models — required careful coordination.

Architecture

The platform was designed around a hub-and-spoke model with Azure Sentinel as the central intelligence layer and Azure Functions as the serverless ingestion fabric.

Platform Overview

┌─────────────────────────────────────────────────────┐
│                  Azure Sentinel                      │
│     Analytics Rules │ Incidents │ Threat Hunting     │
│         Workbooks │ Playbooks (Logic Apps)           │
└──────────────────────┬──────────────────────────────┘
                       │
         ┌─────────────▼────────────┐
         │   Log Analytics Workspace │
         │   (centralised data store)│
         └──┬──────────┬────────────┘
            │          │
    ┌───────▼──┐  ┌────▼────────────────────────────────┐
    │ Defender  │  │       Azure Functions (3 pipelines) │
    │ for       │  │                                     │
    │ Servers   │  │  ┌──────────┐  ┌────────────────┐  │
    │ (agent)   │  │  │  Carbon  │  │ Cisco Umbrella │  │
    └───────────┘  │  │  Black   │  │  S3 Pipeline   │  │
                   │  │  API+S3  │  └───────┬────────┘  │
                   │  └────┬─────┘          │           │
                   └───────┼────────────────┼───────────┘
                           │                │
                    ┌──────▼──┐      ┌──────▼──┐
                    │  Carbon  │      │  Cisco   │
                    │  Black   │      │ Umbrella │
                    │  Cloud   │      │    S3    │
                    └──────────┘      └──────────┘

Log Analytics Workspace Configuration

A single Log Analytics workspace was designated as the canonical data store for all security telemetry. This decision was deliberate — a single workspace enables cross-source KQL queries without workspace federation overhead.

Workspace settings:

• Data retention: 90 days hot (queryable), 365 days archived
• Daily ingestion cap configured per source to prevent runaway cost from a misconfigured pipeline
• Diagnostic settings from all Azure resources forwarded to the same workspace
• Sentinel enabled on the workspace with Contributor role scoped to the security team

Serverless Ingestion Layer

Azure Functions (Python runtime, Consumption plan) were built for each third-party source. The serverless model provided cost efficiency at variable ingestion volumes and isolated each pipeline so a failure in one did not affect others.

Function Architecture Pattern

Each function followed the same structure:

import logging
import json
import os
import requests
from azure.monitor.ingestion import LogsIngestionClient
from azure.identity import DefaultAzureCredential

def main(mytimer) -> None:
    credential = DefaultAzureCredential()
    client = LogsIngestionClient(
        endpoint=os.environ["DATA_COLLECTION_ENDPOINT"],
        credential=credential
    )
    
    logs = fetch_from_source()          # Source-specific fetch
    normalized = normalize_logs(logs)   # Schema normalization
    
    if normalized:
        client.upload(
            rule_id=os.environ["DCR_RULE_ID"],
            stream_name=os.environ["STREAM_NAME"],
            logs=normalized
        )
        logging.info(f"Ingested {len(normalized)} records")
    else:
        logging.warning("No records returned from source — investigating")

Application Insights was connected to all three functions, providing execution traces, dependency tracking, and custom alerts on ingestion failures or empty-payload warnings.

Carbon Black Integration

Carbon Black (VMware Carbon Black Cloud) provides Next-Generation Antivirus (NGAV) and Endpoint Detection and Response (EDR) capabilities across the endpoint estate.

Log Types and Ingestion Methods

Three log categories were integrated, each via a different mechanism:

The dual-path approach (API for near-real-time alerts, S3 for high-volume event telemetry) was driven by Carbon Black's own architecture — attempting to poll all event telemetry via API would have exceeded rate limits and increased cost significantly.

API Integration

def fetch_cb_alerts(api_id: str, api_key: str, org_key: str) -> list:
    url = f"https://defense.conferdeploy.net/appservices/v6/orgs/{org_key}/alerts/search_jobs"
    headers = {
        "X-Auth-Token": f"{api_key}/{api_id}",
        "Content-Type": "application/json"
    }
    payload = {
        "criteria": {"minimum_severity": 3},
        "sort": [{"field": "backend_timestamp", "order": "DESC"}],
        "rows": 500
    }
    response = requests.post(url, headers=headers, json=payload)
    return response.json().get("results", [])

S3 Batch Ingestion

Carbon Black delivers endpoint event logs to an S3 bucket in compressed JSON format. The ingestion function:

1. Polls the S3 bucket prefix on a timer trigger
2. Decompresses and parses each log file
3. Normalizes field names to align with the Sentinel schema
4. Batches records into payloads of up to 1 MB before uploading to Log Analytics

A checkpoint mechanism was implemented to track the last processed S3 object key, preventing duplicate ingestion on function retries.

Detection Value

With Carbon Black data flowing into Sentinel, analytics rules were configured for:

• Processes spawned by Office applications (common malware delivery vector)
• Lateral movement indicators: PsExec, WMI remote execution, net use commands
• Credential access patterns: LSASS memory access, mimikatz-like behaviour
• Persistence mechanisms: scheduled task creation, registry run key modification

Cisco Umbrella Integration

Cisco Umbrella provides DNS-layer security — blocking malicious domains before a connection is ever established — and acts as a secure web gateway for all outbound internet traffic.

Integration Architecture

Umbrella was configured to deliver logs to an S3 bucket in three categories:

• DNS logs — every DNS query made by every device on the network
• Proxy logs — HTTP/HTTPS traffic routed through the Umbrella proxy
• Firewall logs — network-layer traffic decisions at the Umbrella cloud firewall

The Azure Function for Umbrella used AWS SDK (boto3) to pull from the S3 bucket using IAM role credentials stored in Azure Key Vault:

import boto3
from azure.keyvault.secrets import SecretClient

def get_umbrella_credentials(vault_url: str) -> dict:
    credential = DefaultAzureCredential()
    client = SecretClient(vault_url=vault_url, credential=credential)
    return {
        "aws_access_key_id": client.get_secret("umbrella-s3-key-id").value,
        "aws_secret_access_key": client.get_secret("umbrella-s3-secret").value,
        "region_name": "us-east-1"
    }

Ingestion Challenge: Missing DNS Logs

During initial testing, DNS logs were not appearing in Sentinel despite the function executing successfully. The diagnostic process:

1. Verified S3 bucket contents directly via AWS Console — logs were present at the root of the bucket
2. Checked the function's prefix configuration — the connector was polling dnslogs/ but Umbrella was writing to dns-logs/ (underscore vs hyphen)
3. Updated the prefix — logs immediately began flowing
4. Added prefix validation to the function startup check so misconfiguration would surface as a startup error rather than silent empty payloads

This class of issue — silent failure due to incorrect path configuration — was documented and added to the integration runbook.

Detection Value

Umbrella data in Sentinel enabled:

• Detection of DNS requests to known C2 (command and control) domains
• Identification of data exfiltration attempts via DNS tunnelling
• Correlation of users accessing flagged domains with subsequent endpoint alerts from Carbon Black
• Phishing campaign tracking via sudden spikes in DNS queries to newly registered domains

Microsoft Defender for Servers Integration

Defender for Servers Plan 2 was enabled across the full server estate, providing vulnerability assessment, file integrity monitoring, and endpoint detection on both Azure-hosted and hybrid workloads.

Deployment Scope

Azure Arc Onboarding

On-premises and cross-cloud servers were onboarded via Azure Arc, which extends the Azure control plane to non-Azure machines:

# On-premises server onboarding script (generated from Azure portal)
$env:SUBSCRIPTION_ID = "<subscription-id>"
$env:RESOURCE_GROUP  = "rg-security-management"
$env:TENANT_ID       = "<tenant-id>"
$env:LOCATION        = "eastus"

Invoke-WebRequest -Uri https://aka.ms/azcmagent-windows -OutFile "$env:TEMP\install_windows_azcmagent.ps1"
& "$env:TEMP\install_windows_azcmagent.ps1"

azcmagent connect `
  --subscription-id $env:SUBSCRIPTION_ID `
  --resource-group $env:RESOURCE_GROUP `
  --tenant-id $env:TENANT_ID `
  --location $env:LOCATION `
  --cloud AzureCloud

Once onboarded via Arc, Defender for Servers policies applied identically to on-premises machines as to Azure VMs — vulnerability scanning, security recommendations, and alert generation all functioned the same way regardless of hosting location.

Integration with Sentinel

Defender for Servers logs are forwarded to Sentinel automatically via the Log Analytics workspace connection. No custom pipeline was required — the integration uses the native Defender for Cloud → Sentinel connector, configured once in the Sentinel data connectors blade.

Tables populated in the workspace:

• SecurityAlert — Defender-generated alerts
• SecurityEvent — Windows Security event log (process creation, logon events, object access)
• Syslog — Linux system and auth logs
• Update — Patch compliance status per machine

Threat Detection: Analytics Rules

With all three sources flowing into a single workspace, correlated detection rules were built using KQL.

Cross-Source Lateral Movement Detection

// Detect: Carbon Black endpoint alert followed by Umbrella DNS request to suspicious domain
// from the same source IP within 30 minutes
let cb_alerts = CarbonBlackEvents_CL
    | where TimeGenerated > ago(24h)
    | where alert_severity_s >= "MEDIUM"
    | project alert_time = TimeGenerated, src_ip = device_external_ip_s, device_name_s;

let umbrella_suspicious = CiscoUmbrellaProxyEvent_CL
    | where TimeGenerated > ago(24h)
    | where verdict_s == "blocked"
    | project dns_time = TimeGenerated, src_ip = identities_s, domain_s;

cb_alerts
| join kind=inner umbrella_suspicious on src_ip
| where dns_time between (alert_time .. (alert_time + 30m))
| project alert_time, dns_time, src_ip, device_name_s, domain_s
| order by alert_time desc

Credential Access on Monitored Endpoints

// Detect LSASS memory access — common credential dumping indicator
SecurityEvent
| where TimeGenerated > ago(1h)
| where EventID == 4656  // A handle to an object was requested
| where ObjectName has "lsass"
| where AccessMask in ("0x1010", "0x1410", "0x147a", "0x143a")
| join kind=leftouter (
    CarbonBlackEvents_CL
    | where TimeGenerated > ago(1h)
    | project cb_alert = alert_severity_s, device_name_s
) on $left.Computer == $right.device_name_s
| project TimeGenerated, Computer, SubjectUserName, ObjectName, cb_alert

DNS Tunnelling Detection

// High-frequency DNS queries to same domain — indicator of DNS tunnelling
CiscoUmbrellaDnsEvent_CL
| where TimeGenerated > ago(1h)
| summarize query_count = count(), unique_subdomains = dcount(domain_s) 
    by bin(TimeGenerated, 5m), parent_domain = tostring(split(domain_s, ".")[-2])
| where unique_subdomains > 50 and query_count > 200
| order by query_count desc

Incident Response Automation

Azure Logic Apps were configured as Sentinel playbooks to automate first-response actions on high-confidence alerts.

Carbon Black Alert — Automatic Device Isolation

On a Critical severity alert from Carbon Black:

1. Sentinel fires the playbook via analytics rule trigger
2. Logic App calls the Carbon Black API to quarantine the affected device from the network
3. A ticket is created in the ITSM system with the alert details and quarantine confirmation
4. The security team receives a Teams notification with a direct link to the Sentinel incident

{
  "type": "Http",
  "inputs": {
    "method": "POST",
    "uri": "https://defense.conferdeploy.net/appservices/v6/orgs/@{variables('OrgKey')}/device_actions",
    "headers": { "X-Auth-Token": "@{variables('CBAuthToken')}" },
    "body": {
      "action_type": "QUARANTINE",
      "device_id": ["@{triggerBody()?['properties']?['relatedEntities']?[0]?['id']}"],
      "options": { "toggle": "ON" }
    }
  }
}

Umbrella Domain Block — Automatic Response

On a confirmed phishing or C2 domain detection:

1. Domain extracted from the Sentinel incident entity
2. Logic App calls the Cisco Umbrella Enforcement API to add the domain to the block list
3. Block is applied globally across all users within seconds
4. Incident updated with the enforcement action taken

SecOps Enablement

Unified Threat Hunting

The consolidated workspace enables analysts to threat hunt across all three data sources simultaneously. Saved queries were built for common investigation patterns:

• User investigation: all alerts, DNS activity, and logon events for a specific user account over a time window
• IP pivot: all events touching a given internal or external IP — endpoints that communicated with it, DNS queries resolved to it, Defender alerts involving it
• Malware family timeline: reconstructing the full kill chain for a known threat actor across all telemetry sources

Custom Workbooks

Two Sentinel workbooks were built for the security team:

Security Posture Overview — executive-facing dashboard showing:

• Active incident count by severity
• Trend of alerts by source over 7 days
• Top 10 affected endpoints (Carbon Black)
• Top blocked domains (Cisco Umbrella)
• Defender for Cloud secure score

Ingestion Health Monitor — operational dashboard showing:

• Ingestion volume per source per hour
• Last successful function execution timestamp per pipeline
• Record counts vs expected baseline (deviation alerts for silent failures)
• Log Analytics ingestion latency metrics

Results

What used to take an analyst hours of manual pivoting now surfaces as a single Sentinel incident — evidence from all three sources already attached, priority already set. The correlation gap is gone.

For the highest-severity alerts, containment happens in seconds. Device isolation, domain blocking — automated and confirmed before a human opens the incident. The window between detection and response for those alert classes dropped from hours to seconds.