Advanced Monitoring and Detection

Key Control Areas

• Security Event Monitoring and SIEM (AU-02, AU-06, SI-04): These controls form the core detection capability. Auditable events (AU-02) defines what must be logged across the environment -- authentication events, privilege escalation, process execution, network connections, file access, configuration changes, and administrative actions. The scope must cover endpoints, servers, network devices, cloud services, identity systems, and applications. Audit monitoring, analysis, and reporting (AU-06) requires centralising these logs into a SIEM platform with correlation rules, statistical baselines, and threat intelligence enrichment. Information system monitoring (SI-04) extends beyond log analysis to include network-based detection: IDS/IPS, network traffic analysis, DNS monitoring, and full packet capture at key network boundaries. Detection logic should be mapped to the MITRE ATT&CK framework to identify coverage gaps, and custom detections should be developed for organisation-specific attack scenarios that generic vendor rules will not catch.
• Configuration Baseline and Drift Detection (CM-01, CM-02, CM-03, CM-05, CM-06): Advanced detection depends on knowing what 'normal' looks like. Configuration management policy (CM-01) establishes the governance framework. Baseline configurations (CM-02) define the expected state of systems -- operating system versions, installed software, running services, open ports, registry settings, and security configurations. Configuration change control (CM-03) ensures that all authorised changes are documented, enabling the SOC to distinguish legitimate changes from attacker activity. Access restrictions for change (CM-05) limits who can modify configurations, making unauthorised changes a high-fidelity detection signal. Configuration settings (CM-06) defines the security-relevant parameters that must be monitored for drift. File integrity monitoring tools should continuously compare production systems against approved baselines and alert on deviations. This is how you detect an attacker who has modified a system binary, installed a backdoor, or changed a firewall rule.
• Vulnerability and Risk Intelligence (RA-01, RA-02, RA-03, RA-05): Effective detection requires understanding the attack surface. Risk assessment policy (RA-01) and security categorisation (RA-02) establish which systems matter most and what level of monitoring they require -- a compromise of a payment processing server demands a different response than a compromise of a test workstation. Risk assessment (RA-03) identifies the threats the organisation faces and the vulnerabilities that could be exploited, directly informing detection priorities. Vulnerability scanning (RA-05) provides continuous visibility into the patch state and configuration weaknesses across the environment. Vulnerability data should be integrated with the SIEM so that when an exploit attempt is detected, analysts can immediately determine whether the target is actually vulnerable -- dramatically reducing investigation time and enabling confident prioritisation of alerts.
• Access and Identity Monitoring (AC-02, AC-04, AC-17, AC-18, AC-20): Compromised credentials and lateral movement through legitimate access channels are the primary tactics of advanced adversaries. Account management monitoring (AC-02) should detect anomalous account creation, privilege escalation, service account abuse, and dormant account activation. Information flow enforcement (AC-04) monitoring identifies data exfiltration through unexpected channels. Remote access monitoring (AC-17) should flag connections from unusual locations, at unusual times, or using unusual client configurations. Wireless access monitoring (AC-18) detects rogue access points and unauthorised wireless connections. Use of external information systems (AC-20) monitoring identifies data flowing to unsanctioned cloud services or personal devices. Identity-based detection is often the highest-value detection category because adversaries who have compromised credentials can operate within legitimate access channels that network-based controls will not flag.
• Incident Response Integration (IR-01, IR-03, IR-05): Detection without response is academic. Incident response policy (IR-01) must define clear escalation paths from SOC detection through triage, investigation, containment, and remediation. Incident response testing (IR-03) should include regular exercises that test the full detection-to-response chain: inject simulated attack indicators into the environment and measure time to detect, time to triage, and time to contain. Incident monitoring (IR-05) maintains awareness of active incidents, tracks containment progress, and ensures that indicators of compromise from current incidents are fed back into detection systems to catch related activity. The SOC and incident response team must have pre-authorised containment actions -- the ability to isolate endpoints, disable accounts, or block network connections without waiting for management approval during an active attack.
• Software and Information Integrity (SI-03, SI-06, SI-07): These controls detect compromise at the system level. Malicious code protection (SI-03) extends beyond traditional antivirus to include EDR (Endpoint Detection and Response) tools that monitor process behaviour, detect fileless malware, and provide forensic telemetry. Security functionality verification (SI-06) periodically validates that security controls are operating correctly -- verifying that logging agents are running, detection rules are firing, and security tools have not been tampered with. Software and information integrity (SI-07) uses cryptographic hashing and file integrity monitoring to detect unauthorised modifications to system files, application binaries, and configuration files. Integrity monitoring should cover boot processes, kernel modules, critical system libraries, and web application files where attackers commonly plant webshells.
• System Architecture and Secure Development Practices (SA-03, SA-06, SA-07, SA-08): These controls ensure that the monitoring architecture itself is secure and sustainable. Life cycle support (SA-03) ensures monitoring tools and platforms are maintained and updated -- a SIEM running outdated correlation rules or unsupported software becomes a liability rather than an asset. Software usage restrictions (SA-06) and user installed software controls (SA-07) reduce the attack surface by limiting what can execute in the environment, making detection of unauthorised software more reliable. Security engineering principles (SA-08) guide the design of the monitoring architecture: defence in depth (multiple independent detection layers), fail-safe defaults (alert on monitoring gaps), separation of duties (SOC analysts cannot modify the systems they monitor), and resilience (monitoring infrastructure must survive the attacks it is designed to detect).

When to Use

Apply this pattern if your organisation may be a likely target of sophisticated blended attacks characteristic of Advanced Persistent Threats (APTs). This includes organisations in financial services, government, defence, critical infrastructure, healthcare, technology, and any sector handling high-value intellectual property or personally identifiable information. Regulatory requirements increasingly mandate continuous monitoring capabilities (NIST CSF DE.CM, PCI DSS Requirement 10, SOC 2 CC7, ISO 27001 A.8.15-16). Organisations that have experienced a breach or near-miss should implement this pattern as a priority. Any organisation with more than a few hundred endpoints should have at minimum a managed detection and response (MDR) capability.

When NOT to Use

Do not attempt to implement all elements of this pattern unless your organisation has high operational maturity with respect to change and configuration management. Key detective controls are dependent on accurate configuration management data -- without reliable baselines, anomaly detection generates unmanageable false positive volumes. Organisations at low security maturity should first establish basic controls (asset inventory, configuration management, centralised logging) before investing in advanced detection capabilities. Attempting to deploy a SIEM without the underlying data quality and staffing to operate it results in expensive infrastructure that provides little security value.

Typical Challenges

The primary operational challenge is alert fatigue: SOC analysts overwhelmed by false positives will miss genuine attacks buried in the noise. Addressing this requires continuous detection engineering -- tuning rules, retiring ineffective detections, enriching alerts with context, and automating repetitive triage tasks through SOAR (Security Orchestration, Automation, and Response) platforms. The volume of log data can be enormous, and cost management for SIEM licensing (often based on ingestion volume) frequently leads to difficult decisions about which log sources to include. Excluding sources creates blind spots that sophisticated adversaries will exploit. Achieving operational maturity with respect to configuration management is a prerequisite for many detection capabilities; organisations that cannot maintain accurate baselines will struggle with change detection and anomaly-based alerting. Staffing the SOC with skilled analysts is a persistent industry-wide challenge, with high turnover rates driven by burnout, shift work, and competitive demand for experienced security analysts. Measuring detection effectiveness is difficult -- you can count alerts and incidents, but the metric that matters most (attacks that were not detected) is inherently unknowable without regular red team exercises and assumed-breach testing.

Threat Resistance

This pattern is specifically designed to detect and limit the impact of advanced threats that bypass preventive controls. Advanced Persistent Threats (APTs) using multi-stage attack chains involving initial access, privilege escalation, lateral movement, and data exfiltration over extended dwell times. Living-off-the-land attacks that use legitimate system tools (PowerShell, WMI, PsExec) to avoid triggering signature-based detection. Credential-based attacks including pass-the-hash, Kerberoasting, and golden ticket attacks that operate within legitimate authentication channels. Insider threats where authorised users abuse their access for data theft or sabotage. Ransomware in its pre-encryption phases -- reconnaissance, lateral movement, and staging -- where early detection enables containment before encryption begins. Supply chain compromises where tampered updates or dependencies introduce backdoors. Zero-day exploitation where signature-based detection fails and behavioural analytics provide the primary detection mechanism. Data exfiltration through encrypted channels, DNS tunnelling, or steganography where network-based anomaly detection is required.

Assumptions

This pattern assumes that primary defensive layers (perimeter security, endpoint protection, access controls) have already been implemented but are insufficient against sophisticated adversaries who can bypass preventive controls. The organisation has sufficient operational maturity in change and configuration management to establish meaningful baselines against which anomalies can be detected -- without accurate configuration data, many detective controls produce excessive false positives and become operationally useless. The organisation is prepared to invest in skilled analysts or managed detection services, as monitoring technology without qualified human analysis generates noise rather than intelligence. Log sources across the environment are available and can be centralised, and sufficient storage and processing capacity exists to retain and analyse security data at the required scale.

Developing Areas

• AI-driven SOC triage is rapidly evolving from experimental to operational, with platforms using large language models to summarise alerts, correlate related events, and recommend analyst actions. Early deployments report 40-60% reduction in tier-1 triage time, but analyst trust in AI-generated recommendations remains a barrier -- analysts who cannot understand or verify the AI's reasoning are reluctant to act on its conclusions. The discipline is navigating the gap between AI capability and analyst confidence, with explainability and auditability of AI triage decisions as the critical unsolved design problems.
• Detection-as-code is maturing as a discipline but lacks standardised tooling and practices. The concept -- managing detection logic in version-controlled repositories with CI/CD testing, peer review, and automated deployment -- is well-established in principle, but the ecosystem is fragmented across incompatible formats (Sigma, KQL, SPL, YARA-L) and no universal detection language has emerged. Organisations investing in detection engineering face portability challenges when switching SIEM platforms and must maintain translation layers between detection formats.
• Security data lake architectures are challenging the traditional SIEM model by decoupling data storage from analytics. Platforms built on cloud object storage (Snowflake, Databricks, Amazon Security Lake) offer dramatically lower per-GB costs than traditional SIEM, enabling retention of telemetry that would be cost-prohibitive in SIEM-based architectures. However, the query performance, real-time alerting, and out-of-the-box detection capabilities of data lakes still lag purpose-built SIEMs, and most organisations are running hybrid architectures with SIEM for real-time detection and data lakes for threat hunting and long-term retention.
• SOAR effectiveness and maintenance burden is generating industry disillusionment after initial enthusiasm. Organisations that deployed SOAR platforms find that playbook development and maintenance requires dedicated engineering resources that were underestimated at purchase, and that the promise of fully automated response encounters practical barriers -- edge cases, API changes in integrated tools, and the need for human judgement at critical decision points. The emerging pattern is focused SOAR deployment for a small number of high-frequency, well-understood scenarios rather than attempting to automate the entire response lifecycle.
• Cloud-native detection coverage remains a significant gap for organisations with hybrid environments. Cloud provider audit trails (CloudTrail, Azure Activity Log, GCP Audit Logs) provide rich telemetry, but detection logic developed for on-premises environments does not translate directly to cloud-native attack patterns. Techniques like role assumption chaining in AWS, managed identity abuse in Azure, and service account key theft in GCP require purpose-built detections that most SOCs are still developing.