Time series anomaly detection for total volume of traffic

Severity: medium
Time window: 14d
Group by: AnomalyHour, DeviceVendor, Total, anomalies, baseline, score
Author: Microsoft Security Research
Source: github.com/Azure/Azure-Sentinel

'Identifies anamalous spikes in network traffic logs as compared to baseline or normal historical patterns. The query leverages a KQL built-in anomaly detection algorithm to find large deviations from baseline patterns. Sudden increases in network traffic volume may be an indication of data exfiltration attempts and should be investigated. The higher the score, the further it is from the baseline value. The output is aggregated to provide summary view of unique source IP to destination IP address and port traffic observed in the flagged anomaly hour. The source IP addresses which were sending less than percentotalthreshold of the total traffic have been exluded whose value can be adjusted as needed . You may have to run queries for individual source IP addresses from SourceIPlist to determine if anything looks suspicious'

MITRE ATT&CK coverage

Tactic	Techniques
Exfiltration	`T1030` Data Transfer Size Limits

Rule body kusto

id: 06a9b845-6a95-4432-a78b-83919b28c375
name: Time series anomaly detection for total volume of traffic
description: |
  'Identifies anamalous spikes in network traffic logs as compared to baseline or normal historical patterns.
  The query leverages a KQL built-in anomaly detection algorithm to find large deviations from baseline patterns.
  Sudden increases in network traffic volume may be an indication of data exfiltration attempts and should be investigated.
  The higher the score, the further it is from the baseline value.
  The output is aggregated to provide summary view of unique source IP to destination IP address and port traffic observed in the flagged anomaly hour.
  The source IP addresses which were sending less than percentotalthreshold of the total traffic have been exluded whose value can be adjusted as needed .
  You may have to run queries for individual source IP addresses from SourceIPlist to determine if anything looks suspicious'
severity: Medium
requiredDataConnectors:
  - connectorId: Barracuda
    dataTypes:
      - CommonSecurityLog
  - connectorId: CEF
    dataTypes:
      - CommonSecurityLog
  - connectorId: CheckPoint
    dataTypes:
      - CommonSecurityLog
  - connectorId: CiscoASA
    dataTypes:
      - CommonSecurityLog
  - connectorId: F5
    dataTypes:
      - CommonSecurityLog
  - connectorId: Fortinet
    dataTypes:
      - CommonSecurityLog
  - connectorId: PaloAltoNetworks
    dataTypes:
      - CommonSecurityLog
queryFrequency: 1d
queryPeriod: 14d
triggerOperator: gt
triggerThreshold: 3
tactics:
  - Exfiltration
relevantTechniques:
  - T1030
query: |
  let starttime = 14d;
  let endtime = 1d;
  let timeframe = 1h;
  let scorethreshold = 5;
  let percentotalthreshold = 50;
  let TimeSeriesData = CommonSecurityLog
  | where isnotempty(DestinationIP) and isnotempty(SourceIP)
  | where TimeGenerated between (startofday(ago(starttime))..startofday(ago(endtime)))
  | project TimeGenerated,SourceIP, DestinationIP, DeviceVendor
  | make-series Total=count() on TimeGenerated from startofday(ago(starttime)) to startofday(ago(endtime)) step timeframe by DeviceVendor;
  // Filtering specific records associated with spikes as outliers
  let TimeSeriesAlerts=materialize(TimeSeriesData
  | extend (anomalies, score, baseline) = series_decompose_anomalies(Total, scorethreshold, -1, 'linefit')
  | mv-expand Total to typeof(double), TimeGenerated to typeof(datetime), anomalies to typeof(double),score to typeof(double), baseline to typeof(long)
  | where anomalies > 0 | extend score = round(score,2), AnomalyHour = TimeGenerated
  | project DeviceVendor,AnomalyHour, TimeGenerated, Total, baseline, anomalies, score);
  let AnomalyHours = materialize(TimeSeriesAlerts  | where TimeGenerated > ago(2d) | project TimeGenerated);
  // Join anomalies with Base Data to popalate associated records for investigation - Results sorted by score in descending order
  TimeSeriesAlerts
  | where TimeGenerated > ago(2d)
  | join (
      CommonSecurityLog
  | where isnotempty(DestinationIP) and isnotempty(SourceIP)
  | where TimeGenerated > ago(2d)
  | extend DateHour = bin(TimeGenerated, 1h) // create a new column and round to hour
  | where DateHour in ((AnomalyHours)) //filter the dataset to only selected anomaly hours
  | summarize HourlyCount = count(), TimeGeneratedMax = arg_max(TimeGenerated, *), DestinationIPlist = make_set(DestinationIP, 100), DestinationPortlist = make_set(DestinationPort, 100) by DeviceVendor, SourceIP, TimeGeneratedHour= bin(TimeGenerated, 1h)
  | extend AnomalyHour = TimeGeneratedHour
  ) on AnomalyHour, DeviceVendor
  | extend PercentTotal = round((HourlyCount / Total) * 100, 3)
  | where PercentTotal > percentotalthreshold
  | project DeviceVendor , AnomalyHour, TimeGeneratedMax, SourceIP, DestinationIPlist, DestinationPortlist, HourlyCount, PercentTotal, Total, baseline, score, anomalies
  | summarize HourlyCount=sum(HourlyCount), StartTimeUtc=min(TimeGeneratedMax), EndTimeUtc=max(TimeGeneratedMax), SourceIPlist = make_set(SourceIP, 100), SourceIPMax= arg_max(SourceIP, *), DestinationIPlist = make_set(DestinationIPlist, 100), DestinationPortlist = make_set(DestinationPortlist, 100) by DeviceVendor , AnomalyHour, Total, baseline, score, anomalies
  | project DeviceVendor , AnomalyHour, EndTimeUtc, SourceIPMax ,SourceIPlist, DestinationIPlist, DestinationPortlist, HourlyCount, Total, baseline, score, anomalies
entityMappings:
  - entityType: IP
    fieldMappings:
      - identifier: Address
        columnName: SourceIPMax
version: 1.0.4
kind: Scheduled
metadata:
    source:
        kind: Community
    author:
        name: Microsoft Security Research
    support:
        tier: Community
    categories:
        domains: [ "Security - Others" ]

Stages and Predicates

Parameters

let starttime = 14d;
let endtime = 1d;
let timeframe = 1h;
let scorethreshold = 5;
let percentotalthreshold = 50;

Let binding: `AnomalyHours`

let AnomalyHours = materialize(TimeSeriesAlerts  | where TimeGenerated > ago(2d) | project TimeGenerated);

Derived from TimeSeriesAlerts.

The stages below define let TimeSeriesAlerts (the rule's main pipeline source).

Stage 1: `source`

TimeSeriesData

Stage 2: `extend`

extend anomalies, baseline, score

Stage 3: `mv-expand`

mv-expand Total

Stage 4: `where`

where anomalies > 0

Stage 5: `extend`

extend AnomalyHour, score

Stage 6: `project`

project AnomalyHour, DeviceVendor, TimeGenerated, Total, anomalies, baseline, score

Stage 7: `where`

where ...

Stage 8: `join`

join (...)

Stage 9: `extend`

extend PercentTotal

Stage 10: `where`

where PercentTotal > 50

The stages below run on TimeSeriesAlerts (the outer pipeline).

Stage 11: `project`

project AnomalyHour, DestinationIPlist, DestinationPortlist, DeviceVendor, HourlyCount, PercentTotal, SourceIP, TimeGeneratedMax, Total, anomalies, baseline, score

Stage 12: `summarize`

summarize DestinationIPlist, DestinationPortlist, EndTimeUtc, HourlyCount, SourceIPMax, SourceIPlist, StartTimeUtc by DeviceVendor, AnomalyHour, Total, baseline, score, anomalies

Stage 13: `project`

project AnomalyHour, DestinationIPlist, DestinationPortlist, DeviceVendor, EndTimeUtc, HourlyCount, SourceIPMax, SourceIPlist, Total, anomalies, baseline, score

Indicators

Each row is a field, operator, and value that the rule matches. The corpus column counts how many other rules in the catalog look for the same combination: high numbers point to widely-used, community-vetted indicators. Blank or 1 shows that the indicator is specific to this rule.

Field	Kind	Values
`DestinationIP`	is_not_null	(no value, null check)
`PercentTotal`	gt	`50` transforms: `cased`
`SourceIP`	is_not_null	(no value, null check)
`anomalies`	gt	`0` transforms: `cased`

Output fields

Fields the rule emits when it matches. Chronicle authors list these in the outcome block; they appear on the detection and $risk_score drives alerting. Sentinel / Defender XDR rules build them up through project / summarize / extend stages. Sentinel maps these into alert fields via entityMappings and customDetails; Defender XDR custom detections surface them as alert fields directly.

Field	Source
`AnomalyHour`	`project`
`DestinationIPlist`	`project`
`DestinationPortlist`	`project`
`DeviceVendor`	`project`
`EndTimeUtc`	`project`
`HourlyCount`	`project`
`SourceIPMax`	`project`
`SourceIPlist`	`project`
`Total`	`project`
`anomalies`	`project`
`baseline`	`project`
`score`	`project`

`j` / `k`	Scroll down / up
`d` / `u`	Half-page down / up
`gg` / `G`	Top / bottom
`h` / `l`	History back / forward
`f`	Follow link (`Shift` = new tab)
`/`	Focus search
`?`	Toggle this help
`↑` / `↓`	Navigate search results
`Enter`	Open highlighted result
`Esc`	Close results / dialog

`type:`	`events` / `rules` / `providers`
`vendor:`	`sigma` / `elastic` / `splunk` / `kusto` / `chronicle` (vendor name alone also works: `sigma:`, `kql:`, `secops:`…)
`tactic:`	TA-id, slug, or name: `credential_access`, `TA0006`
`technique:`	technique or sub-technique ID: `T1003`, `T1003.001` (alias `tech:`)
`severity:`	`critical` / `high` / `medium` / `low` / `informational` (alias `sev:`)
`risk_score`	Numeric comparison on the Elastic risk score (0 to 100): `risk_score>50`, `risk_score<=20`, `risk_score=99` (alias `risk`; Elastic rules only)
`stages:`	Rules with exactly N pipeline stages
`correlation:`	`single_event` / `sequence` / `alternatives` / `alternatives_cross_log` / `all_required` / `correlated`
`with:`	Co-occurrence event-id; stacks (`with:4624 with:4769`) to require all, while a comma list in one occurrence (`with:4624,4769`) is an either-or group. Implies multi-event
`like:`	Structural neighbors of a rule slug (equivalents + subsumption stricter / broader): `like:comsvcs_lsass_memory_dump-splunk-sysmon`
`groupby:`	Entity-grouping substring match against `group_by_keys`: `groupby:user`, `groupby:host`
`uses:`	Rules whose predicate tree touches the field (any kind, any value): `uses:CommandLine`
`excludes:`	Rules with top-level `not()` clauses on the field (FP whitelists): `excludes:ParentImage`
`field:` / `value:`	Predicate search; narrows rule cards to those with a matching leaf and drives the indicator tier. Unquoted = substring, wildcards allowed (`value:mimikatz`)
`indicator:`	Shorthand for `field:F value:V`: `indicator:Image=*\powershell.exe`
`kind:`	Filter by predicate kind. Narrows rule cards to those carrying a matching predicate leaf (`vendor:elastic kind:cidr_match`) and drives the indicator tier: `contains` / `starts_with` / `ends_with` / `regex` / `cidr` / `eq` / `in` … (operator aliases `op:`/`match:`)
`has:` / `no:`	`sample`, `field`, `notes`, `refs`, `trace`, `thirdparty`, `rule`, `pattern`, `timewindow`, `threshold`, `newterms`, `sigma`/`elastic`/`splunk`/`kusto`/`chronicle`
`-op:val`	Exclude matches; works on most operators but not `type:`/`like:`/`has:`/`no:` (use `no:<flag>` to exclude a rule flag): `tactic:execution -vendor:splunk`. Standalone `-kind:`/`-field:`/`-value:` drop every rule carrying a matching predicate leaf (`type:rules -kind:is_null`)
`field:"…"` / `value:"…"`	Quoted value = anchored exact match (also allows spaces): `value:"net user"`
`a,b`	Comma = OR inside one operator (`vendor:sigma,elastic`, `severity:high,critical`); repeating a facet merges the same way. `field:`/`value:` never split (literal commas)
`vendors:` / `stage:`	Singular and plural spellings fold to the canonical operator and value: `tactics:` = `tactic:`, `type:event` = `type:events`, `correlation:sequences` = `correlation:sequence`, `has:thresholds` = `has:threshold`
`"quoted phrase"`	Exact-match a multi-word phrase (free text)

Time series anomaly detection for total volume of traffic

MITRE ATT&CK coverage

Rule body kusto

Stages and Predicates

Parameters

Let binding: `AnomalyHours`

Stage 1: `source`

Stage 2: `extend`

Stage 3: `mv-expand`

Stage 4: `where`

Stage 5: `extend`

Stage 6: `project`

Stage 7: `where`

Stage 8: `join`

Stage 9: `extend`

Stage 10: `where`

Stage 11: `project`

Stage 12: `summarize`

Stage 13: `project`

Indicators

Output fields

Keyboard shortcuts

Search operators

Time series anomaly detection for total volume of traffic

MITRE ATT&CK coverage

Rule body kusto

Stages and Predicates

Parameters

Let binding: AnomalyHours

Stage 1: source

Stage 2: extend

Stage 3: mv-expand

Stage 4: where

Stage 5: extend

Stage 6: project

Stage 7: where

Stage 8: join

Stage 9: extend

Stage 10: where

Stage 11: project

Stage 12: summarize

Stage 13: project

Indicators

Output fields

Let binding: `AnomalyHours`

Stage 1: `source`

Stage 2: `extend`

Stage 3: `mv-expand`

Stage 4: `where`

Stage 5: `extend`

Stage 6: `project`

Stage 7: `where`

Stage 8: `join`

Stage 9: `extend`

Stage 10: `where`

Stage 11: `project`

Stage 12: `summarize`

Stage 13: `project`