GCP BigQuery Large Scan

Severity: informational
Entities: emails, ip_addresses
Log types: GCP.AuditLog
Reference: https://cloud.google.com/bigquery/docs/running-queries
Source: github.com/panther-labs/panther-analysis

Detect any BigQuery query that is doing a very large scan (> 1 GB).

Rule body yaml

AnalysisType: rule
Description: Detect any BigQuery query that is doing a very large scan (> 1 GB).
DisplayName: "GCP BigQuery Large Scan"
Enabled: true
Filename: gcp_bigquery_large_scan.py
Reference: https://cloud.google.com/bigquery/docs/running-queries
Severity: Info
Tests:
  - ExpectedResult: false
    Log:
      insertid: ABCDEFGHIJKL
      logname: projects/gcp-project1/logs/cloudaudit.googleapis.com%2Fdata_access
      operation:
        id: 0123456789012-gcp-project1:abcdefg_hijklmnop_1234567abcd
        last: true
        producer: bigquery.googleapis.com
      p_any_emails:
        - user@company.io
      p_any_ip_addresses:
        - 1.2.3.4
      p_event_time: "2023-03-28 17:37:02.096"
      p_log_type: GCP.AuditLog
      p_parse_time: "2023-03-28 17:38:14.621"
      p_row_id: de00d3dcdaeee4b4d5f7fa9d17b4c203
      p_schema_version: 0
      p_source_id: 964c7894-9a0d-4ddf-864f-0193438221d6
      p_source_label: gcp-logsource
      protoPayload:
        at_sign_type: type.googleapis.com/google.cloud.audit.AuditLog
        authenticationInfo:
          principalEmail: user@company.io
        authorizationInfo:
          - granted: true
            permission: bigquery.jobs.create
            resource: projects/gcp-project1
        metadata:
          "@type": type.googleapis.com/google.cloud.audit.BigQueryAuditMetadata
          jobChange:
            after: DONE
            job:
              jobConfig:
                queryConfig:
                  createDisposition: CREATE_IF_NEEDED
                  destinationTable: projects/gcp-project1/datasets/_c2b49f742788f188022fcec1f1622e7404b40ce5/tables/anondea8e77183dcdf1cf93a47b5003036409a525808207450ab80db17fcc8cdcf53
                  priority: QUERY_INTERACTIVE
                  query: |-
                    -- This query shows a list of the daily top Google Search terms.
                    SELECT
                       *
                    FROM `bigquery-public-data.google_trends.top_terms`
                  statementType: SELECT
                  writeDisposition: WRITE_TRUNCATE
                type: QUERY
              jobName: projects/gcp-project1/jobs/abcdefg_hijklmnop_1234567abcd
              jobStats:
                createTime: "2023-03-28T17:36:49.087Z"
                endTime: "2023-03-28T17:37:02.092Z"
                queryStats:
                  billingTier: 1
                  outputRowCount: "43683701"
                  referencedTables:
                    - projects/bigquery-public-data/datasets/google_trends/tables/top_terms
                  totalBilledBytes: "100000"
                  totalProcessedBytes: "100000"
                startTime: "2023-03-28T17:36:49.224Z"
                totalSlotMs: "259581"
              jobStatus:
                jobState: DONE
        methodName: google.cloud.bigquery.v2.JobService.InsertJob
        requestMetadata:
          callerIP: 1.2.3.4
          callerSuppliedUserAgent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36,gzip(gfe),gzip(gfe)
        resourceName: projects/gcp-project1/jobs/abcdefg_hijklmnop_1234567abcd
        serviceName: bigquery.googleapis.com
        status: {}
      receivetimestamp: "2023-03-28 17:37:02.114"
      resource:
        labels:
          location: US
          project_id: gcp-project1
        type: bigquery_project
      severity: INFO
      timestamp: "2023-03-28 17:37:02.096"
    Name: small query
  - ExpectedResult: true
    Log:
      insertid: ABCDEFGHIJKL
      logname: projects/gcp-project1/logs/cloudaudit.googleapis.com%2Fdata_access
      operation:
        id: 0123456789012-gcp-project1:abcdefg_hijklmnop_1234567abcd
        last: true
        producer: bigquery.googleapis.com
      p_any_emails:
        - user@company.io
      p_any_ip_addresses:
        - 1.2.3.4
      p_event_time: "2023-03-28 17:37:02.096"
      p_log_type: GCP.AuditLog
      p_parse_time: "2023-03-28 17:38:14.621"
      p_row_id: de00d3dcdaeee4b4d5f7fa9d17b4c203
      p_schema_version: 0
      p_source_id: 964c7894-9a0d-4ddf-864f-0193438221d6
      p_source_label: gcp-logsource
      protoPayload:
        at_sign_type: type.googleapis.com/google.cloud.audit.AuditLog
        authenticationInfo:
          principalEmail: user@company.io
        authorizationInfo:
          - granted: true
            permission: bigquery.jobs.create
            resource: projects/gcp-project1
        metadata:
          "@type": type.googleapis.com/google.cloud.audit.BigQueryAuditMetadata
          jobChange:
            after: DONE
            job:
              jobConfig:
                queryConfig:
                  createDisposition: CREATE_IF_NEEDED
                  destinationTable: projects/gcp-project1/datasets/_c2b49f742788f188022fcec1f1622e7404b40ce5/tables/anondea8e77183dcdf1cf93a47b5003036409a525808207450ab80db17fcc8cdcf53
                  priority: QUERY_INTERACTIVE
                  query: |-
                    -- This query shows a list of the daily top Google Search terms.
                    SELECT
                       *
                    FROM `bigquery-public-data.google_trends.top_terms`
                  statementType: SELECT
                  writeDisposition: WRITE_TRUNCATE
                type: QUERY
              jobName: projects/gcp-project1/jobs/abcdefg_hijklmnop_1234567abcd
              jobStats:
                createTime: "2023-03-28T17:36:49.087Z"
                endTime: "2023-03-28T17:37:02.092Z"
                queryStats:
                  billingTier: 1
                  outputRowCount: "43683701"
                  referencedTables:
                    - projects/bigquery-public-data/datasets/google_trends/tables/top_terms
                  totalBilledBytes: "3097493504"
                  totalProcessedBytes: "3096864198"
                startTime: "2023-03-28T17:36:49.224Z"
                totalSlotMs: "259581"
              jobStatus:
                jobState: DONE
        methodName: google.cloud.bigquery.v2.JobService.InsertJob
        requestMetadata:
          callerIP: 1.2.3.4
          callerSuppliedUserAgent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36,gzip(gfe),gzip(gfe)
        resourceName: projects/gcp-project1/jobs/abcdefg_hijklmnop_1234567abcd
        serviceName: bigquery.googleapis.com
        status: {}
      receivetimestamp: "2023-03-28 17:37:02.114"
      resource:
        labels:
          location: US
          project_id: gcp-project1
        type: bigquery_project
      severity: INFO
      timestamp: "2023-03-28 17:37:02.096"
    Name: Large Query
DedupPeriodMinutes: 60
LogTypes:
  - GCP.AuditLog
RuleID: "GCP.BigQuery.Large.Scan"
Threshold: 1

Detection logic

Condition

resource.type starts_with "bigquery"
operation.last eq "True"
protoPayload.metadata.jobChange.job.jobConfig.type eq "QUERY"
protoPayload.metadata.jobChange.job.jobConfig.queryConfig.statementType eq "SELECT"
protoPayload.metadata.jobChange.job.jobStats.queryStats.totalBilledBytes gt "1073741824"

Indicators

Each row is a field, operator, and value that the rule matches. The corpus column counts how many other rules in the catalog look for the same combination: high numbers point to widely-used, community-vetted indicators. Blank or 1 shows that the indicator is specific to this rule.

Field	Kind	Values
`operation.last`	eq	`True`
`protoPayload.metadata.jobChange.job.jobConfig.queryConfig.statementType`	eq	`SELECT`
`protoPayload.metadata.jobChange.job.jobConfig.type`	eq	`QUERY`
`protoPayload.metadata.jobChange.job.jobStats.queryStats.totalBilledBytes`	gt	`1073741824`
`resource.type`	starts_with	`bigquery`

Output fields

Fields the rule emits when it matches. Chronicle authors list these in the outcome block; they appear on the detection and $risk_score drives alerting. Sentinel / Defender XDR rules build them up through project / summarize / extend stages. Sentinel maps these into alert fields via entityMappings and customDetails; Defender XDR custom detections surface them as alert fields directly.

Field	Source
`query`	`protoPayload.metadata.jobChange.job.jobConfig.queryConfig.query`
`actor`	`protoPayload.authenticationInfo.principalEmail`
`query_size`	`protoPayload.metadata.jobChange.job.jobStats.queryStats.totalBilledBytes`

`j` / `k`	Scroll down / up
`d` / `u`	Half-page down / up
`gg` / `G`	Top / bottom
`h` / `l`	History back / forward
`f`	Follow link (`Shift` = new tab)
`/`	Focus search
`?`	Toggle this help
`↑` / `↓`	Navigate search results
`Enter`	Open highlighted result
`Esc`	Close results / dialog

`type:`	`events` / `rules` / `providers`
`vendor:`	`sigma` / `elastic` / `splunk` / `kusto` / `chronicle` (vendor name alone also works: `sigma:`, `kql:`, `secops:`…)
`tactic:`	TA-id, slug, or name: `credential_access`, `TA0006`
`technique:`	technique or sub-technique ID: `T1003`, `T1003.001` (alias `tech:`)
`severity:`	`critical` / `high` / `medium` / `low` / `informational` (alias `sev:`)
`risk_score`	Numeric comparison on the Elastic risk score (0 to 100): `risk_score>50`, `risk_score<=20`, `risk_score=99` (alias `risk`; Elastic rules only)
`stages:`	Rules with exactly N pipeline stages
`correlation:`	`single_event` / `sequence` / `alternatives` / `alternatives_cross_log` / `all_required` / `correlated`
`with:`	Co-occurrence event-id; stacks (`with:4624 with:4769`) to require all, while a comma list in one occurrence (`with:4624,4769`) is an either-or group. Implies multi-event
`like:`	Structural neighbors of a rule slug (equivalents + subsumption stricter / broader): `like:comsvcs_lsass_memory_dump-splunk-sysmon`
`groupby:`	Entity-grouping substring match against `group_by_keys`: `groupby:user`, `groupby:host`
`uses:`	Rules whose predicate tree touches the field (any kind, any value): `uses:CommandLine`
`excludes:`	Rules with top-level `not()` clauses on the field (FP whitelists): `excludes:ParentImage`
`field:` / `value:`	Predicate search; narrows rule cards to those with a matching leaf and drives the indicator tier. Unquoted = substring, wildcards allowed (`value:mimikatz`)
`indicator:`	Shorthand for `field:F value:V`: `indicator:Image=*\powershell.exe`
`kind:`	Filter by predicate kind. Narrows rule cards to those carrying a matching predicate leaf (`vendor:elastic kind:cidr_match`) and drives the indicator tier: `contains` / `starts_with` / `ends_with` / `regex` / `cidr` / `eq` / `in` … (operator aliases `op:`/`match:`)
`has:` / `no:`	`sample`, `field`, `notes`, `refs`, `trace`, `thirdparty`, `rule`, `pattern`, `timewindow`, `threshold`, `newterms`, `sigma`/`elastic`/`splunk`/`kusto`/`chronicle`
`-op:val`	Exclude matches; works on most operators but not `type:`/`like:`/`has:`/`no:` (use `no:<flag>` to exclude a rule flag): `tactic:execution -vendor:splunk`. Standalone `-kind:`/`-field:`/`-value:` drop every rule carrying a matching predicate leaf (`type:rules -kind:is_null`)
`field:"…"` / `value:"…"`	Quoted value = anchored exact match (also allows spaces): `value:"net user"`
`a,b`	Comma = OR inside one operator (`vendor:sigma,elastic`, `severity:high,critical`); repeating a facet merges the same way. `field:`/`value:` never split (literal commas)
`vendors:` / `stage:`	Singular and plural spellings fold to the canonical operator and value: `tactics:` = `tactic:`, `type:event` = `type:events`, `correlation:sequences` = `correlation:sequence`, `has:thresholds` = `has:threshold`
`"quoted phrase"`	Exact-match a multi-word phrase (free text)