Migrate TI provider to new ThreatIntelIndicators table schema

[Copilot]

Microsoft Sentinel deprecated the ThreatIntelligenceIndicator table in favor of ThreatIntelIndicators with STIX 2.1 schema. The old table stops ingesting data August 2025, causing TI lookups to return zero results.

Query Schema Migration

Updated all KQL queries to use STIX observable pattern matching:

# Old schema - specific column names
| where NetworkIP in ({ip_address_list})
  or NetworkDestinationIP in ({ip_address_list})
  or NetworkSourceIP in ({ip_address_list})

# New schema - STIX observable keys
| where ObservableKey in ("ipv4-addr:value", "ipv6-addr:value")
  and ObservableValue in ({ip_address_list})

Observable key mappings:

• IPs: ipv4-addr:value, ipv6-addr:value
• Hashes: file:hashes.MD5, file:hashes.SHA-1, file:hashes.SHA-256
• Domains: domain-name:value
• URLs: url:value
• Email: email-addr:value
• File paths: file:name, directory:path

Changed unique identifier from IndicatorId to Id for deduplication.

Provider Backward Compatibility

Enhanced result parsing to handle both schemas:

# Checks old schema fields first (Active, ConfidenceScore, ThreatType)
# Falls back to new schema (IsActive, Confidence, Data.labels)
if "ThreatType" in result_series.index:
    extracted_data["ThreatType"] = result_series["ThreatType"]
elif "Data" in result_series.index and pd.notna(result_series["Data"]):
    stix_data = result_series["Data"]
    if isinstance(stix_data, dict):
        labels = stix_data.get("labels", [])
        extracted_data["ThreatType"] = labels[0] if labels else "unknown"

Extracted confidence threshold to class constant _HIGH_CONFIDENCE_THRESHOLD = 80.

Files Changed

• kql_sent_threatintel.yaml: Table name and query patterns for all 7 IOC types
• azure_sent_byoti.py: Table reference, result parsing with dual schema support
• test_tiprovider_kql.py: Mock driver updated for new query patterns

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• learn.microsoft.com
  • Triggering command: /home/REDACTED/work/_temp/ghcca-node/node/bin/node /home/REDACTED/work/_temp/ghcca-node/node/bin/node --enable-source-maps /home/REDACTED/work/_temp/copilot-developer-action-main/dist/index.js (dns block)
• publicsuffix.org
  • Triggering command: /usr/bin/python python -m pytest tests/context/test_tiprovider_kql.py -v tem ]; then /usr/lib/php/sessionclean; fi (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

Original prompt

---

This section details on the original issue you should resolve

<issue_title>[Bug]: TI Lookups using Microsoft Sentinel provider failing due to new table and schema in Sentinel</issue_title>
<issue_description>Describe the bug
The TI provider (and built in queries) for Sentinel Threat Intel use the old ThreatIntelligenceIndicator table instead of the new ThreatIntelIndicators. As the old table is no longer being populated with TI data, it will return 0 results even if there is matching TI stored in Sentinel

To Reproduce
Steps to reproduce the behavior:

1. Create a piece of TI in Sentinel
2. Use IpAddress.tilookup_ip() to try and find it
3. Get 0 results

Expected behavior
Matching TI from Sentinel should be returned

Additional context
https://learn.microsoft.com/en-us/azure/sentinel/work-with-threat-indicators?tabs=defender-portal#find-and-view-threat-intelligence-with-queries
</issue_description>

<agent_instructions>Check the queries used for TI lookups and adjust them to the new schema</agent_instructions>

Comments on the Issue (you are @copilot in this section)

• Fixes [Bug]: TI Lookups using Microsoft Sentinel provider failing due to new table and schema in Sentinel #855

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.