Skip to content

Commit

Permalink
[Azure] Update Azure Firewall pipeline (#9428)
Browse files Browse the repository at this point in the history
  • Loading branch information
@lucian-ioan
lucian-ioan authored Jul 11, 2024
1 parent 78f3eae commit 55e5f4f
Show file tree
Hide file tree
Showing 15 changed files with 657 additions and 46 deletions.
18 changes: 13 additions & 5 deletions packages/azure/_dev/build/docs/firewall_logs.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,19 @@ Azure Firewall Logs are records of events such as network and application rules

Supported log categories:

| Log Category | Description |
|:----------------------------:|:------------------------------------------------------------------------------------------------------------------------------------:|
| AzureFirewallApplicationRule | These logs capture information about the traffic that is allowed or denied by application rules configured in Azure Firewall. |
| AzureFirewallNetworkRule | These logs capture information about the traffic that is allowed or denied by network rules configured in Azure Firewall. |
| AzureFirewallDnsProxy | These logs capture information about DNS requests and responses that are processed by Azure Firewall's DNS proxy. |
| Log Category | Description | Destination Table |
|:----------------------------:|:------------------------------------------------------------------------------------------------------------------------------------:|:------------------:|
| AzureFirewallApplicationRule | These logs capture information about the traffic that is allowed or denied by application rules configured in Azure Firewall. | Azure diagnostics |
| AzureFirewallNetworkRule | These logs capture information about the traffic that is allowed or denied by network rules configured in Azure Firewall. | Azure diagnostics |
| AzureFirewallDnsProxy | These logs capture information about DNS requests and responses that are processed by Azure Firewall's DNS proxy. | Azure diagnostics |
| AZFWApplicationRule | These logs capture resource specific information about the traffic that is allowed or denied by application rules configured in Azure Firewall. | Resource specific |
| AZFWNetworkRule | These logs capture resource specific information about the traffic that is allowed or denied by network rules configured in Azure Firewall. | Resource specific |
| AZFWNatRule | These logs capture resource specific information about all DNAT (Destination Network Address Translation) events log data. | Resource specific |
| AZFWDnsQuery | These logs capture resource specific information about DNS requests and responses that are processed by Azure Firewall's DNS proxy. | Resource specific |

For detailed information and instructions on how to migrate to Resource-specific mode, please refer to the following Microsoft documentation: [Azure Monitor Resource Logs](https://learn.microsoft.com/en-gb/azure/azure-monitor/essentials/resource-logs#resource-specific).

All Azure services will eventually use the resource-specific mode. As part of this transition, some resources allow you to select a mode in the diagnostic setting. Specify resource-specific mode for any new diagnostic settings because this mode makes the data easier to manage.

## Requirements and setup

Expand Down
6 changes: 6 additions & 0 deletions packages/azure/changelog.yml
View file
Original file line number Diff line number Diff line change
@@ -1,3 +1,9 @@

- version: "1.13.0"
changes:
- description: Add structured log categories to Azure Firewall.
type: enhancement
link: https://github.com/elastic/integrations/pull/9428
- version: "1.12.0"
changes:
- description: ECS version updated to 8.11.0. Update the kibana constraint to ^8.13.0. Modified the field definitions to remove ECS fields made redundant by the ecs@mappings component template.
Expand Down
1 change: 1 addition & 0 deletions ...ure/data_stream/firewall_logs/_dev/test/pipeline/test-applicationrules-structured-raw.log
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"category":"AZFWApplicationRule","properties":{"Action":"Allow","ActionReason":"reason","DestinationIp":"1.128.0.0","DestinationPort": 123,"Fqdn":"","IsExplicitProxyRequest":false,"IsTlsInspected":false,"Policy":"policy","Protocol":"HTTP","Rule":"ApplicationRule","RuleCollection":"ApplicationRuleSet","RuleCollectionGroup":"ApplicationRuleGroup","SourceIp":"1.128.0.0","SourcePort":1234,"TargetUrl":"https://www.microsoft.com/en-us/about","WebCategory":"category"},"resourceId":"/SUBSCRIPTIONS/23103928-B2CF-472A-8CDB-0146E2849129/RESOURCEGROUPS/TEST-FW-RG/PROVIDERS/MICROSOFT.NETWORK/AZUREFIREWALLS/TEST-FW01","time":"2024-03-20T23:39:59.8494370Z"}
92 changes: 92 additions & 0 deletions ...m/firewall_logs/_dev/test/pipeline/test-applicationrules-structured-raw.log-expected.json
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
{
"expected": [
{
"@timestamp": "2024-03-20T23:39:59.849Z",
"azure": {
"firewall": {
"category": "AZFWApplicationRule",
"is_explicit_proxy_request": false,
"is_tls_inspected": false,
"policy": "policy",
"rule_collection_group": "ApplicationRuleGroup",
"web_category": "category"
},
"resource": {
"group": "TEST-FW-RG",
"id": "/SUBSCRIPTIONS/23103928-B2CF-472A-8CDB-0146E2849129/RESOURCEGROUPS/TEST-FW-RG/PROVIDERS/MICROSOFT.NETWORK/AZUREFIREWALLS/TEST-FW01",
"name": "TEST-FW01",
"provider": "MICROSOFT.NETWORK/AZUREFIREWALLS"
},
"subscription_id": "23103928-B2CF-472A-8CDB-0146E2849129"
},
"cloud": {
"account": {
"id": "23103928-B2CF-472A-8CDB-0146E2849129"
},
"provider": "azure"
},
"destination": {
"as": {
"number": 1221,
"organization": {
"name": "Telstra Pty Ltd"
}
},
"ip": "1.128.0.0",
"port": 123
},
"ecs": {
"version": "8.11.0"
},
"event": {
"action": "Allow",
"category": [
"network"
],
"kind": "event",
"original": "{\"category\":\"AZFWApplicationRule\",\"properties\":{\"Action\":\"Allow\",\"ActionReason\":\"reason\",\"DestinationIp\":\"1.128.0.0\",\"DestinationPort\": 123,\"Fqdn\":\"\",\"IsExplicitProxyRequest\":false,\"IsTlsInspected\":false,\"Policy\":\"policy\",\"Protocol\":\"HTTP\",\"Rule\":\"ApplicationRule\",\"RuleCollection\":\"ApplicationRuleSet\",\"RuleCollectionGroup\":\"ApplicationRuleGroup\",\"SourceIp\":\"1.128.0.0\",\"SourcePort\":1234,\"TargetUrl\":\"https://www.microsoft.com/en-us/about\",\"WebCategory\":\"category\"},\"resourceId\":\"/SUBSCRIPTIONS/23103928-B2CF-472A-8CDB-0146E2849129/RESOURCEGROUPS/TEST-FW-RG/PROVIDERS/MICROSOFT.NETWORK/AZUREFIREWALLS/TEST-FW01\",\"time\":\"2024-03-20T23:39:59.8494370Z\"}",
"reason": "reason",
"type": [
"connection"
]
},
"network": {
"protocol": "http"
},
"observer": {
"name": "TEST-FW01",
"product": "Network Firewall",
"type": "firewall",
"vendor": "Azure"
},
"related": {
"ip": [
"1.128.0.0"
]
},
"rule": {
"name": "ApplicationRule",
"ruleset": "ApplicationRuleSet"
},
"source": {
"as": {
"number": 1221,
"organization": {
"name": "Telstra Pty Ltd"
}
},
"ip": "1.128.0.0",
"port": 1234
},
"tags": [
"preserve_original_event"
],
"url": {
"domain": "www.microsoft.com",
"original": "https://www.microsoft.com/en-us/about",
"path": "/en-us/about",
"scheme": "https"
}
}
]
}
1 change: 1 addition & 0 deletions packages/azure/data_stream/firewall_logs/_dev/test/pipeline/test-dnsproxy-structured-raw.log
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"category":"AZFWDnsQuery","properties":{"DnssecOkBit":false,"EDNS0BufferSize":512,"ErrorMessage":"","ErrorNumber":0,"Protocol":"udp","QueryClass":"IN","QueryId":35817,"QueryName":"ntp.ubuntu.com.","QueryType":"A","RequestDurationSecs":0.0000286,"RequestSize":32,"ResponseCode":"NOERROR","ResponseFlags":"qr,aa,rd,ra","ResponseSize":152,"SourceIp":"1.128.0.0","SourcePort":47785},"resourceId":"/SUBSCRIPTIONS/56D199A8-724E-436D-B7F2-5F6F7694EABD/RESOURCEGROUPS/PROD-WESTUS3/PROVIDERS/MICROSOFT.NETWORK/AZUREFIREWALLS/PROD-WESTUS3","time":"2024-01-31T23:39:50.8623870Z"}
93 changes: 93 additions & 0 deletions ...ta_stream/firewall_logs/_dev/test/pipeline/test-dnsproxy-structured-raw.log-expected.json
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
{
"expected": [
{
"@timestamp": "2024-01-31T23:39:50.862Z",
"azure": {
"firewall": {
"category": "AZFWDnsQuery",
"dnssec_ok_bit": false,
"edns0_buffer_size": 512,
"request_duration_secs": 2.86E-5,
"request_size": 32,
"response_size": 152
},
"resource": {
"group": "PROD-WESTUS3",
"id": "/SUBSCRIPTIONS/56D199A8-724E-436D-B7F2-5F6F7694EABD/RESOURCEGROUPS/PROD-WESTUS3/PROVIDERS/MICROSOFT.NETWORK/AZUREFIREWALLS/PROD-WESTUS3",
"name": "PROD-WESTUS3",
"provider": "MICROSOFT.NETWORK/AZUREFIREWALLS"
},
"subscription_id": "56D199A8-724E-436D-B7F2-5F6F7694EABD"
},
"cloud": {
"account": {
"id": "56D199A8-724E-436D-B7F2-5F6F7694EABD"
},
"provider": "azure"
},
"dns": {
"header_flags": [
"QR",
"AA",
"RD",
"RA"
],
"id": "35817",
"question": {
"class": "IN",
"name": "ntp.ubuntu.com.",
"type": "A"
},
"response_code": "NOERROR",
"type": "query"
},
"ecs": {
"version": "8.11.0"
},
"error": {
"id": "0"
},
"event": {
"category": [
"network"
],
"kind": "event",
"original": "{\"category\":\"AZFWDnsQuery\",\"properties\":{\"DnssecOkBit\":false,\"EDNS0BufferSize\":512,\"ErrorMessage\":\"\",\"ErrorNumber\":0,\"Protocol\":\"udp\",\"QueryClass\":\"IN\",\"QueryId\":35817,\"QueryName\":\"ntp.ubuntu.com.\",\"QueryType\":\"A\",\"RequestDurationSecs\":0.0000286,\"RequestSize\":32,\"ResponseCode\":\"NOERROR\",\"ResponseFlags\":\"qr,aa,rd,ra\",\"ResponseSize\":152,\"SourceIp\":\"1.128.0.0\",\"SourcePort\":47785},\"resourceId\":\"/SUBSCRIPTIONS/56D199A8-724E-436D-B7F2-5F6F7694EABD/RESOURCEGROUPS/PROD-WESTUS3/PROVIDERS/MICROSOFT.NETWORK/AZUREFIREWALLS/PROD-WESTUS3\",\"time\":\"2024-01-31T23:39:50.8623870Z\"}",
"type": [
"connection"
]
},
"network": {
"iana_number": "17",
"transport": "udp"
},
"observer": {
"name": "PROD-WESTUS3",
"product": "Network Firewall",
"type": "firewall",
"vendor": "Azure"
},
"related": {
"hosts": [
"ntp.ubuntu.com."
],
"ip": [
"1.128.0.0"
]
},
"source": {
"as": {
"number": 1221,
"organization": {
"name": "Telstra Pty Ltd"
}
},
"ip": "1.128.0.0",
"port": 47785
},
"tags": [
"preserve_original_event"
]
}
]
}
1 change: 1 addition & 0 deletions packages/azure/data_stream/firewall_logs/_dev/test/pipeline/test-natrule-structured-raw.log
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"category":"AZFWNatRule","properties":{"DestinationIp":"1.128.0.0","DestinationPort":999,"Policy":"policy","Protocol":"TCP","Rule":"NATSecurityRule","RuleCollection":"NATRuleSet","RuleCollectionGroup":"NATRuleGroup","SourceIp":"1.128.0.0","SourcePort":1234,"TranslatedIp":"1.128.0.0","TranslatedPort":999},"resourceId":"/SUBSCRIPTIONS/23103928-B2CF-472A-8CDB-0146E2849129/RESOURCEGROUPS/TEST-FW-RG/PROVIDERS/MICROSOFT.NETWORK/AZUREFIREWALLS/TEST-FW01","time":"2024-01-31T23:39:49.6798940Z"}
82 changes: 82 additions & 0 deletions ...ata_stream/firewall_logs/_dev/test/pipeline/test-natrule-structured-raw.log-expected.json
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
{
"expected": [
{
"@timestamp": "2024-01-31T23:39:49.679Z",
"azure": {
"firewall": {
"category": "AZFWNatRule",
"policy": "policy",
"rule_collection_group": "NATRuleGroup"
},
"resource": {
"group": "TEST-FW-RG",
"id": "/SUBSCRIPTIONS/23103928-B2CF-472A-8CDB-0146E2849129/RESOURCEGROUPS/TEST-FW-RG/PROVIDERS/MICROSOFT.NETWORK/AZUREFIREWALLS/TEST-FW01",
"name": "TEST-FW01",
"provider": "MICROSOFT.NETWORK/AZUREFIREWALLS"
},
"subscription_id": "23103928-B2CF-472A-8CDB-0146E2849129"
},
"cloud": {
"account": {
"id": "23103928-B2CF-472A-8CDB-0146E2849129"
},
"provider": "azure"
},
"destination": {
"as": {
"number": 1221,
"organization": {
"name": "Telstra Pty Ltd"
}
},
"ip": "1.128.0.0",
"port": 999
},
"ecs": {
"version": "8.11.0"
},
"event": {
"category": [
"network"
],
"kind": "event",
"original": "{\"category\":\"AZFWNatRule\",\"properties\":{\"DestinationIp\":\"1.128.0.0\",\"DestinationPort\":999,\"Policy\":\"policy\",\"Protocol\":\"TCP\",\"Rule\":\"NATSecurityRule\",\"RuleCollection\":\"NATRuleSet\",\"RuleCollectionGroup\":\"NATRuleGroup\",\"SourceIp\":\"1.128.0.0\",\"SourcePort\":1234,\"TranslatedIp\":\"1.128.0.0\",\"TranslatedPort\":999},\"resourceId\":\"/SUBSCRIPTIONS/23103928-B2CF-472A-8CDB-0146E2849129/RESOURCEGROUPS/TEST-FW-RG/PROVIDERS/MICROSOFT.NETWORK/AZUREFIREWALLS/TEST-FW01\",\"time\":\"2024-01-31T23:39:49.6798940Z\"}",
"type": [
"connection"
]
},
"network": {
"iana_number": "6",
"transport": "tcp"
},
"observer": {
"name": "TEST-FW01",
"product": "Network Firewall",
"type": "firewall",
"vendor": "Azure"
},
"related": {
"ip": [
"1.128.0.0"
]
},
"rule": {
"name": "NATSecurityRule",
"ruleset": "NATRuleSet"
},
"source": {
"as": {
"number": 1221,
"organization": {
"name": "Telstra Pty Ltd"
}
},
"ip": "1.128.0.0",
"port": 1234
},
"tags": [
"preserve_original_event"
]
}
]
}
1 change: 1 addition & 0 deletions ...es/azure/data_stream/firewall_logs/_dev/test/pipeline/test-networkrule-structured-raw.log
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"category":"AZFWNetworkRule","properties":{"Action":"Allow","ActionReason":"action reason","DestinationIp":"1.128.0.0","DestinationPort":1234,"Policy":"policy","Protocol":"TCP","Rule":"NetworkSecurityRule","RuleCollection":"NetworkRuleSet","RuleCollectionGroup":"NetworkRuleGroup","SourceIp":"1.128.0.0","SourcePort":1234},"resourceId":"/SUBSCRIPTIONS/23103928-B2CF-472A-8CDB-0146E2849129/RESOURCEGROUPS/TEST-FW-RG/PROVIDERS/MICROSOFT.NETWORK/AZUREFIREWALLS/TEST-FW01","time":"2024-03-20T23:39:59.8494370Z"}
Loading

0 comments on commit 55e5f4f

Please sign in to comment.