Bug #9149
Updated by Aneesh Patel 29 days ago
# Performance regression on firewall mode as opposed to system/IPS mode When This references V1 vs V2 policies - V1 policy refers to suricata in system mode and V2 policy refers to suricata in firewall mode. The two policies compared were a **maxed V2 policy** and a **V1 (classic stateful) control policy**. Both were attached to the same firewall (flipped via `associate-firewall-policy`) so the only variable was the policy itself. All tests were using 8.0.6. --- ## 1. V2 policy — `ahp-ew-perf-policyv2-gamma` **Type:** Firewall mode **PolicyMode:** `MONITOR` · **FirewallDefaultAction:** `ACCEPT` (accept:hook) **Consumed capacity:** 30,000 firewall-mode + 50,000 IPS = 80,000 rules ### Rule groups referenced | Slot | Rule group | Type | Capacity | Rules | |------|-----------|------|---------:|------:| | FirewallRuleGroupEntries (prio 1) | `ahp-ew-perf-fm-0` | FIREWALL | 15,000 | 15,000 | | FirewallRuleGroupEntries (prio 2) | `ahp-ew-perf-fm-1` | FIREWALL | 15,000 | 15,000 | | IpsRuleGroupEntries | `ahp-ew-perf-ips-0` | IPS | 12,500 | 12,500 | | IpsRuleGroupEntries | `ahp-ew-perf-ips-1` | IPS | 12,500 | 12,500 | | IpsRuleGroupEntries | `ahp-ew-perf-ips-2` | IPS | 12,500 | 12,500 | | IpsRuleGroupEntries | `ahp-ew-perf-ips-3` | IPS | 12,500 | 12,500 | ### Firewall-mode ruleset (30,000 rules) — composition Auto-generated, diverse, repeating 6-rule cycle. sids `100001`–`130000`. These are stateful-**flow** firewall-mode rules (actions `accept`/`drop`/`reject`; **no `pass`** — `pass` is enabled and there is not a firewall-mode action). **By action:** | Action | Count | % | |--------|------:|------:| | accept | 20,000 | 66.7% | | drop | 5,000 | 16.7% | | reject | 5,000 | 16.7% | **By engine qualifier / protocol scope:** | Scope | Count | % | |-------|------:|--:| | `tcp:all` | 10,000 | 33.3% | | `tls:<client_hello_done` | 5,000 | 16.7% | | `http1:<request_headers` | 5,000 | 16.7% | | `http2:stream:<request_headers` | 5,000 | 16.7% | | `ip:all` | 5,000 | 16.7% | **L7 content matching:** 15,000 port/IP-only · 10,000 `http.host` content · 5,000 `tls.sni` content. **Destination:** 20,000 → `any`; 10,000 → a random `10.x.y.0/24` CIDR. **19 distinct dest ports** (21, 22, 25, 53, 80, 110, 143, 443, 587, 993, 1433, 3306, 5432, 6379, 8080, 8443, 9200, 27017, plus `any`). **Example firewall-mode rules (one full cycle):** ``` accept:flow tls:<client_hello_done any any -> any 8443 (tls.sni; content:"cdn1238.perf-test8.net"; endswith; nocase; sid:100001;) accept:flow http1:<request_headers any any -> any 6379 (http.host; content:"api74.svc14.internal"; endswith; sid:100002;) drop:flow tcp:all any any -> 10.1.20.0/24 22 (sid:100003;) accept:flow ip:all any any -> 10.1.12.0/24 any (sid:100004;) reject:flow tcp:all any any -> any 143 (sid:100005;) accept:flow http2:stream:<request_headers any any -> any 8080 (http.host; content:"host244.example44.com"; endswith; sid:100006;) ``` ### IPS ruleset (50,000 rules) 4 IPS-type rule groups × 12,500. Suricata IPS/alert rules in a separate sid space from the firewall-mode set above. --- ## 2. V1 control policy **Type:** V1 (classic stateful; system mode) Purpose: a permissive **control** for the perf comparison — half the rules `pass`, half `alert`, no L7 content matching — to isolate stateful-engine rule-count overhead without blocking traffic or doing deep inspection. ### Rule groups referenced | Rule group | Type | Capacity | Rules | |-----------|------|---------:|------:| | `ahp-ctrl-v1-sf-0` | STATEFUL | 12,500 | 12,500 | | `ahp-ctrl-v1-sf-1` | STATEFUL | 12,500 | 12,500 | | `ahp-ctrl-v1-sf-2` | STATEFUL | 12,500 | 12,500 | | `ahp-ctrl-v1-sf-3` | STATEFUL | 12,500 | 12,500 | ### Stateful ruleset (50,000 rules) — composition Auto-generated, diverse. sids `200000`–`249999`. **By action:** | Action | Count | % | |--------|------:|----:| | pass | 25,000 | 50% | | alert | 25,000 | 50% | **By protocol:** | Protocol | Count | % | |----------|------:|--:| | udp | 16,667 | 33.3% | | tcp | 8,334 | 16.7% | | tls | 8,333 | 16.7% | | http | 8,333 | 16.7% | | ip | 8,333 | 16.7% | **flow keyword:** 8,333 carry `flow:to_server` (the tls/http alert rules); 41,667 plain. **11 distinct dest ports** (`any`, 53, 5432, 443, 80, `[80,443]`, 22, 8080, 3306, 500, 123). No L7 content matches (action/proto/port only). **Example stateful rules (rules files are attached to the redmine):** ``` pass tcp any any -> any 443 (msg:"east-west perf 0"; sid:200000; rev:1;) alert udp any any -> any any (msg:"control pass 1"; sid:200001; rev:1;) pass tls any any -> any 22 (msg:"control alert 2"; sid:200002; rev:1;) alert http any any -> any 8080 (msg:"tls flow 3"; flow:to_server; sid:200003; rev:1;) pass udp any any -> any 53 (msg:"http flow 4"; sid:200004; rev:1;) alert ip any any -> any any (msg:"dns flow 5"; sid:200005; rev:1;) ``` --- ## 3. Policy comparison summary | | V2 | V1 control | |---|---|---| | Policy type | Policy **V2** | Policy **V1** (classic system mode) | | Firewall-mode rules | 30,000 | — (no FM engine in V1) | | IPS / stateful rules | 50,000 (IPS) | 50,000 (stateful) | | Total rules | 80,000 | 50,000 | | Actions | accept/drop/reject (FM) + IPS alerts | pass 50% / alert 50% | | L7 content matching | yes (tls.sni + http.host) | none | | Mode / order | MONITOR, ACCEPT default accept:hook | DEFAULT_ACTION_ORDER | --- ## 4. Soak-test load generation Goal: sustained east-west load through the endpoint ### Topology - there seems **Load generators (clients) in `10.0.2.0/24`:** 8 × `c6i.12xlarge` (48 vCPU each) `10.0.2.20, .83, .91, .224` (original 4) + `10.0.2.134, .179, .5, .241` (scaled-out 4). - **Targets (servers) in `10.0.1.0/24`:** 5 × nginx-over-TLS on :443 `10.0.1.10, .32, .216, .254, .73`. - Clients route the server CIDR through the endpoint, so all client→server traffic traverses endpoint. - Tool: **`h2load`** (from `nghttp2`), HTTP/2 over TLS. Installed on clients via `sudo yum install -y nghttp2` where missing. Driven over **SSM Run Command** (no SSH). ### Soak script (`/tmp/soak.sh`, staged on each client) Self-restarting loop so load is continuous overnight; one `h2load` process per server (5 parallel per client); stops when `/tmp/soak.stop` exists. ```bash #!/bin/bash CONC=${1:-100} # connections per server MAXCONC=${2:-10} # concurrent streams per connection THREADS=${3:-16} SERVERS="10.0.1.10 10.0.1.32 10.0.1.216 10.0.1.254 10.0.1.73" rm -f /tmp/soak.stop echo "soak start $(date -u) conc=$CONC m=$MAXCONC threads=$THREADS servers=5" >> /tmp/soak.log while [ ! -f /tmp/soak.stop ]; do for ip in $SERVERS; do h2load -D 120 -c "$CONC" -m "$MAXCONC" -t "$THREADS" "https://$ip/" >>"/tmp/soak_$ip.log" 2>&1 & done wait echo "cycle done $(date -u)" >> /tmp/soak.log done ``` ### Launch (detached so it survives the session) ```bash # staged via base64 (nested heredoc through SSM mangled the file; base64 decode was reliable) echo "$B64" | base64 -d > /tmp/soak.sh && chmod +x /tmp/soak.sh setsid nohup /tmp/soak.sh 100 10 16 >/tmp/soak_boot.log 2>&1 </dev/null & ``` Run on all 8 clients → **40 `h2load` streams total** (8 clients × 5 servers) through the one endpoint. **Stop:** `touch /tmp/soak.stop` on each client (or `pkill -f soak.sh`). ### Tuning / calibration notes - Settled on **`c=100, m=10, t=16`** per stream as the clean, near-zero-failure profile. - Higher concurrency (c=200–500, m=50–100) produced heavy `Process Request Failure` with *lower* goodput — connection churn, not more throughput. Lower streams-per-connection on more targets was the stable regime. - Single isolated stream hit ~20,000 req/s cleanly; servers stayed ~99% idle under the soak, so the limiter was load-gen connection setup, not server or firewall capacity. ### Logging Netflow and alert logging was turned on for these tests. ### Soak test regression on PPS we can achieve on same load gen through the firewall In attempts to be isolate the bottleneck - we performed a large PPS drop by >50%. Tests were done using couple tests: 1. We compared performance to a non-firewall mode suricata config that had 50k IPS rules on system mode vs (rules attached) to a firewall mode suricata config that had 50k IPS rules that were the same, but with 0 firewall rules - we saw PPS drop down after flipping to the firewall mode one from 3.2M PPS to ~189k PPS. 2. We then tried to compare the same non-firewall mode suricata config with a firewall mode one that had no firewall or ips rules, but just had a default firewall mode action set to accept:hook for - This showed similar levels of PPS with the default action in non-firewall mode config that had the maxed out 50k ips rules. 3. We then tried to flip to one that was using around 30k domain based tls.sni and http.host rules on the firewall mode side (attached to ticket) with no ips rules - We could see the PPS jump from around 213k to 4.37M PPS, which was a significant jump, but still was lower than the control and the same no rules V2 test. This leads us to believe there may be a bottleneck around the IPS rule matching, but would like you all to see if you see things different on your side. The results can be visualized on the attached graph. ### h2load testing We also tried to test more fine grained on the initial comparison (30k firewall rules + 50k IPS rules. ips rules compared with 50k ips rules running in system mode). The results showed for that for a constant number of requests (100000) - we could increase the concurrent connections to roughly 1000 more concurrent connections to be able to go through system mode. The command we used for these was: ``` h2load -n 100000 -c 100 -m 100 -t 8 https://east-west.ahp.people.aws.dev ``` Where we would just increment -c until it would result in connections either erroring or timing out (saturation point). Plotting the mean statistics between the runs, we could see the largest diff was in mean time for request (graph attached). The saturation point was found was `-c 2400` on V2, but `-c 3400` on V1. **We are hoping for any insight you may have on how to improve the performance further.**