Bug #453
closeddetection_filter's "count" misbehaving - not accurate
Added by Peter Manev about 14 years ago. Updated about 14 years ago.
Description
Please see files/example/explanation attached.
Files
| detectionfilter_count.zip (3.96 KB) detectionfilter_count.zip | Peter Manev, 04/22/2012 10:52 AM | ||
| 0001-fix-FNs-for-flow-only_stream-and-no_stream-options.patch (2.68 KB) 0001-fix-FNs-for-flow-only_stream-and-no_stream-options.patch | Anoop Saldanha, 04/24/2012 11:41 AM | ||
| 0002-fix-detection-filter.-Had-one-extra-alert-than-norma.patch (1.51 KB) 0002-fix-detection-filter.-Had-one-extra-alert-than-norma.patch | Anoop Saldanha, 04/24/2012 11:41 AM | ||
| 0003-considering-the-tenths-of-a-seconds-in-a-packet-when.patch (2.36 KB) 0003-considering-the-tenths-of-a-seconds-in-a-packet-when.patch | Anoop Saldanha, 04/24/2012 11:41 AM | ||
| 0004-code-cleanup.patch (1.05 KB) 0004-code-cleanup.patch | Anoop Saldanha, 04/24/2012 11:41 AM | ||
| 0005-fix-detection-filter-unittests-to-reflect-recent-fix.patch (1.01 KB) 0005-fix-detection-filter-unittests-to-reflect-recent-fix.patch | Anoop Saldanha, 04/24/2012 11:41 AM |
PM Updated by Peter Manev about 14 years ago Actions #1
- Description updated (diff)
correction in the rules file (previously attached)
the line
#in the above rule when we have "count 1" we have 19 alerts (which is correct!), with "count 2" we have 13 alerts (should be 9), with "count 3" we have 8 alerts
is wrong
apologies
AS Updated by Anoop Saldanha about 14 years ago Actions #2
- Description updated (diff)
the only_stream FN for the rule is a bug. Shouldn't be any different for no_stream as well which would actually FP for matches on content across payload boundaries. You can open a new ticket for this bug.
AS Updated by Anoop Saldanha about 14 years ago Actions #3
- File 0001-fix-FNs-for-flow-only_stream-and-no_stream-options.patch added
patch attached for the flow only_stream and no_stream bug.
AS Updated by Anoop Saldanha about 14 years ago Actions #4
- File 0002-fix-detection-filter.-Had-one-extra-alert-than-norma.patch added
Patch attached for detection_filter.
PM Updated by Peter Manev about 14 years ago Actions #5
Hi,
only_stream was one thing, however -
No. Time Source Destination Protocol Length Info
6 2010-12-03 18:04:14.660419 10.16.5.5 10.13.146.1 HTTP 264 HTTP/1.1 304 Not Modified
7 2010-12-03 18:04:14.913972 10.13.146.1 10.16.5.5 TCP 52 39337 > http [ACK] Seq=484 Ack=213 Win=6912 Len=0 TSval=4950187 TSecr=34268534
8 2010-12-03 18:04:15.578059 10.13.146.1 10.16.5.5 HTTP 572 GET /flags/team_167_flag.png HTTP/1.1
9 2010-12-03 18:04:15.582178 10.16.5.5 10.13.146.1 HTTP 240 HTTP/1.1 304 Not Modified
11 2010-12-03 18:04:15.841140 10.13.146.1 10.16.5.5 HTTP 572 GET /flags/team_152_flag.png HTTP/1.1
12 2010-12-03 18:04:15.841586 10.16.5.5 10.13.146.1 HTTP 240 HTTP/1.1 304 Not Modified
13 2010-12-03 18:04:16.094824 10.13.146.1 10.16.5.5 HTTP 572 GET /flags/team_103_flag.png HTTP/1.1
14 2010-12-03 18:04:16.095312 10.16.5.5 10.13.146.1 HTTP 240 HTTP/1.1 304 Not Modified
15 2010-12-03 18:04:16.348858 10.13.146.1 10.16.5.5 HTTP 572 GET /flags/team_160_flag.png HTTP/1.1
16 2010-12-03 18:04:16.349492 10.16.5.5 10.13.146.1 HTTP 240 HTTP/1.1 304 Not Modified
17 2010-12-03 18:04:16.603903 10.13.146.1 10.16.5.5 HTTP 572 GET /flags/team_128_flag.png HTTP/1.1
18 2010-12-03 18:04:16.604554 10.16.5.5 10.13.146.1 HTTP 240 HTTP/1.1 304 Not Modified
19 2010-12-03 18:04:16.862050 10.13.146.1 10.16.5.5 HTTP 570 GET /flags/flag_canada.png HTTP/1.1
20 2010-12-03 18:04:16.862865 10.16.5.5 10.13.146.1 HTTP 240 HTTP/1.1 304 Not Modified
21 2010-12-03 18:04:17.117127 10.13.146.1 10.16.5.5 HTTP 574 GET /flags/flag_netherlands.png HTTP/1.1
22 2010-12-03 18:04:17.117527 10.16.5.5 10.13.146.1 HTTP 239 HTTP/1.1 304 Not Modified
@Anoop - what I meant here (for detect_filter in particular, in my own crappy way) is the following:
imagine the following rule fo this particular situation above:
alert tcp $HTTP_SERVERS $HTTP_PORTS -> $EXTERNAL_NET any (msg:"WEB-MISC Multiple Products excessive HTTP 304 Not Modified responses exploit attempt";
flow:to_client,established,only_stream; content:"HTTP/1.1 304 Not Modified";
detection_filter:track by_dst, count 2, seconds 3; metadata:policy balanced-ips drop, policy security-ips drop, service http;
reference:cve,2007-0947; reference:cve,2007-6239; reference:url,technet.microsoft.com/en-us/security/bulletin/ms07-027;
classtype:misc-activity; sid:16008; rev:9;)
the rule above is an edit of the original one !
how do we "count" ?
EX
detection_filter:track by_dst, count 2, seconds 2; - we fire on any occurance after the first 2, within a 2 sec period? correct?
6 2010-12-03 18:04:14.660419 10.16.5.5 10.13.146.1 HTTP 264 HTTP/1.1 304 Not Modified
9 2010-12-03 18:04:15.582178 10.16.5.5 10.13.146.1 HTTP 240 HTTP/1.1 304 Not Modified
12 2010-12-03 18:04:15.841586 10.16.5.5 10.13.146.1 HTTP 240 HTTP/1.1 304 Not Modifieddo we stop logging alerts here (packet num 12) , for this particular occurrence in the rule?14 2010-12-03 18:04:16.095312 10.16.5.5 10.13.146.1 HTTP 240 HTTP/1.1 304 Not Modified
16 2010-12-03 18:04:16.349492 10.16.5.5 10.13.146.1 HTTP 240 HTTP/1.1 304 Not Modified
18 2010-12-03 18:04:16.604554 10.16.5.5 10.13.146.1 HTTP 240 HTTP/1.1 304 Not Modified
20 2010-12-03 18:04:16.862865 10.16.5.5 10.13.146.1 HTTP 240 HTTP/1.1 304 Not ModifiedOR do we stop logging alerts here (packet num 20) ?
22 2010-12-03 18:04:17.117527 10.16.5.5 10.13.146.1 HTTP 239 HTTP/1.1 304 Not Modifiedin other words is the sec that the first alert is seen inclusive?
because i think there is the point where i saw some inaccuracy.
thanks
PM Updated by Peter Manev about 14 years ago Actions #6
@Anoop - I just saw you updated with a patch for detect_filter - i think that is probably fixing the issue that I saw. Will retest and let you know.
Thank you
AS Updated by Anoop Saldanha about 14 years ago Actions #7
Peter Manev wrote:
Hi,
only_stream was one thing, however -
No. Time Source Destination Protocol Length Info
6 2010-12-03 18:04:14.660419 10.16.5.5 10.13.146.1 HTTP 264 HTTP/1.1 304 Not Modified
7 2010-12-03 18:04:14.913972 10.13.146.1 10.16.5.5 TCP 52 39337 > http [ACK] Seq=484 Ack=213 Win=6912 Len=0 TSval=4950187 TSecr=34268534
8 2010-12-03 18:04:15.578059 10.13.146.1 10.16.5.5 HTTP 572 GET /flags/team_167_flag.png HTTP/1.1
9 2010-12-03 18:04:15.582178 10.16.5.5 10.13.146.1 HTTP 240 HTTP/1.1 304 Not Modified
11 2010-12-03 18:04:15.841140 10.13.146.1 10.16.5.5 HTTP 572 GET /flags/team_152_flag.png HTTP/1.1
12 2010-12-03 18:04:15.841586 10.16.5.5 10.13.146.1 HTTP 240 HTTP/1.1 304 Not Modified
13 2010-12-03 18:04:16.094824 10.13.146.1 10.16.5.5 HTTP 572 GET /flags/team_103_flag.png HTTP/1.1
14 2010-12-03 18:04:16.095312 10.16.5.5 10.13.146.1 HTTP 240 HTTP/1.1 304 Not Modified
15 2010-12-03 18:04:16.348858 10.13.146.1 10.16.5.5 HTTP 572 GET /flags/team_160_flag.png HTTP/1.1
16 2010-12-03 18:04:16.349492 10.16.5.5 10.13.146.1 HTTP 240 HTTP/1.1 304 Not Modified
17 2010-12-03 18:04:16.603903 10.13.146.1 10.16.5.5 HTTP 572 GET /flags/team_128_flag.png HTTP/1.1
18 2010-12-03 18:04:16.604554 10.16.5.5 10.13.146.1 HTTP 240 HTTP/1.1 304 Not Modified
19 2010-12-03 18:04:16.862050 10.13.146.1 10.16.5.5 HTTP 570 GET /flags/flag_canada.png HTTP/1.1
20 2010-12-03 18:04:16.862865 10.16.5.5 10.13.146.1 HTTP 240 HTTP/1.1 304 Not Modified
21 2010-12-03 18:04:17.117127 10.13.146.1 10.16.5.5 HTTP 574 GET /flags/flag_netherlands.png HTTP/1.1
22 2010-12-03 18:04:17.117527 10.16.5.5 10.13.146.1 HTTP 239 HTTP/1.1 304 Not Modified@Anoop - what I meant here (for detect_filter in particular, in my own crappy way) is the following:
imagine the following rule fo this particular situation above:alert tcp $HTTP_SERVERS $HTTP_PORTS -> $EXTERNAL_NET any (msg:"WEB-MISC Multiple Products excessive HTTP 304 Not Modified responses exploit attempt";
flow:to_client,established,only_stream; content:"HTTP/1.1 304 Not Modified";
detection_filter:track by_dst, count 2, seconds 3; metadata:policy balanced-ips drop, policy security-ips drop, service http;
reference:cve,2007-0947; reference:cve,2007-6239; reference:url,technet.microsoft.com/en-us/security/bulletin/ms07-027;
classtype:misc-activity; sid:16008; rev:9;)the rule above is an edit of the original one !
how do we "count" ?
EX
detection_filter:track by_dst, count 2, seconds 2; - we fire on any occurance after the first 2, within a 2 sec period? correct?6 2010-12-03 18:04:14.660419 10.16.5.5 10.13.146.1 HTTP 264 HTTP/1.1 304 Not Modified
9 2010-12-03 18:04:15.582178 10.16.5.5 10.13.146.1 HTTP 240 HTTP/1.1 304 Not Modified
12 2010-12-03 18:04:15.841586 10.16.5.5 10.13.146.1 HTTP 240 HTTP/1.1 304 Not Modifieddo we stop logging alerts here (packet num 12) , for this particular occurrence in the rule?
14 2010-12-03 18:04:16.095312 10.16.5.5 10.13.146.1 HTTP 240 HTTP/1.1 304 Not Modified
16 2010-12-03 18:04:16.349492 10.16.5.5 10.13.146.1 HTTP 240 HTTP/1.1 304 Not Modified
18 2010-12-03 18:04:16.604554 10.16.5.5 10.13.146.1 HTTP 240 HTTP/1.1 304 Not Modified
20 2010-12-03 18:04:16.862865 10.16.5.5 10.13.146.1 HTTP 240 HTTP/1.1 304 Not ModifiedOR do we stop logging alerts here (packet num 20) ?
22 2010-12-03 18:04:17.117527 10.16.5.5 10.13.146.1 HTTP 239 HTTP/1.1 304 Not Modified
in other words is the sec that the first alert is seen inclusive?
because i think there is the point where i saw some inaccuracy.thanks
you would stop logging after the 24th packet. So basically you will have alerts for data from 12 - 24(6 and 9 are not logged). The 26th packet would reset the threshold.
Do note that the timestamp stored by the threshold engine is for the first packet that generates the alert, which for us is the 9th packet. So all timestamp calculations(which in your case is 2 seconds) is against the 9th packet one.
PM Updated by Peter Manev about 14 years ago Actions #8
so with the pcap attached and this rule:
alert tcp $HTTP_SERVERS $HTTP_PORTS -> $EXTERNAL_NET any (msg:"WEB-MISC Multiple Products excessive HTTP 304 Not Modified responses exploit attempt";
flow:to_client,established,only_stream; content:"HTTP/1.1 304 Not Modified";
detection_filter:track by_dst, count 2, seconds 3; metadata:policy balanced-ips drop, policy security-ips drop, service http;
reference:cve,2007-0947; reference:cve,2007-6239; reference:url,technet.microsoft.com/en-us/security/bulletin/ms07-027;
classtype:misc-activity; sid:16008; rev:9;)
we should log 17 alerts, correct? (with the new patch added)
AS Updated by Anoop Saldanha about 14 years ago Actions #9
Peter Manev wrote:
so with the pcap attached and this rule:
alert tcp $HTTP_SERVERS $HTTP_PORTS -> $EXTERNAL_NET any (msg:"WEB-MISC Multiple Products excessive HTTP 304 Not Modified responses exploit attempt";
flow:to_client,established,only_stream; content:"HTTP/1.1 304 Not Modified";
detection_filter:track by_dst, count 2, seconds 3; metadata:policy balanced-ips drop, policy security-ips drop, service http;
reference:cve,2007-0947; reference:cve,2007-6239; reference:url,technet.microsoft.com/en-us/security/bulletin/ms07-027;
classtype:misc-activity; sid:16008; rev:9;)we should log 17 alerts, correct? (with the new patch added)
you would have 14 actually
6 0.253728 10.16.5.5 10.13.146.1 HTTP HTTP/1.1 304 Not Modified
9 1.175487 10.16.5.5 10.13.146.1 HTTP HTTP/1.1 304 Not Modified
12 1.434895 10.16.5.5 10.13.146.1 HTTP HTTP/1.1 304 Not Modified
14 1.688621 10.16.5.5 10.13.146.1 HTTP HTTP/1.1 304 Not Modified
16 1.942801 10.16.5.5 10.13.146.1 HTTP HTTP/1.1 304 Not Modified
18 2.197863 10.16.5.5 10.13.146.1 HTTP HTTP/1.1 304 Not Modified
20 2.456174 10.16.5.5 10.13.146.1 HTTP HTTP/1.1 304 Not Modified
22 2.710836 10.16.5.5 10.13.146.1 HTTP HTTP/1.1 304 Not Modified
24 2.967862 10.16.5.5 10.13.146.1 HTTP HTTP/1.1 304 Not Modified
26 3.223546 10.16.5.5 10.13.146.1 HTTP HTTP/1.1 304 Not Modified
28 3.479146 10.16.5.5 10.13.146.1 HTTP HTTP/1.1 304 Not Modified
30 3.736715 10.16.5.5 10.13.146.1 HTTP HTTP/1.1 304 Not Modified
32 3.991349 10.16.5.5 10.13.146.1 HTTP HTTP/1.1 304 Not Modified
34 4.248674 10.16.5.5 10.13.146.1 HTTP HTTP/1.1 304 Not Modified
36 4.502518 10.16.5.5 10.13.146.1 HTTP HTTP/1.1 304 Not Modified
38 4.760021 10.16.5.5 10.13.146.1 HTTP HTTP/1.1 304 Not Modified
40 5.014425 10.16.5.5 10.13.146.1 HTTP HTTP/1.1 304 Not Modified
42 5.268604 10.16.5.5 10.13.146.1 HTTP HTTP/1.1 304 Not Modified
44 5.526490 10.16.5.5 10.13.146.1 HTTP HTTP/1.1 304 Not Modified
The first alert is generated for the 9th packet. This is the start timestamp for the threshold against which we will do our timestamp comparisons
start_timestamp = 1.17
The alerts for the 9th and 12th packets are discarded.
Alerts for packets 14 - 32 are logged - So far we have 10 alerts
34th packet would reset the threshold since we would have crossed 3 seconds. start_timestamp is reset
start_timestamp = 4.24
Alerts from 34th and 36th packets are discarded.
We next log alerts for packets 38 - 44. That makes it 44 alerts.
We will have one final alert for the 46th packet during the fin which would again be discarded since the threshold timer has exceeded.
How did you arrive at 17 btw?
PM Updated by Peter Manev about 14 years ago Actions #10
sorry,
copied the wrong rule - i meant this
detection_filter:track by_dst, count 2, seconds 5;
then we should have 17.
for the earlier example - the previous description by you -
"The alerts for the 9th and 12th packets are discarded." - why is 12 discarded, shouldn't it just be that 6,9 are discarded ?
AS Updated by Anoop Saldanha about 14 years ago Actions #11
Peter Manev wrote:
sorry,
copied the wrong rule - i meant thisdetection_filter:track by_dst, count 2, seconds 5;
then we should have 17.
We would have 16. The first alert is seen with our 9th packet which is discarded and threshold sets the first timestamp at 1.17 seconds. So the max time limit for threshold is 6.17 seconds and our next set of packets are within that range, except for the alert generated by the 46th packet against 44th packet's data, which would be discarded as it exceeds the threshold time limit.
for the earlier example - the previous description by you -
"The alerts for the 9th and 12th packets are discarded." - why is 12 discarded, shouldn't it just be that 6,9 are discarded ?
The stream data from the 6th packet is inspected by the detection engine when it sees another packet in the relevant direction. The 6th packet gets acked by the 7th packet and this is when data is ready to be inspected by the detection engine, following which it will wait till it sees another packet in the same toclient direction which is the 9th packet, which is when we generate an alert for the data from the 6th packet.
So the alerts generated by the 9th and 12th packets(against data from 6th and 9th packets) are discarded by the threshold.
PM Updated by Peter Manev about 14 years ago Actions #12
understood.
Thank you Anoop
AS Updated by Anoop Saldanha about 14 years ago Actions #13
- File 0003-considering-the-tenths-of-a-seconds-in-a-packet-when.patch added
- Status changed from New to Resolved
Attached another patch. Should improve threshold alerting down to the microseconds.
AS Updated by Anoop Saldanha about 14 years ago Actions #14
- Assignee set to Anoop Saldanha
VJ Updated by Victor Julien about 14 years ago Actions #15
1 doesn't apply cleanly to new master, please rebase.
2 leads to 2 (extra) failed unittests, can you look into that?
AS Updated by Anoop Saldanha about 14 years ago Actions #16
- File deleted (
0001-fix-FNs-for-flow-only_stream-and-no_stream-options.patch)
AS Updated by Anoop Saldanha about 14 years ago Actions #17
- File deleted (
0002-fix-detection-filter.-Had-one-extra-alert-than-norma.patch)
AS Updated by Anoop Saldanha about 14 years ago Actions #18
- File deleted (
0003-considering-the-tenths-of-a-seconds-in-a-packet-when.patch)
AS Updated by Anoop Saldanha about 14 years ago Actions #19
- File 0001-fix-FNs-for-flow-only_stream-and-no_stream-options.patch 0001-fix-FNs-for-flow-only_stream-and-no_stream-options.patch added
- File 0002-fix-detection-filter.-Had-one-extra-alert-than-norma.patch 0002-fix-detection-filter.-Had-one-extra-alert-than-norma.patch added
- File 0003-considering-the-tenths-of-a-seconds-in-a-packet-when.patch 0003-considering-the-tenths-of-a-seconds-in-a-packet-when.patch added
- File 0004-code-cleanup.patch 0004-code-cleanup.patch added
- File 0005-fix-detection-filter-unittests-to-reflect-recent-fix.patch 0005-fix-detection-filter-unittests-to-reflect-recent-fix.patch added
rebased patches.
VJ Updated by Victor Julien about 14 years ago Actions #20
- Status changed from Resolved to Closed
- Target version set to 1.3beta2
- % Done changed from 0 to 100
Applied and pushed to the master, thanks Anoop.