byte_test checks before byte_extract happens in some cases
When there is a byte_extract in the http_raw_header (or http_header) buffer, and it is used in a byte_test in the file_data (or http_server_body) buffer, it doesn't work as expected. I am not sure if other buffer combinations have this issue as well.
The byte_test comparison (DetectBytetestDoMatch() in detect-bytetest.c, called by DetectEngineContentInspection() in detect-engine-content-inspection.c) happens before the bytes in byte_extract are actually extracted (extraction done in ByteExtractString() in util-byte.c, which is called by DetectByteExtractDoMatch() in detect-byte-extract.c). This results in the value used in the byte_test comparison being 0 instead of being the value from the byte_extract.
The issue isn't the extractions themselves in byte_extract or byte_test (these work fine), the problem is that the comparison done by byte_test (using a variable from byte_extract) is done before the byte_extract (resulting in the wrong value -- 0 -- being used in the byte_test comparison).
Here is a sample rule that can be used with the atached pcap:
alert http any any -> any any (msg:"byte_extract Test"; flow:established,to_client; content:"Content|2D|Length|3A 20|"; http_raw_header; content:!"|0D 0A|"; within:3; http_raw_header; byte_extract:2,0,content-length,relative,string,dec; content:"|0D 0A|"; distance:0; within:2; http_raw_header; file_data; content:"test"; byte_test:2,=,content-length,0,relative,little; priority:3; sid:44412999;)
If you trace the code when you run the pcap against the rule, you can see that the byte(s) in byte_test are extracted successfully (in the pcap the extracted value is (decimal) 11) but the comparison compares it with the number 0 instead of the extracted value (because the extraction hasn't happened yet). If you change the operator in the byte_test from '=' to '>', this will pass (since 11 > 0) and then you can see that the byte_extract happens after that (and the byte_extract accurately extracts the value from the http_raw_header as (decimal) 11), but by then it is too late.
Tested in Suricata 2.07.
Updated by Victor Julien over 7 years ago
If I understand the issue correctly, it seems the real problem is that some buffers have a fixed inspection order. This is currently independent of the keyword order in the rule. Within one buffer (e.g. uri), the order of the rule is followed.
I think it would make sense to inspect things in the order of the rule where possible. This would require some internal changes, but it doesn't look impossible.
One open issue would be how to handle 'stateful' detection (e.g. URI is matched with packet X, client_body with packet X+N). Would we need to store the byte_extract values in the state so that it's available at time X+N?