Suricata

I have been trying the the new xbit/hostbit features. I've put some test rules in https://github.com/EmergingThreats/et-luajit-scripts experimental.rules

The problem is that when using autofp or workers mode there's no guarantee the relevant traffic will all be in the same thread and hence match the xbits. For example the Neutrino exploit kit pcap in http://www.malware-traffic-analysis.net/2015/08/31/index.html will hit the "ET EXPERIMENTAL (data) download with high entropy after plugin file download" rule for the RC4-encrypted Flash payload with runmode=single but not for runmode=autofp :-

alert http $EXTERNAL_NET any -> $HOME_NET any (msg:"ET EXPERIMENTAL Flash file download (CWS)"; flow:established,from_server; file_data; content:"CWS"; within:3; xbits:set,ET.pluginfile,track ip_pair, expire 120; flowbits:noalert; classtype:bad-unknown; sid:380000001; rev:1;)

alert http $EXTERNAL_NET any -> $HOME_NET any (msg:"ET EXPERIMENTAL (data) download with high entropy after plugin file download"; flow:from_server,established; filesize:>10240; filemagic:"data"; filemagic:!" data"; xbits:isset,ET.pluginfile,track ip_pair; luajit:suri-high-entropy.lua; classtype:bad-unknown; sid:380000015; rev:1;)

One possibility is to use packet-schedulers that hash on IP pair rather than 5-tuple; e.g. for PF_RING it would be possible to use libzero and a custom distribution function.

I created a patch to provide an "ippair" scheduler for autofp (attached), but I'm not sure it's a sensible thing in practice (except perhaps in sandboxes), the downside of course being that traffic may be very unevenly distributed between threads.

I think Victor mentioned that there may be some improvements to be had by prioritising IP pair checks in the engine. Are there other suggestions to make this feature more usable?