Project

General

Profile

Actions

Bug #4842

closed

smb: excessive memory use during file transfer

Added by Srini J 10 months ago. Updated 5 months ago.

Status:
Closed
Priority:
Normal
Assignee:
Target version:
Affected Versions:
Effort:
Difficulty:
Label:
Needs backport to 5.0, Needs backport to 6.0

Description

Run away memory allocations in Suricata-6.0.3 at suricata::filetracker::FileTransferTracker::update()->::extend() while transferring file between Windows 10 clients. While transferring a 2.6GB ova file between Windows 10 clients, Suricata consumed about 2.6GB memory. I have a pcap file as well to reproduce this issue at will (about 400MB file), please let me know if it would be needed. Thanks to my teammate Santosh K for finding this issue (credit goes to him).

Before posting this, checked with Suricata-6.0.4 and found that issue is seen there as well.

Content of smb.rules (single ETOpen rule is enough to trigger the issue)
drop smb $HOME_NET any -> any any (msg:"ET EXPLOIT ETERNALBLUE Probe Vulnerable System Response MS17-010"; flow:from_server,established; content:"|ff|SMB|25 05 02 00 c0 98 01|"; offset:4; depth:11; content:"|00 00 00 00 00 00 00 00 00 00|"; distance:3; within:10; content:"|00 00 00|"; distance:8; within:3; isdataat:!1,relative; threshold: type limit, track by_src, count 1, seconds 30; reference:url,github.com/rapid7/metasploit-framework/blob/master/modules/auxiliary/scanner/smb/smb_ms17_010.rb; classtype:trojan-activity; sid:2025650; rev:2; metadata:affected_product Windows_XP_Vista_7_8_10_Server_32_64_Bit, attack_target Client_Endpoint, created_at 2018_07_11, deployment Internal, former_category EXPLOIT, signature_severity Major, tag Metasploit, tag ETERNALBLUE, updated_at 2019_09_28;)

RES memory consumption of Suricata-6.0.3 while transfering an ova file of about 2.6GB
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
31889 root 20 0 3251724 2.590g 20844 S 3.0 16.5 0:10.26 FM#01
31890 root 20 0 3251724 2.590g 20844 S 3.0 16.5 0:10.12 FR#01
31886 root 20 0 3251724 2.590g 20844 S 0.3 16.5 0:00.95 Suricata-Main
31887 root 20 0 3251724 2.590g 20844 S 0.0 16.5 0:21.76 W#01-ens192
31888 root 20 0 3251724 2.590g 20844 S 0.0 16.5 0:03.67 W#01-ens224
31891 root 20 0 3251724 2.590g 20844 S 0.0 16.5 0:00.00 CW
31892 root 20 0 3251724 2.590g 20844 S 0.0 16.5 0:00.00 CS
31893 root 20 0 3251724 2.590g 20844 S 0.0 16.5 0:00.03 US

Call backtrace and location where memory is held. Please refer jeprof attached gif for more details
AppLayerParserParse()->rs_smb_parse_request_tcp()->suricata::smb::smb::SMBState::parse_tcp_data_ts()->suricata::smb::files::::filetracker_update()->suricata::filetracker::FileTransferTracker::update()->::extend()->::spec_extend()

19/11/2021 -- 19:42:42 - <Info> - Configuration node 'enabled' redefined.
19/11/2021 -- 19:42:42 - <Info> - Configuration node 'filetype' redefined.
19/11/2021 -- 19:42:42 - <Info> - Configuration node 'filename' redefined.
19/11/2021 -- 19:42:42 - <Info> - Configuration node 'types' redefined.
19/11/2021 -- 19:42:42 - <Info> - Configuration node 'enabled' redefined.
vars = (null)
vars.address-groups = (null)
vars.address-groups.HOME_NET = any
vars.address-groups.EXTERNAL_NET = any
vars.address-groups.HTTP_SERVERS = $HOME_NET
vars.address-groups.SMTP_SERVERS = $HOME_NET
vars.address-groups.SQL_SERVERS = $HOME_NET
vars.address-groups.DNS_SERVERS = $HOME_NET
vars.address-groups.TELNET_SERVERS = $HOME_NET
vars.address-groups.AIM_SERVERS = $EXTERNAL_NET
vars.address-groups.DC_SERVERS = $HOME_NET
vars.address-groups.DNP3_SERVER = $HOME_NET
vars.address-groups.DNP3_CLIENT = $HOME_NET
vars.address-groups.MODBUS_CLIENT = $HOME_NET
vars.address-groups.MODBUS_SERVER = $HOME_NET
vars.address-groups.ENIP_CLIENT = $HOME_NET
vars.address-groups.ENIP_SERVER = $HOME_NET
vars.port-groups = (null)
vars.port-groups.HTTP_PORTS = 80
vars.port-groups.SHELLCODE_PORTS = !80
vars.port-groups.ORACLE_PORTS = 1521
vars.port-groups.SSH_PORTS = 22
vars.port-groups.DNP3_PORTS = 20000
vars.port-groups.MODBUS_PORTS = 502
vars.port-groups.FILE_DATA_PORTS = [$HTTP_PORTS,110,143]
vars.port-groups.FTP_PORTS = 21
default-log-dir = /var/log/suricata/
stats = (null)
stats.enabled = yes
stats.interval = 30
stats.decoder-events-prefix = decoder.event
outputs = (null)
outputs.0 = fast
outputs.0.fast = (null)
outputs.0.fast.enabled = no
outputs.0.fast.filename = fast.log
outputs.0.fast.append = yes
outputs.1 = eve-log
outputs.1.eve-log = (null)
outputs.1.eve-log.enabled = yes
outputs.1.eve-log.filetype = regular
outputs.1.eve-log.filename = eve.json
outputs.1.eve-log.facility = local5
outputs.1.eve-log.types = (null)
outputs.1.eve-log.types.0 = alert
outputs.1.eve-log.types.0.alert = (null)
outputs.1.eve-log.types.0.alert.tagged-packets = yes
outputs.1.eve-log.types.1 = http
outputs.1.eve-log.types.1.http = (null)
outputs.1.eve-log.types.1.http.enabled = no
outputs.1.eve-log.types.2 = dns
outputs.1.eve-log.types.2.dns = (null)
outputs.1.eve-log.types.2.dns.enabled = no
outputs.1.eve-log.types.2.dns.version = 2
outputs.1.eve-log.types.3 = tls
outputs.1.eve-log.types.3.tls = (null)
outputs.1.eve-log.types.3.tls.enabled = no
outputs.1.eve-log.types.4 = files
outputs.1.eve-log.types.4.files = (null)
outputs.1.eve-log.types.4.files.enabled = no
outputs.1.eve-log.types.4.files.force-magic = no
outputs.1.eve-log.types.5 = smtp
outputs.1.eve-log.types.5.smtp = (null)
outputs.1.eve-log.types.5.smtp.enabled = no
outputs.1.eve-log.types.6 = nfs
outputs.1.eve-log.types.6.nfs = (null)
outputs.1.eve-log.types.6.nfs.enabled = no
outputs.1.eve-log.types.7 = smb
outputs.1.eve-log.types.7.smb = (null)
outputs.1.eve-log.types.7.smb.enabled = no
outputs.1.eve-log.types.8 = tftp
outputs.1.eve-log.types.8.tftp = (null)
outputs.1.eve-log.types.8.tftp.enabled = no
outputs.1.eve-log.types.9 = ikev2
outputs.1.eve-log.types.9.ikev2 = (null)
outputs.1.eve-log.types.9.ikev2.enabled = no
outputs.1.eve-log.types.10 = krb5
outputs.1.eve-log.types.10.krb5 = (null)
outputs.1.eve-log.types.10.krb5.enabled = no
outputs.1.eve-log.types.11 = dhcp
outputs.1.eve-log.types.11.dhcp = (null)
outputs.1.eve-log.types.11.dhcp.enabled = no
outputs.1.eve-log.types.11.dhcp.extended = no
outputs.1.eve-log.types.12 = ssh
outputs.1.eve-log.types.12.ssh = (null)
outputs.1.eve-log.types.12.ssh.enabled = no
outputs.1.eve-log.pcap-file = false
outputs.1.eve-log.community-id = false
outputs.1.eve-log.community-id-seed = 0
outputs.1.eve-log.xff = (null)
outputs.1.eve-log.xff.enabled = no
outputs.1.eve-log.xff.mode = extra-data
outputs.1.eve-log.xff.deployment = reverse
outputs.1.eve-log.xff.header = X-Forwarded-For
outputs.2 = unified2-alert
outputs.2.unified2-alert = (null)
outputs.2.unified2-alert.enabled = no
outputs.2.unified2-alert.filename = unified2.alert
outputs.2.unified2-alert.xff = (null)
outputs.2.unified2-alert.xff.enabled = no
outputs.2.unified2-alert.xff.mode = extra-data
outputs.2.unified2-alert.xff.deployment = reverse
outputs.2.unified2-alert.xff.header = X-Forwarded-For
outputs.3 = http-log
outputs.3.http-log = (null)
outputs.3.http-log.enabled = no
outputs.3.http-log.filename = http.log
outputs.3.http-log.append = yes
outputs.4 = tls-log
outputs.4.tls-log = (null)
outputs.4.tls-log.enabled = no
outputs.4.tls-log.filename = tls.log
outputs.4.tls-log.append = yes
outputs.5 = tls-store
outputs.5.tls-store = (null)
outputs.5.tls-store.enabled = no
outputs.6 = dns-log
outputs.6.dns-log = (null)
outputs.6.dns-log.enabled = no
outputs.6.dns-log.filename = dns.log
outputs.6.dns-log.append = yes
outputs.7 = pcap-log
outputs.7.pcap-log = (null)
outputs.7.pcap-log.enabled = no
outputs.7.pcap-log.filename = log6.pcap
outputs.7.pcap-log.limit = 1500mb
outputs.7.pcap-log.max-files = 2
outputs.7.pcap-log.mode = normal
outputs.7.pcap-log.use-stream-depth = no
outputs.7.pcap-log.honor-pass-rules = no
outputs.8 = alert-debug
outputs.8.alert-debug = (null)
outputs.8.alert-debug.enabled = no
outputs.8.alert-debug.filename = alert-debug.log
outputs.8.alert-debug.append = yes
outputs.9 = alert-prelude
outputs.9.alert-prelude = (null)
outputs.9.alert-prelude.enabled = no
outputs.9.alert-prelude.profile = suricata
outputs.9.alert-prelude.log-packet-content = no
outputs.9.alert-prelude.log-packet-header = yes
outputs.10 = stats
outputs.10.stats = (null)
outputs.10.stats.enabled = yes
outputs.10.stats.filename = stats.log
outputs.10.stats.append = yes
outputs.10.stats.totals = yes
outputs.10.stats.threads = no
outputs.11 = syslog
outputs.11.syslog = (null)
outputs.11.syslog.enabled = no
outputs.11.syslog.facility = local5
outputs.12 = drop
outputs.12.drop = (null)
outputs.12.drop.enabled = no
outputs.12.drop.filename = drop.log
outputs.12.drop.append = yes
outputs.13 = file-store
outputs.13.file-store = (null)
outputs.13.file-store.version = 2
outputs.13.file-store.enabled = no
outputs.13.file-store.xff = (null)
outputs.13.file-store.xff.enabled = no
outputs.13.file-store.xff.mode = extra-data
outputs.13.file-store.xff.deployment = reverse
outputs.13.file-store.xff.header = X-Forwarded-For
outputs.14 = file-store
outputs.14.file-store = (null)
outputs.14.file-store.enabled = no
outputs.14.file-store.log-dir = files
outputs.14.file-store.force-magic = no
outputs.14.file-store.force-filestore = no
outputs.14.file-store.include-pid = no
outputs.15 = file-log
outputs.15.file-log = (null)
outputs.15.file-log.enabled = no
outputs.15.file-log.filename = files-json.log
outputs.15.file-log.append = yes
outputs.15.file-log.force-magic = no
outputs.16 = tcp-data
outputs.16.tcp-data = (null)
outputs.16.tcp-data.enabled = no
outputs.16.tcp-data.type = file
outputs.16.tcp-data.filename = tcp-data.log
outputs.17 = http-body-data
outputs.17.http-body-data = (null)
outputs.17.http-body-data.enabled = no
outputs.17.http-body-data.type = file
outputs.17.http-body-data.filename = http-data.log
outputs.18 = lua
outputs.18.lua = (null)
outputs.18.lua.enabled = no
outputs.18.lua.scripts =
logging = (null)
logging.default-log-level = notice
logging.default-output-filter =
logging.outputs = (null)
logging.outputs.0 = console
logging.outputs.0.console = (null)
logging.outputs.0.console.enabled = yes
logging.outputs.1 = file
logging.outputs.1.file = (null)
logging.outputs.1.file.enabled = yes
logging.outputs.1.file.level = info
logging.outputs.1.file.filename = /var/log/suricata/suricata.log
logging.outputs.2 = syslog
logging.outputs.2.syslog = (null)
logging.outputs.2.syslog.enabled = no
logging.outputs.2.syslog.facility = local5
logging.outputs.2.syslog.format = [%i] <%d> --
af-packet = (null)
af-packet.0 = interface
af-packet.0.interface = ens192
af-packet.0.threads = 1
af-packet.0.cluster-id = 99
af-packet.0.cluster-type = cluster_flow
af-packet.0.defrag = yes
af-packet.0.use-mmap = yes
af-packet.0.tpacket-v3 = yes
af-packet.0.ring-size = 2048
af-packet.0.block-size = 32768
af-packet.0.copy-mode = ips
af-packet.0.copy-iface = ens224
af-packet.1 = interface
af-packet.1.interface = ens224
af-packet.1.cluster-id = 100
af-packet.1.cluster-type = cluster_flow
af-packet.1.defrag = yes
af-packet.1.threads = 1
af-packet.1.use-mmap = yes
af-packet.1.tpacket-v3 = yes
af-packet.1.ring-size = 2048
af-packet.1.block-size = 32768
af-packet.1.copy-mode = ips
af-packet.1.copy-iface = ens192
af-packet.2 = interface
af-packet.2.interface = default
pcap = (null)
pcap.0 = interface
pcap.0.interface = eth0
pcap.1 = interface
pcap.1.interface = default
pcap-file = (null)
pcap-file.checksum-checks = auto
app-layer = (null)
app-layer.protocols = (null)
app-layer.protocols.krb5 = (null)
app-layer.protocols.krb5.enabled = yes
app-layer.protocols.ikev2 = (null)
app-layer.protocols.ikev2.enabled = yes
app-layer.protocols.tls = (null)
app-layer.protocols.tls.enabled = yes
app-layer.protocols.tls.detection-ports = (null)
app-layer.protocols.tls.detection-ports.dp = 443
app-layer.protocols.tls.ja3-fingerprints = no
app-layer.protocols.dcerpc = (null)
app-layer.protocols.dcerpc.enabled = yes
app-layer.protocols.ftp = (null)
app-layer.protocols.ftp.enabled = yes
app-layer.protocols.ftp.memcap = 64mb
app-layer.protocols.ssh = (null)
app-layer.protocols.ssh.enabled = yes
app-layer.protocols.smtp = (null)
app-layer.protocols.smtp.enabled = yes
app-layer.protocols.smtp.mime = (null)
app-layer.protocols.smtp.mime.decode-mime = yes
app-layer.protocols.smtp.mime.decode-base64 = yes
app-layer.protocols.smtp.mime.decode-quoted-printable = yes
app-layer.protocols.smtp.mime.header-value-depth = 2000
app-layer.protocols.smtp.mime.extract-urls = yes
app-layer.protocols.smtp.mime.body-md5 = no
app-layer.protocols.smtp.inspected-tracker = (null)
app-layer.protocols.smtp.inspected-tracker.content-limit = 100000
app-layer.protocols.smtp.inspected-tracker.content-inspect-min-size = 32768
app-layer.protocols.smtp.inspected-tracker.content-inspect-window = 4096
app-layer.protocols.imap = (null)
app-layer.protocols.imap.enabled = detection-only
app-layer.protocols.msn = (null)
app-layer.protocols.msn.enabled = detection-only
app-layer.protocols.smb = (null)
app-layer.protocols.smb.enabled = yes
app-layer.protocols.smb.detection-ports = (null)
app-layer.protocols.smb.detection-ports.dp = 139, 445
app-layer.protocols.nfs = (null)
app-layer.protocols.nfs.enabled = yes
app-layer.protocols.tftp = (null)
app-layer.protocols.tftp.enabled = yes
app-layer.protocols.dns = (null)
app-layer.protocols.dns.global-memcap = 32mb
app-layer.protocols.dns.state-memcap = 1mb
app-layer.protocols.dns.tcp = (null)
app-layer.protocols.dns.tcp.enabled = yes
app-layer.protocols.dns.tcp.detection-ports = (null)
app-layer.protocols.dns.tcp.detection-ports.dp = 53
app-layer.protocols.dns.udp = (null)
app-layer.protocols.dns.udp.enabled = yes
app-layer.protocols.dns.udp.detection-ports = (null)
app-layer.protocols.dns.udp.detection-ports.dp = 53
app-layer.protocols.http = (null)
app-layer.protocols.http.enabled = yes
app-layer.protocols.http.memcap = 128mb
app-layer.protocols.http.libhtp = (null)
app-layer.protocols.http.libhtp.default-config = (null)
app-layer.protocols.http.libhtp.default-config.personality = IDS
app-layer.protocols.http.libhtp.default-config.request-body-limit = 100kb
app-layer.protocols.http.libhtp.default-config.response-body-limit = 100kb
app-layer.protocols.http.libhtp.default-config.request-body-minimal-inspect-size = 32kb
app-layer.protocols.http.libhtp.default-config.request-body-inspect-window = 4kb
app-layer.protocols.http.libhtp.default-config.response-body-minimal-inspect-size = 40kb
app-layer.protocols.http.libhtp.default-config.response-body-inspect-window = 16kb
app-layer.protocols.http.libhtp.default-config.response-body-decompress-layer-limit = 2
app-layer.protocols.http.libhtp.default-config.http-body-inline = auto
app-layer.protocols.http.libhtp.default-config.swf-decompression = (null)
app-layer.protocols.http.libhtp.default-config.swf-decompression.enabled = yes
app-layer.protocols.http.libhtp.default-config.swf-decompression.type = both
app-layer.protocols.http.libhtp.default-config.swf-decompression.compress-depth = 0
app-layer.protocols.http.libhtp.default-config.swf-decompression.decompress-depth = 0
app-layer.protocols.http.libhtp.default-config.double-decode-path = no
app-layer.protocols.http.libhtp.default-config.double-decode-query = no
app-layer.protocols.http.libhtp.server-config =
app-layer.protocols.modbus = (null)
app-layer.protocols.modbus.enabled = yes
app-layer.protocols.modbus.detection-ports = (null)
app-layer.protocols.modbus.detection-ports.dp = 502
app-layer.protocols.modbus.stream-depth = 0
app-layer.protocols.dnp3 = (null)
app-layer.protocols.dnp3.enabled = yes
app-layer.protocols.dnp3.detection-ports = (null)
app-layer.protocols.dnp3.detection-ports.dp = 20000
app-layer.protocols.enip = (null)
app-layer.protocols.enip.enabled = yes
app-layer.protocols.enip.detection-ports = (null)
app-layer.protocols.enip.detection-ports.dp = 44818
app-layer.protocols.enip.detection-ports.sp = 44818
app-layer.protocols.ntp = (null)
app-layer.protocols.ntp.enabled = yes
app-layer.protocols.dhcp = (null)
app-layer.protocols.dhcp.enabled = yes
asn1-max-frames = 256
pid-file = /var/run/suricata.pid
coredump = (null)
coredump.max-dump = unlimited
host-mode = auto
unix-command = (null)
unix-command.enabled = yes
unix-command.filename = /usr/local/var/run/suricata/suricata-command.socket
legacy = (null)
legacy.uricontent = enabled
engine-analysis = (null)
engine-analysis.rules-fast-pattern = yes
engine-analysis.rules = yes
pcre = (null)
pcre.match-limit = 3500
pcre.match-limit-recursion = 1500
host-os-policy = (null)
host-os-policy.windows = (null)
host-os-policy.windows.0 = 0.0.0.0/0
host-os-policy.bsd = (null)
host-os-policy.bsd-right = (null)
host-os-policy.old-linux = (null)
host-os-policy.linux = (null)
host-os-policy.old-solaris = (null)
host-os-policy.solaris = (null)
host-os-policy.hpux10 = (null)
host-os-policy.hpux11 = (null)
host-os-policy.irix = (null)
host-os-policy.macos = (null)
host-os-policy.vista = (null)
host-os-policy.windows2k3 = (null)
defrag = (null)
defrag.memcap = 32mb
defrag.hash-size = 65536
defrag.trackers = 65535
defrag.max-frags = 65535
defrag.prealloc = yes
defrag.timeout = 60
flow = (null)
flow.memcap = 32mb
flow.hash-size = 65536
flow.prealloc = 10000
flow.emergency-recovery = 30
vlan = (null)
vlan.use-for-tracking = false
flow-timeouts = (null)
flow-timeouts.default = (null)
flow-timeouts.default.new = 30
flow-timeouts.default.established = 140
flow-timeouts.default.closed = 0
flow-timeouts.default.bypassed = 100
flow-timeouts.default.emergency-new = 10
flow-timeouts.default.emergency-established = 100
flow-timeouts.default.emergency-closed = 0
flow-timeouts.default.emergency-bypassed = 50
flow-timeouts.tcp = (null)
flow-timeouts.tcp.new = 50
flow-timeouts.tcp.established = 180
flow-timeouts.tcp.closed = 60
flow-timeouts.tcp.bypassed = 100
flow-timeouts.tcp.emergency-new = 5
flow-timeouts.tcp.emergency-established = 100
flow-timeouts.tcp.emergency-closed = 10
flow-timeouts.tcp.emergency-bypassed = 50
flow-timeouts.udp = (null)
flow-timeouts.udp.new = 30
flow-timeouts.udp.established = 140
flow-timeouts.udp.bypassed = 100
flow-timeouts.udp.emergency-new = 10
flow-timeouts.udp.emergency-established = 100
flow-timeouts.udp.emergency-bypassed = 50
flow-timeouts.icmp = (null)
flow-timeouts.icmp.new = 30
flow-timeouts.icmp.established = 140
flow-timeouts.icmp.bypassed = 100
flow-timeouts.icmp.emergency-new = 10
flow-timeouts.icmp.emergency-established = 100
flow-timeouts.icmp.emergency-bypassed = 50
stream = (null)
stream.memcap = 64mb
stream.checksum-validation = no
stream.inline = yes
stream.drop-invalid = no
stream.reassembly = (null)
stream.reassembly.memcap = 256mb
stream.reassembly.depth = 1mb
stream.reassembly.toserver-chunk-size = 2560
stream.reassembly.toclient-chunk-size = 2560
stream.reassembly.randomize-chunk-size = yes
host = (null)
host.hash-size = 4096
host.prealloc = 1000
host.memcap = 32mb
decoder = (null)
decoder.teredo = (null)
decoder.teredo.enabled = true
detect = (null)
detect.profile = medium
detect.custom-values = (null)
detect.custom-values.toclient-groups = 3
detect.custom-values.toserver-groups = 25
detect.sgh-mpm-context = auto
detect.inspection-recursion-limit = 3000
detect.prefilter = (null)
detect.prefilter.default = mpm
detect.grouping =
detect.profiling = (null)
detect.profiling.grouping = (null)
detect.profiling.grouping.dump-to-disk = false
detect.profiling.grouping.include-rules = false
detect.profiling.grouping.include-mpm-stats = false
mpm-algo = auto
spm-algo = auto
threading = (null)
threading.set-cpu-affinity = yes
threading.cpu-affinity = (null)
threading.cpu-affinity.0 = management-cpu-set
threading.cpu-affinity.0.management-cpu-set = (null)
threading.cpu-affinity.0.management-cpu-set.cpu = (null)
threading.cpu-affinity.0.management-cpu-set.cpu.0 = 1
threading.cpu-affinity.0.management-cpu-set.cpu.1 = 2
threading.cpu-affinity.1 = receive-cpu-set
threading.cpu-affinity.1.receive-cpu-set = (null)
threading.cpu-affinity.1.receive-cpu-set.cpu = (null)
threading.cpu-affinity.1.receive-cpu-set.cpu.0 = 1
threading.cpu-affinity.1.receive-cpu-set.cpu.1 = 2
threading.cpu-affinity.2 = worker-cpu-set
threading.cpu-affinity.2.worker-cpu-set = (null)
threading.cpu-affinity.2.worker-cpu-set.cpu = (null)
threading.cpu-affinity.2.worker-cpu-set.cpu.0 = 1
threading.cpu-affinity.2.worker-cpu-set.cpu.1 = 2
threading.cpu-affinity.2.worker-cpu-set.mode = exclusive
threading.cpu-affinity.2.worker-cpu-set.threads = 1
threading.cpu-affinity.2.worker-cpu-set.prio = (null)
threading.cpu-affinity.2.worker-cpu-set.prio.low = (null)
threading.cpu-affinity.2.worker-cpu-set.prio.low.0 = 0
threading.cpu-affinity.2.worker-cpu-set.prio.medium = (null)
threading.cpu-affinity.2.worker-cpu-set.prio.medium.0 = 1-2
threading.cpu-affinity.2.worker-cpu-set.prio.high = (null)
threading.cpu-affinity.2.worker-cpu-set.prio.high.0 = 3
threading.cpu-affinity.2.worker-cpu-set.prio.default = medium
threading.detect-thread-ratio = 1.0
luajit = (null)
luajit.states = 128
profiling = (null)
profiling.rules = (null)
profiling.rules.enabled = yes
profiling.rules.filename = rule_perf.log
profiling.rules.append = yes
profiling.rules.limit = 10
profiling.rules.json = yes
profiling.keywords = (null)
profiling.keywords.enabled = yes
profiling.keywords.filename = keyword_perf.log
profiling.keywords.append = yes
profiling.prefilter = (null)
profiling.prefilter.enabled = yes
profiling.prefilter.filename = prefilter_perf.log
profiling.prefilter.append = yes
profiling.rulegroups = (null)
profiling.rulegroups.enabled = yes
profiling.rulegroups.filename = rule_group_perf.log
profiling.rulegroups.append = yes
profiling.packets = (null)
profiling.packets.enabled = yes
profiling.packets.filename = packet_stats.log
profiling.packets.append = yes
profiling.packets.csv = (null)
profiling.packets.csv.enabled = no
profiling.packets.csv.filename = packet_stats.csv
profiling.locks = (null)
profiling.locks.enabled = no
profiling.locks.filename = lock_stats.log
profiling.locks.append = yes
profiling.pcap-log = (null)
profiling.pcap-log.enabled = no
profiling.pcap-log.filename = pcaplog_stats.log
profiling.pcap-log.append = yes
nfq =
nflog = (null)
nflog.0 = group
nflog.0.group = 2
nflog.0.buffer-size = 18432
nflog.1 = group
nflog.1.group = default
nflog.1.qthreshold = 1
nflog.1.qtimeout = 100
nflog.1.max-size = 20000
capture = (null)
capture.disable-offloading = true
netmap = (null)
netmap.0 = interface
netmap.0.interface = 0
netmap.0.threads = 1
netmap.0.copy-mode = ips
netmap.0.copy-iface = 1
netmap.1 = interface
netmap.1.interface = 1
netmap.1.threads = 1
netmap.1.copy-mode = ips
netmap.1.copy-iface = 0
netmap.2 = interface
netmap.2.interface = default
pfring = (null)
pfring.0 = interface
pfring.0.interface = eth0
pfring.0.threads = auto
pfring.0.cluster-id = 99
pfring.0.cluster-type = cluster_flow
pfring.1 = interface
pfring.1.interface = default
ipfw =
napatech = (null)
napatech.hba = -1
napatech.use-all-streams = yes
napatech.streams = (null)
napatech.streams.0 = 0-3
mpipe = (null)
mpipe.load-balance = dynamic
mpipe.iqueue-packets = 2048
mpipe.inputs = (null)
mpipe.inputs.0 = interface
mpipe.inputs.0.interface = xgbe2
mpipe.inputs.1 = interface
mpipe.inputs.1.interface = xgbe3
mpipe.inputs.2 = interface
mpipe.inputs.2.interface = xgbe4
mpipe.stack = (null)
mpipe.stack.size128 = 0
mpipe.stack.size256 = 9
mpipe.stack.size512 = 0
mpipe.stack.size1024 = 0
mpipe.stack.size1664 = 7
mpipe.stack.size4096 = 0
mpipe.stack.size10386 = 0
mpipe.stack.size16384 = 0
default-rule-path = /home/user/rules/
rule-files = (null)
rule-files.0 = smb.rules
classification-file = /home/user/rules/classification.config
reference-config-file = /etc/suricata/reference.config
threshold-file = /etc/suricata/threshold.config


Files

SMB-Suricata-6.0.3-large-ubuntu.gif (265 KB) SMB-Suricata-6.0.3-large-ubuntu.gif Srini J, 11/19/2021 02:47 PM
config.yaml (73.7 KB) config.yaml Srini J, 11/19/2021 08:34 PM

Related issues 3 (1 open2 closed)

Related to Feature #4861: smb: support multi-stream file transfersIn ReviewPhilippe AntoineActions
Copied to Bug #5136: smb: excessive memory use during file transferClosedShivani BhardwajActions
Copied to Bug #5137: smb: excessive memory use during file transferClosedShivani BhardwajActions
Actions #1

Updated by Victor Julien 10 months ago

I would like to get access to this pcap. How can we arrange this?

Actions #2

Updated by Victor Julien 10 months ago

  • Subject changed from Memory: Run away memory allocations in Suricata-6.0.3 or Suricata-6.0.4 at suricata::filetracker::FileTransferTracker::update()->::extend() while transferring files between Windows 10 clients over SMB to smb: excessive memory use during file transfer
Actions #3

Updated by Srini J 10 months ago

Hello Victor, Please let me know the preferred way to share the file. It's a 412Mb file (it's a pcap of the partial file transfer). Pcap was captured using pcap-log option of Suricata.

Actions #4

Updated by Victor Julien 10 months ago

A google drive or dropbox or similar link is probably easiest. Assuming the file has to remain private we should probably switch to email vjulien@oisf.net. Thanks!

Actions #5

Updated by Srini J 10 months ago

Hello Victor.
Have sent you a OneDrive link. The md5sum of the file for reference.

md5sum /var/log/suricata/log6.pcap.7z
7c6f5a1076d3b32b838814df8a1c1eaf /var/log/suricata/log6.pcap.7z

Regards,
Srini

Actions #6

Updated by Victor Julien 10 months ago

I received the file, thanks.

Can you also reproduce this by reading the pcap in Suricata directly? Using -r.

If not, how are you starting Suricata and how are you replaying the pcap?

Actions #7

Updated by Srini J 10 months ago

Yes the memory increase did seem to happen with the pcap using the -r option as well. Used jemalloc to confirm the allocation path.

Actions #8

Updated by Srini J 10 months ago

Including the used Suricata config.yaml as well for easy reference.

Actions #9

Updated by Srini J 10 months ago

Commands used to compile and install jemalloc-5.2.1 on the system running Suricata
cd /home/user/source/jemalloc-5.2.1
./autogen.sh
./configure --enable-prof
make
make install

Commands used for sample run using jemalloc + Suricata + pcap

#pwd
#/home/user/source/suricata/suricata-6.0.4
MALLOC_CONF="prof:true,prof_prefix:/tmp/suricata64p_mem.out,lg_prof_interval:28" LD_PRELOAD=/usr/local/lib/libjemalloc.so LD_LIBRARY_PATH=/home/user/source/suricata/suricata-6.0.4/libhtp/htp/.libs/ ./src/.libs/suricata -c /etc/suricata/config.yaml -r /var/log/suricata/log6.pcap

Post run:
ls -ltrh /tmp/suricata64p*

rw-r--r- 1 user user 13K Nov 20 09:02 suricata64p_mem.out.20177.0.i0.heap
rw-r--r- 1 user user 13K Nov 20 09:02 suricata64p_mem.out.20177.1.i1.heap
rw-r--r- 1 user user 20K Nov 20 09:02 suricata64p_mem.out.20177.2.i2.heap
rw-r--r- 1 user user 20K Nov 20 09:02 suricata64p_mem.out.20177.3.i3.heap
rw-r--r- 1 user user 20K Nov 20 09:02 suricata64p_mem.out.20177.4.i4.heap
rw-r--r- 1 user user 20K Nov 20 09:02 suricata64p_mem.out.20177.5.i5.heap
rw-r--r- 1 user user 20K Nov 20 09:02 suricata64p_mem.out.20177.6.i6.heap
rw-r--r- 1 user user 20K Nov 20 09:02 suricata64p_mem.out.20177.7.i7.heap

Memory log analysis:
jeprof --show_bytes --gif /home/user/source/suricata/suricata-6.0.4/src/.libs/suricata --base /tmp/suricata64p_mem.out.20177.0.i0.heap /tmp/suricata64p_mem.out.20177.7.i7.heap > /tmp/Suricata64PcapMmap.gif

Actions #10

Updated by Victor Julien 10 months ago

Here's my analysis so far:
  • the file transfer is distributed over multiple parallel TCP connections
  • the out order chunk handling isn't expecting this, and is queueing the chunks until an in-order chunk is received
  • since the chunks are distributed over the connections, they are all missing "in-order" chunks and thus keep queueing
  • no limits are imposed on this queueing, so memory use grows w/o limit

I think in the short term we need to address the last point, so that queueing won't exceed reasonable limits. Supporting file transfers over multiple TCP connections would be a more thorough solution, but this will be a non-trivial addition.

Actions #11

Updated by Srini J 10 months ago

Thank you for the analysis Victor. Is there a work around for this issue in Suricata (other than disabling SMB in Suricata config)? If I understood it correctly, setting a stream-depth size for SMB, would help, but once the stream-depth is reached, all the remaining files would not be scanned in a multi file transfer session. Please correct me as needed.

Actions #12

Updated by Victor Julien 10 months ago

Yes, I think that would work as a workaround with the drawbacks you mentioned. I think short term we'll impose some limits. But a real solution needs to come with supporting these multi-stream file transfers.

Actions #13

Updated by Victor Julien 10 months ago

  • Related to Feature #4861: smb: support multi-stream file transfers added
Actions #14

Updated by Victor Julien 8 months ago

  • Target version set to 7.0rc1
  • Label Needs backport to 5.0, Needs backport to 6.0 added
Actions #16

Updated by Victor Julien 8 months ago

  • Status changed from New to Assigned
  • Assignee set to Victor Julien
Actions #17

Updated by Jeff Lucovsky 7 months ago

  • Copied to Bug #5136: smb: excessive memory use during file transfer added
Actions #18

Updated by Jeff Lucovsky 7 months ago

  • Copied to Bug #5137: smb: excessive memory use during file transfer added
Actions #19

Updated by Victor Julien 5 months ago

  • Status changed from Assigned to In Progress
  • Priority changed from Urgent to High

Work around resource limits: https://github.com/OISF/suricata/pull/7254

Actions #20

Updated by Victor Julien 5 months ago

  • Status changed from In Progress to Closed
  • Priority changed from High to Normal
Actions

Also available in: Atom PDF