Project

General

Profile

Actions

Bug #2186

closed

smb dcerpc segfaults in StubDataParser

Added by Jack Covington over 6 years ago. Updated over 6 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Target version:
Affected Versions:
Effort:
Difficulty:
Label:

Description

Suricata is segfaulting during a memcpy in app-layer-dcerpc.c StubDataParser when long running smb flows contain dcerpc traffic. I think the stub_data_buffer_len variable is the culprit.

I modified the app-layer-smb.c DataParser, app-layer-dcerpc.c DCERPCParser and StubDataParser methods to pass along the flow_hash and it looks something like this:
(printing in StubDataParser)

13/7/2017 -- 20:35:02 - <Error> - [ERRCODE: SC_ERR_DCERPC(172)] - flow_hash: 459222427, stub_data_buffer_len: 0, dcerpc->fragged: 0, dcerpc->transaction_id: 1
13/7/2017 -- 20:35:02 - <Error> - [ERRCODE: SC_ERR_DCERPC(172)] - flow_hash: 459222427, stub_data_buffer_len: 1000, dcerpc->fragged: 1, dcerpc->transaction_id: 2
13/7/2017 -- 20:35:02 - <Error> - [ERRCODE: SC_ERR_DCERPC(172)] - flow_hash: 459222427, stub_data_buffer_len: 2024, dcerpc->fragged: 1, dcerpc->transaction_id: 3
13/7/2017 -- 20:35:02 - <Error> - [ERRCODE: SC_ERR_DCERPC(172)] - flow_hash: 459222427, stub_data_buffer_len: 3048, dcerpc->fragged: 1, dcerpc->transaction_id: 4
...
13/7/2017 -- 20:36:09 - <Error> - [ERRCODE: SC_ERR_DCERPC(172)] - flow_hash: 459222427, stub_data_buffer_len: 15302088, dcerpc->fragged: 1, dcerpc->transaction_id: 18626
13/7/2017 -- 20:36:09 - <Error> - [ERRCODE: SC_ERR_DCERPC(172)] - flow_hash: 459222427, stub_data_buffer_len: 15302272, dcerpc->fragged: 0, dcerpc->transaction_id: 18627
13/7/2017 -- 20:36:09 - <Error> - [ERRCODE: SC_ERR_DCERPC(172)] - flow_hash: 459222427, stub_data_buffer_len: 15303272, dcerpc->fragged: 1, dcerpc->transaction_id: 18628

Printing the stub_data_buffer_len with gdb from the coredump shows that it eventually reaches 4294967256. This value was consistent across multiple core dumps.

$ gdb /usr/local/bin/suricata core.10416
...
bt full 2
#0  0x00007febd6540f8e in __memcpy_ssse3_back () from /lib64/libc.so.6
No symbol table info available.
#1  0x000000000041837a in StubDataParser (dcerpc=dcerpc@entry=0x7feb3fddf7b8, input=input@entry=0x7feb04a9f5d0 "", input_len=input_len@entry=1024) at app-layer-dcerpc.c:1236
        stub_data_buffer = 0x7feb3fddf830
        stub_data_buffer_len = 0x7feb3fddf838
        stub_data_fresh = 0x7feb3fddf83c "" 
        stub_len = 1024
        ptmp = <optimized out>
        __FUNCTION__ = "StubDataParser" 
(More stack frames follow...)

x/u 0x7feb3fddf838
0x7feb3fddf838: 4294967256

I have tried taking a pcap of a 5 minute interval around when the segfault occurs, but reading or replaying it to Suricata does not recreate the error. I think this only occurs after a long time of building up state, about once every 24 hours in my case.

Setting smb to "detection-only" in the yaml stops the segfault from occurring.
While trying to debug this I also noticed that DataParser in app-layer-smb.c checks the return value of DCERPCParser against -1, although it seems that DCERPCParser only returns 0 on failure. But changing this did not stop the segfaults.

I can't share a pcap or coredump sadly, sorry, but I attached gdb.txt with some thread info.

Running Suricata with pf_ring with this build-info:

$ suricata --build-info
This is Suricata version 3.2 RELEASE
Features: PCAP_SET_BUFF LIBPCAP_VERSION_MAJOR=1 PF_RING AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_NSS HAVE_LIBJANSSON TLS
SIMD support: SSE_4_2 SSE_4_1 SSE_3
Atomic intrisics: 1 2 4 8 16 byte(s)
64-bits, Little-endian architecture
GCC version 4.8.5 20150623 (Red Hat 4.8.5-11), C version 199901
compiled with _FORTIFY_SOURCE=0
L1 cache line size (CLS)=64
thread local storage method: __thread
compiled with LibHTP v0.5.23, linked against LibHTP v0.5.23

Suricata Configuration:
  AF_PACKET support:                       yes
  PF_RING support:                         yes
  NFQueue support:                         no
  NFLOG support:                           no
  IPFW support:                            no
  Netmap support:                          no
  DAG enabled:                             no
  Napatech enabled:                        no

  Unix socket enabled:                     yes
  Detection enabled:                       yes

  libnss support:                          yes
  libnspr support:                         yes
  libjansson support:                      yes
  hiredis support:                         no
  Prelude support:                         no
  PCRE jit:                                yes
  LUA support:                             no
  libluajit:                               no
  libgeoip:                                no
  Non-bundled htp:                         no
  Old barnyard2 support:                   no
  CUDA enabled:                            no
  Hyperscan support:                       no
  Libnet support:                          no

  Suricatasc install:                      yes

  Profiling enabled:                       no
  Profiling locks enabled:                 no

Development settings:
  Coccinelle / spatch:                     no
  Unit tests enabled:                      no
  Debug output enabled:                    no
  Debug validation enabled:                no

Generic build parameters:
  Installation prefix:                     /usr/local
  Configuration directory:                 /etc/suricata/
  Log directory:                           /var/log/suricata/

  --prefix                                 /usr/local
  --sysconfdir                             /etc
  --localstatedir                          /var

  Host:                                    x86_64-pc-linux-gnu
  Compiler:                                gcc (exec name) / gcc (real)
  GCC Protect enabled:                     no
  GCC march native enabled:                yes
  GCC Profile enabled:                     no
  Position Independent Executable enabled: no
  CFLAGS                                   -g -O2 -march=native
  PCAP_CFLAGS
  SECCFLAGS


Files

gdb.txt (20.4 KB) gdb.txt Output from "thread apply all bt" Jack Covington, 07/18/2017 10:35 AM
Actions #1

Updated by Andreas Herz over 6 years ago

  • Assignee set to OISF Dev
  • Target version set to TBD
Actions #2

Updated by Victor Julien over 6 years ago

  • Status changed from New to Assigned
  • Assignee changed from OISF Dev to Victor Julien
  • Target version changed from TBD to 70
Actions #3

Updated by Victor Julien over 6 years ago

Jack, in frame 1 can you also 'print *dcerpc'?

Actions #4

Updated by Victor Julien over 6 years ago

Are you able to give this pull request a test? https://github.com/inliniac/suricata/pull/2856

Actions #5

Updated by Victor Julien over 6 years ago

  • Status changed from Assigned to Closed
  • Target version changed from 70 to 4.0.0

I merged https://github.com/inliniac/suricata/pull/2856 into 4.0. Assuming this fixed. Please reopen the ticket if otherwise.

Actions #6

Updated by Jack Covington over 6 years ago

Hello Victor,

Sorry for the late reply. Here is the 'print *dcerpc' output in case you need it in the future:

(gdb) frame 1
#1  0x000000000041837a in StubDataParser (dcerpc=dcerpc@entry=0x7feb3fddf7b8, input=input@entry=0x7feb04a9f5d0 "", input_len=input_len@entry=1024) at app-layer-dcerpc.c:1236
1236    in app-layer-dcerpc.c
(gdb) print *dcerpc
$1 = {dcerpchdr = {rpc_vers = 5 '\005', rpc_vers_minor = 0 '\000', type = 2 '\002', pfc_flags = 0 '\000', packed_drep = "\020\000\000", frag_length = 4280, auth_length = 0, call_id = 365833216}, dcerpcbindbindack = {
    numctxitems = 0 '\000', numctxitemsleft = 0 '\000', ctxbytesprocessed = 0 '\000', ctxid = 0, uuid = '\000' <repeats 15 times>, version = 0, versionminor = 0, uuid_entry = 0x0, uuid_list = {tqh_first = 0x0,
      tqh_last = 0x7feb3fddf7f0}, accepted_uuid_list = {tqh_first = 0x0, tqh_last = 0x7feb3fddf800}, uuid_internal_id = 0, secondaryaddrlen = 0, secondaryaddrlenleft = 0, result = 0}, dcerpcrequest = {ctxid = 0, opnum = 0,
    stub_data_buffer = 0x0, stub_data_buffer_len = 0, stub_data_fresh = 0 '\000', first_request_seen = 0 '\000'}, dcerpcresponse = {stub_data_buffer = 0x7fe883ffd010 "", stub_data_buffer_len = 4294967256,
    stub_data_fresh = 0 '\000'}, bytesprocessed = 1024, pad = 0 '\000', padleft = 3256, transaction_id = 50613, pdu_fragged = 1 '\001'}
(gdb) print *input
$2 = 0 '\000'

I will test your changes and reopen the ticket if there are any issues.

Thank you for your help.

Actions #7

Updated by Jack Covington over 6 years ago

Been running for 120+ hours with no errors. This fixed the issue.

Actions

Also available in: Atom PDF