Bug #2186
closedsmb dcerpc segfaults in StubDataParser
Description
Suricata is segfaulting during a memcpy in app-layer-dcerpc.c StubDataParser when long running smb flows contain dcerpc traffic. I think the stub_data_buffer_len variable is the culprit.
I modified the app-layer-smb.c DataParser, app-layer-dcerpc.c DCERPCParser and StubDataParser methods to pass along the flow_hash and it looks something like this:
(printing in StubDataParser)
13/7/2017 -- 20:35:02 - <Error> - [ERRCODE: SC_ERR_DCERPC(172)] - flow_hash: 459222427, stub_data_buffer_len: 0, dcerpc->fragged: 0, dcerpc->transaction_id: 1 13/7/2017 -- 20:35:02 - <Error> - [ERRCODE: SC_ERR_DCERPC(172)] - flow_hash: 459222427, stub_data_buffer_len: 1000, dcerpc->fragged: 1, dcerpc->transaction_id: 2 13/7/2017 -- 20:35:02 - <Error> - [ERRCODE: SC_ERR_DCERPC(172)] - flow_hash: 459222427, stub_data_buffer_len: 2024, dcerpc->fragged: 1, dcerpc->transaction_id: 3 13/7/2017 -- 20:35:02 - <Error> - [ERRCODE: SC_ERR_DCERPC(172)] - flow_hash: 459222427, stub_data_buffer_len: 3048, dcerpc->fragged: 1, dcerpc->transaction_id: 4 ... 13/7/2017 -- 20:36:09 - <Error> - [ERRCODE: SC_ERR_DCERPC(172)] - flow_hash: 459222427, stub_data_buffer_len: 15302088, dcerpc->fragged: 1, dcerpc->transaction_id: 18626 13/7/2017 -- 20:36:09 - <Error> - [ERRCODE: SC_ERR_DCERPC(172)] - flow_hash: 459222427, stub_data_buffer_len: 15302272, dcerpc->fragged: 0, dcerpc->transaction_id: 18627 13/7/2017 -- 20:36:09 - <Error> - [ERRCODE: SC_ERR_DCERPC(172)] - flow_hash: 459222427, stub_data_buffer_len: 15303272, dcerpc->fragged: 1, dcerpc->transaction_id: 18628
Printing the stub_data_buffer_len with gdb from the coredump shows that it eventually reaches 4294967256. This value was consistent across multiple core dumps.
$ gdb /usr/local/bin/suricata core.10416
...
bt full 2
#0 0x00007febd6540f8e in __memcpy_ssse3_back () from /lib64/libc.so.6
No symbol table info available.
#1 0x000000000041837a in StubDataParser (dcerpc=dcerpc@entry=0x7feb3fddf7b8, input=input@entry=0x7feb04a9f5d0 "", input_len=input_len@entry=1024) at app-layer-dcerpc.c:1236
stub_data_buffer = 0x7feb3fddf830
stub_data_buffer_len = 0x7feb3fddf838
stub_data_fresh = 0x7feb3fddf83c ""
stub_len = 1024
ptmp = <optimized out>
__FUNCTION__ = "StubDataParser"
(More stack frames follow...)
x/u 0x7feb3fddf838
0x7feb3fddf838: 4294967256
I have tried taking a pcap of a 5 minute interval around when the segfault occurs, but reading or replaying it to Suricata does not recreate the error. I think this only occurs after a long time of building up state, about once every 24 hours in my case.
Setting smb to "detection-only" in the yaml stops the segfault from occurring.
While trying to debug this I also noticed that DataParser in app-layer-smb.c checks the return value of DCERPCParser against -1, although it seems that DCERPCParser only returns 0 on failure. But changing this did not stop the segfaults.
I can't share a pcap or coredump sadly, sorry, but I attached gdb.txt with some thread info.
Running Suricata with pf_ring with this build-info:
$ suricata --build-info This is Suricata version 3.2 RELEASE Features: PCAP_SET_BUFF LIBPCAP_VERSION_MAJOR=1 PF_RING AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_NSS HAVE_LIBJANSSON TLS SIMD support: SSE_4_2 SSE_4_1 SSE_3 Atomic intrisics: 1 2 4 8 16 byte(s) 64-bits, Little-endian architecture GCC version 4.8.5 20150623 (Red Hat 4.8.5-11), C version 199901 compiled with _FORTIFY_SOURCE=0 L1 cache line size (CLS)=64 thread local storage method: __thread compiled with LibHTP v0.5.23, linked against LibHTP v0.5.23 Suricata Configuration: AF_PACKET support: yes PF_RING support: yes NFQueue support: no NFLOG support: no IPFW support: no Netmap support: no DAG enabled: no Napatech enabled: no Unix socket enabled: yes Detection enabled: yes libnss support: yes libnspr support: yes libjansson support: yes hiredis support: no Prelude support: no PCRE jit: yes LUA support: no libluajit: no libgeoip: no Non-bundled htp: no Old barnyard2 support: no CUDA enabled: no Hyperscan support: no Libnet support: no Suricatasc install: yes Profiling enabled: no Profiling locks enabled: no Development settings: Coccinelle / spatch: no Unit tests enabled: no Debug output enabled: no Debug validation enabled: no Generic build parameters: Installation prefix: /usr/local Configuration directory: /etc/suricata/ Log directory: /var/log/suricata/ --prefix /usr/local --sysconfdir /etc --localstatedir /var Host: x86_64-pc-linux-gnu Compiler: gcc (exec name) / gcc (real) GCC Protect enabled: no GCC march native enabled: yes GCC Profile enabled: no Position Independent Executable enabled: no CFLAGS -g -O2 -march=native PCAP_CFLAGS SECCFLAGS
Files
Updated by Andreas Herz over 7 years ago
- Assignee set to OISF Dev
- Target version set to TBD
Updated by Victor Julien over 7 years ago
- Status changed from New to Assigned
- Assignee changed from OISF Dev to Victor Julien
- Target version changed from TBD to 70
Updated by Victor Julien over 7 years ago
Jack, in frame 1 can you also 'print *dcerpc'?
Updated by Victor Julien over 7 years ago
Are you able to give this pull request a test? https://github.com/inliniac/suricata/pull/2856
Updated by Victor Julien over 7 years ago
- Status changed from Assigned to Closed
- Target version changed from 70 to 4.0.0
I merged https://github.com/inliniac/suricata/pull/2856 into 4.0. Assuming this fixed. Please reopen the ticket if otherwise.
Updated by Jack Covington over 7 years ago
Hello Victor,
Sorry for the late reply. Here is the 'print *dcerpc' output in case you need it in the future:
(gdb) frame 1
#1 0x000000000041837a in StubDataParser (dcerpc=dcerpc@entry=0x7feb3fddf7b8, input=input@entry=0x7feb04a9f5d0 "", input_len=input_len@entry=1024) at app-layer-dcerpc.c:1236
1236 in app-layer-dcerpc.c
(gdb) print *dcerpc
$1 = {dcerpchdr = {rpc_vers = 5 '\005', rpc_vers_minor = 0 '\000', type = 2 '\002', pfc_flags = 0 '\000', packed_drep = "\020\000\000", frag_length = 4280, auth_length = 0, call_id = 365833216}, dcerpcbindbindack = {
numctxitems = 0 '\000', numctxitemsleft = 0 '\000', ctxbytesprocessed = 0 '\000', ctxid = 0, uuid = '\000' <repeats 15 times>, version = 0, versionminor = 0, uuid_entry = 0x0, uuid_list = {tqh_first = 0x0,
tqh_last = 0x7feb3fddf7f0}, accepted_uuid_list = {tqh_first = 0x0, tqh_last = 0x7feb3fddf800}, uuid_internal_id = 0, secondaryaddrlen = 0, secondaryaddrlenleft = 0, result = 0}, dcerpcrequest = {ctxid = 0, opnum = 0,
stub_data_buffer = 0x0, stub_data_buffer_len = 0, stub_data_fresh = 0 '\000', first_request_seen = 0 '\000'}, dcerpcresponse = {stub_data_buffer = 0x7fe883ffd010 "", stub_data_buffer_len = 4294967256,
stub_data_fresh = 0 '\000'}, bytesprocessed = 1024, pad = 0 '\000', padleft = 3256, transaction_id = 50613, pdu_fragged = 1 '\001'}
(gdb) print *input
$2 = 0 '\000'
I will test your changes and reopen the ticket if there are any issues.
Thank you for your help.
Updated by Jack Covington over 7 years ago
Been running for 120+ hours with no errors. This fixed the issue.