Project

General

Profile

Actions

Bug #2090

closed

Rule-reload in multi-tenancy is buggy

Added by Antti Tönkyrä over 5 years ago. Updated about 4 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Target version:
Affected Versions:
Effort:
Difficulty:
Label:

Description

Observed version 3.2.1-1 (Debian Stretch)

I have observed several failure cases when performing rule reloads in multi-tenancy setup which may or may not be the result of same underlying bug:

Case 1: Suricata needs rule-reload before tenants can be reloaded safely

# systemctl start suricata
# gdb attach --pid=suripid
( in another window ) # /usr/bin/suricatasc -c 'reload-tenant 1001 /etc/suricata/tenants/tenant-test.yaml'
Thread 15 "US" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7f5a20ff9700 (LWP 12865)]
0x0000000000000000 in ?? ()
(gdb) bt
#0  0x0000000000000000 in ?? ()
#1  0x000055fa66f56ef9 in ThreadCtxDoInit (de_ctx=de_ctx@entry=0x55fa6955ec30, det_ctx=det_ctx@entry=0x7f5a04002320) at detect-engine.c:1228
#2  0x000055fa66f57dc6 in DetectEngineThreadCtxInitForReload (tv=tv@entry=0x55fa69571f80, new_de_ctx=new_de_ctx@entry=0x55fa6955ec30, mt=mt@entry=1) at detect-engine.c:1398
#3  0x000055fa66f5b296 in DetectEngineReloadThreads (new_de_ctx=0x55fa6955ec30) at detect-engine.c:460
#4  0x000055fa66f5e6e8 in DetectEngineMTApply () at detect-engine.c:2454
#5  0x000055fa67016c7e in UnixSocketReloadTenant (cmd=<optimized out>, answer=0x7f5a04000f10, data=<optimized out>) at runmode-unix-socket.c:691
#6  0x000055fa6704d576 in UnixCommandExecute ...

However, if reload happens beforehand everything is fine and tenant can be reloaded many times

systemctl start suricata
# /usr/bin/suricatasc -c reload-rules
{"message": "done", "return": "OK"}
# /usr/bin/suricatasc -c 'reload-tenant 1001 /etc/suricata/tenants/tenant-test.yaml'
{"message": "reloading tenant succeeded", "return": "OK"}
# /usr/bin/suricatasc -c 'reload-tenant 1001 /etc/suricata/tenants/tenant-test.yaml'
{"message": "reloading tenant succeeded", "return": "OK"}
# /usr/bin/suricatasc -c 'reload-tenant 1001 /etc/suricata/tenants/tenant-test.yaml'
{"message": "reloading tenant succeeded", "return": "OK"}
# /usr/bin/suricatasc -c 'reload-tenant 1001 /etc/suricata/tenants/tenant-test.yaml'
{"message": "reloading tenant succeeded", "return": "OK"}

However, in rapid succession of rule-reload and tenant reload, the tenant breaks:

# systemctl restart suricata
# /usr/bin/suricatasc -c reload-rules
{"message": "done", "return": "OK"}
#  /usr/bin/suricatasc -c 'reload-tenant 1001 /etc/suricata/tenants/tenant-test.yaml'
{"message": "reloading tenant succeeded", "return": "OK"}
#  /usr/bin/suricatasc -c 'reload-tenant 1001 /etc/suricata/tenants/tenant-test.yaml'
{"message": "reloading tenant succeeded", "return": "OK"}
# /usr/bin/suricatasc -c reload-rules; /usr/bin/suricatasc -c 'reload-tenant 1001 /etc/suricata/tenants/tenant-test.yaml'
{"message": "done", "return": "OK"}
{"message": "reload tenant failed", "return": "NOK"}
# /usr/bin/suricatasc -c reload-rules; /usr/bin/suricatasc -c 'reload-tenant 1001 /etc/suricata/tenants/tenant-test.yaml'
{"message": "done", "return": "OK"}
{"message": "reload tenant failed", "return": "NOK"}
#  /usr/bin/suricatasc -c 'reload-tenant 1001 /etc/suricata/tenants/tenant-test.yaml'
{"message": "reload tenant failed", "return": "NOK"}


Related issues 1 (0 open1 closed)

Related to Bug #2518: Tenant rules reload completely broken in 4.x.xClosedVictor JulienActions
Actions #1

Updated by Antti Tönkyrä over 5 years ago

From debian sources, detect-engine.c:1228 looks to be

    PatternMatchThreadPrepare(&det_ctx->mtc, de_ctx->mpm_matcher);

Actions #2

Updated by Peter Manev over 5 years ago

What does your suricata.log look like with the rapid succession reloads ?
I think I have tried that before and it was simply saying (suricata.log) that it is in the middle of reload so wont do any new one till the current one is finished - is that the case?

Actions #3

Updated by Antti Tönkyrä over 5 years ago

suricata.log when run with -vv

-- RULE RELOAD --
7/4/2017 -- 15:40:51 - <Notice> - rule reload starting
7/4/2017 -- 15:40:51 - <Warning> - [ERRCODE: SC_ERR_NO_RULES_LOADED(43)] - 1 rule files specified, but no rule was loaded at all!
7/4/2017 -- 15:40:51 - <Info> - 0 signatures processed. 0 are IP-only rules, 0 are inspecting packet payload, 0 inspect application layer, 0 are decoder event
only
7/4/2017 -- 15:40:51 - <Info> - Threshold config parsed: 0 rule(s) found
7/4/2017 -- 15:40:51 - <Info> - cleaning up signature grouping structure... complete
7/4/2017 -- 15:40:51 - <Notice> - rule reload complete

-- TENANT RELOAD --
7/4/2017 -- 15:41:00 - <Info> - prefix multi-detect.1001.reload.1
7/4/2017 -- 15:41:00 - <Info> - Including configuration file /etc/suricata/tenants/tenant-base.yaml.
7/4/2017 -- 15:41:00 - <Info> - Including configuration file /etc/suricata/tenants/tenant-base.yaml.
7/4/2017 -- 15:41:00 - <Info> - Configuration node 'classification-file' redefined.
7/4/2017 -- 15:41:00 - <Info> - Configuration node 'reference-config-file' redefined.
7/4/2017 -- 15:41:00 - <Info> - Configuration node 'vars' redefined.
7/4/2017 -- 15:41:00 - <Info> - Configuration node 'default-rule-path' redefined.
7/4/2017 -- 15:41:00 - <Info> - Configuration node 'rule-files' redefined.
7/4/2017 -- 15:41:00 - <Warning> - [ERRCODE: SC_ERR_NO_RULES_LOADED(43)] - 2 rule files specified, but no rule was loaded at all!
7/4/2017 -- 15:41:00 - <Info> - 0 signatures processed. 0 are IP-only rules, 0 are inspecting packet payload, 0 inspect application layer, 0 are decoder event
only
7/4/2017 -- 15:41:00 - <Info> - cleaning up signature grouping structure... complete

-- ANOTHER TENANT RELOAD --
7/4/2017 -- 15:41:07 - <Info> - prefix multi-detect.1001.reload.2
7/4/2017 -- 15:41:07 - <Info> - Including configuration file /etc/suricata/tenants/tenant-base.yaml.
7/4/2017 -- 15:41:07 - <Info> - Including configuration file /etc/suricata/tenants/tenant-base.yaml.
7/4/2017 -- 15:41:07 - <Info> - Configuration node 'classification-file' redefined.
7/4/2017 -- 15:41:07 - <Info> - Configuration node 'reference-config-file' redefined.
7/4/2017 -- 15:41:07 - <Info> - Configuration node 'vars' redefined.
7/4/2017 -- 15:41:07 - <Info> - Configuration node 'default-rule-path' redefined.
7/4/2017 -- 15:41:07 - <Info> - Configuration node 'rule-files' redefined.
7/4/2017 -- 15:41:07 - <Warning> - [ERRCODE: SC_ERR_NO_RULES_LOADED(43)] - 2 rule files specified, but no rule was loaded at all!
7/4/2017 -- 15:41:07 - <Info> - 0 signatures processed. 0 are IP-only rules, 0 are inspecting packet payload, 0 inspect application layer, 0 are decoder event only
7/4/2017 -- 15:41:07 - <Info> - cleaning up signature grouping structure... complete

-- RAPID RULE-TENANT-RELOAD -- (rule reload; tenant reload)
7/4/2017 -- 15:41:16 - <Notice> - rule reload starting
7/4/2017 -- 15:41:16 - <Warning> - [ERRCODE: SC_ERR_NO_RULES_LOADED(43)] - 1 rule files specified, but no rule was loaded at all!
7/4/2017 -- 15:41:16 - <Info> - 0 signatures processed. 0 are IP-only rules, 0 are inspecting packet payload, 0 inspect application layer, 0 are decoder event only
7/4/2017 -- 15:41:16 - <Info> - Threshold config parsed: 0 rule(s) found
7/4/2017 -- 15:41:16 - <Info> - cleaning up signature grouping structure... complete
7/4/2017 -- 15:41:16 - <Notice> - rule reload complete
7/4/2017 -- 15:41:16 - <Info> - prefix multi-detect.1001.reload.3
7/4/2017 -- 15:41:16 - <Info> - Including configuration file /etc/suricata/tenants/tenant-base.yaml.

-- TENANT IS NOW STUCK, TENANT RELOAD --
7/4/2017 -- 15:41:33 - <Info> - prefix multi-detect.1001.reload.3
7/4/2017 -- 15:41:33 - <Info> - Including configuration file /etc/suricata/tenants/tenant-base.yaml.
7/4/2017 -- 15:41:33 - <Info> - Configuration node 'classification-file' redefined.
7/4/2017 -- 15:41:33 - <Info> - Configuration node 'reference-config-file' redefined.
7/4/2017 -- 15:41:33 - <Info> - Configuration node 'vars' redefined.
7/4/2017 -- 15:41:33 - <Info> - Configuration node 'default-rule-path' redefined.
7/4/2017 -- 15:41:33 - <Info> - Configuration node 'rule-files' redefined.

-- ANOTHER TENANT RELOAD ATTEMPT --
7/4/2017 -- 15:41:44 - <Info> - prefix multi-detect.1001.reload.3
7/4/2017 -- 15:41:44 - <Info> - Including configuration file /etc/suricata/tenants/tenant-base.yaml.
7/4/2017 -- 15:41:44 - <Info> - Configuration node 'classification-file' redefined.
7/4/2017 -- 15:41:44 - <Info> - Configuration node 'reference-config-file' redefined.
7/4/2017 -- 15:41:44 - <Info> - Configuration node 'vars' redefined.
7/4/2017 -- 15:41:44 - <Info> - Configuration node 'default-rule-path' redefined.
7/4/2017 -- 15:41:44 - <Info> - Configuration node 'rule-files' redefined.

Actions #4

Updated by Andreas Herz over 5 years ago

  • Assignee set to OISF Dev
  • Target version set to TBD
Actions #5

Updated by Victor Julien over 4 years ago

Antti, are you able to try again with the current git master (ffc847db01fbf81df8a647d7a794d99894e4939d or later)? I did some changes that I think may be related.

Actions #6

Updated by Victor Julien over 4 years ago

Possibly related:

=================================================================
==110497==ERROR: AddressSanitizer: attempting double-free on 0x613000037e80 in thread T3 (DL#03):
    #0 0x7f9b7ffaf2ca in __interceptor_free (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x982ca)
    #1 0x1316ee4 in SCThresholdConfDeInitContext /home/victor/devel/eidps/src/util-threshold-config.c:263
    #2 0x1316b15 in SCThresholdConfInitContext /home/victor/devel/eidps/src/util-threshold-config.c:220
    #3 0xa8d077 in SigLoadSignatures /home/victor/devel/eidps/src/detect-engine-loader.c:363
    #4 0x90a25c in DetectEngineMultiTenantLoadTenant /home/victor/devel/eidps/src/detect-engine.c:2774
    #5 0x90b4ca in DetectLoaderFuncLoadTenant /home/victor/devel/eidps/src/detect-engine.c:2854
    #6 0xa8fa0a in DetectLoader /home/victor/devel/eidps/src/detect-engine-loader.c:586
    #7 0x113563d in TmThreadsManagement /home/victor/devel/eidps/src/tm-threads.c:719
    #8 0x7f9b7e36f6b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9)
    #9 0x7f9b7d03441c in clone (/lib/x86_64-linux-gnu/libc.so.6+0x10741c)

0x613000037e80 is located 0 bytes inside of 357-byte region [0x613000037e80,0x613000037fe5)
freed by thread T2 (DL#02) here:
    #0 0x7f9b7ffaf2ca in __interceptor_free (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x982ca)
    #1 0x1316ee4 in SCThresholdConfDeInitContext /home/victor/devel/eidps/src/util-threshold-config.c:263
    #2 0x1316b15 in SCThresholdConfInitContext /home/victor/devel/eidps/src/util-threshold-config.c:220
    #3 0xa8d077 in SigLoadSignatures /home/victor/devel/eidps/src/detect-engine-loader.c:363
    #4 0x90a25c in DetectEngineMultiTenantLoadTenant /home/victor/devel/eidps/src/detect-engine.c:2774
    #5 0x90b4ca in DetectLoaderFuncLoadTenant /home/victor/devel/eidps/src/detect-engine.c:2854
    #6 0xa8fa0a in DetectLoader /home/victor/devel/eidps/src/detect-engine-loader.c:586
    #7 0x113563d in TmThreadsManagement /home/victor/devel/eidps/src/tm-threads.c:719
    #8 0x7f9b7e36f6b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9)

previously allocated by thread T3 (DL#03) here:
    #0 0x7f9b7ffaf602 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x98602)
    #1 0x7f9b7dee4641 in pcre_compile2 (/lib/x86_64-linux-gnu/libpcre.so.3+0xb641)

Thread T3 (DL#03) created by T0 (Suricata-Main) here:
    #0 0x7f9b7ff4d253 in pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x36253)
    #1 0x113e2db in TmThreadSpawn /home/victor/devel/eidps/src/tm-threads.c:1895
    #2 0xa9011b in DetectLoaderThreadSpawn /home/victor/devel/eidps/src/detect-engine-loader.c:628
    #3 0x90c82f in DetectEngineMultiTenantSetup /home/victor/devel/eidps/src/detect-engine.c:2955
    #4 0x111f541 in PostConfLoadedDetectSetup /home/victor/devel/eidps/src/suricata.c:2493
    #5 0x1122b8d in main /home/victor/devel/eidps/src/suricata.c:2874
    #6 0x7f9b7cf4d82f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)

Thread T2 (DL#02) created by T0 (Suricata-Main) here:
    #0 0x7f9b7ff4d253 in pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x36253)
    #1 0x113e2db in TmThreadSpawn /home/victor/devel/eidps/src/tm-threads.c:1895
    #2 0xa9011b in DetectLoaderThreadSpawn /home/victor/devel/eidps/src/detect-engine-loader.c:628
    #3 0x90c82f in DetectEngineMultiTenantSetup /home/victor/devel/eidps/src/detect-engine.c:2955
    #4 0x111f541 in PostConfLoadedDetectSetup /home/victor/devel/eidps/src/suricata.c:2493
    #5 0x1122b8d in main /home/victor/devel/eidps/src/suricata.c:2874
    #6 0x7f9b7cf4d82f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)

Fix incoming.

Actions #7

Updated by Andreas Herz over 4 years ago

  • Related to Bug #2518: Tenant rules reload completely broken in 4.x.x added
Actions #8

Updated by Victor Julien over 4 years ago

  • Status changed from New to Assigned
  • Assignee changed from OISF Dev to Victor Julien
  • Target version changed from TBD to 4.1rc1
Actions #9

Updated by Victor Julien over 4 years ago

Can you test https://github.com/OISF/suricata/pull/3407 and report if this fixes the issue for you?

Actions #10

Updated by Victor Julien about 4 years ago

  • Status changed from Assigned to Closed
Actions

Also available in: Atom PDF