Bug #2090: Rule-reload in multi-tenancy is buggy - Suricata - Open Information Security Foundation

Actions

Copy link

Bug #2090

closed

Rule-reload in multi-tenancy is buggy

Added by Antti Tönkyrä over 7 years ago. Updated over 6 years ago.

Status:

Closed

Priority:

Normal

Assignee:

Victor Julien

Target version:

4.1rc1

Affected Versions:

Effort:

Difficulty:

Label:

Description

Observed version 3.2.1-1 (Debian Stretch)

I have observed several failure cases when performing rule reloads in multi-tenancy setup which may or may not be the result of same underlying bug:

Case 1: Suricata needs rule-reload before tenants can be reloaded safely

# systemctl start suricata
# gdb attach --pid=suripid
( in another window ) # /usr/bin/suricatasc -c 'reload-tenant 1001 /etc/suricata/tenants/tenant-test.yaml'
Thread 15 "US" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7f5a20ff9700 (LWP 12865)]
0x0000000000000000 in ?? ()
(gdb) bt
#0  0x0000000000000000 in ?? ()
#1  0x000055fa66f56ef9 in ThreadCtxDoInit (de_ctx=de_ctx@entry=0x55fa6955ec30, det_ctx=det_ctx@entry=0x7f5a04002320) at detect-engine.c:1228
#2  0x000055fa66f57dc6 in DetectEngineThreadCtxInitForReload (tv=tv@entry=0x55fa69571f80, new_de_ctx=new_de_ctx@entry=0x55fa6955ec30, mt=mt@entry=1) at detect-engine.c:1398
#3  0x000055fa66f5b296 in DetectEngineReloadThreads (new_de_ctx=0x55fa6955ec30) at detect-engine.c:460
#4  0x000055fa66f5e6e8 in DetectEngineMTApply () at detect-engine.c:2454
#5  0x000055fa67016c7e in UnixSocketReloadTenant (cmd=<optimized out>, answer=0x7f5a04000f10, data=<optimized out>) at runmode-unix-socket.c:691
#6  0x000055fa6704d576 in UnixCommandExecute ...

However, if reload happens beforehand everything is fine and tenant can be reloaded many times

systemctl start suricata
# /usr/bin/suricatasc -c reload-rules
{"message": "done", "return": "OK"}
# /usr/bin/suricatasc -c 'reload-tenant 1001 /etc/suricata/tenants/tenant-test.yaml'
{"message": "reloading tenant succeeded", "return": "OK"}
# /usr/bin/suricatasc -c 'reload-tenant 1001 /etc/suricata/tenants/tenant-test.yaml'
{"message": "reloading tenant succeeded", "return": "OK"}
# /usr/bin/suricatasc -c 'reload-tenant 1001 /etc/suricata/tenants/tenant-test.yaml'
{"message": "reloading tenant succeeded", "return": "OK"}
# /usr/bin/suricatasc -c 'reload-tenant 1001 /etc/suricata/tenants/tenant-test.yaml'
{"message": "reloading tenant succeeded", "return": "OK"}

However, in rapid succession of rule-reload and tenant reload, the tenant breaks:

# systemctl restart suricata
# /usr/bin/suricatasc -c reload-rules
{"message": "done", "return": "OK"}
#  /usr/bin/suricatasc -c 'reload-tenant 1001 /etc/suricata/tenants/tenant-test.yaml'
{"message": "reloading tenant succeeded", "return": "OK"}
#  /usr/bin/suricatasc -c 'reload-tenant 1001 /etc/suricata/tenants/tenant-test.yaml'
{"message": "reloading tenant succeeded", "return": "OK"}
# /usr/bin/suricatasc -c reload-rules; /usr/bin/suricatasc -c 'reload-tenant 1001 /etc/suricata/tenants/tenant-test.yaml'
{"message": "done", "return": "OK"}
{"message": "reload tenant failed", "return": "NOK"}
# /usr/bin/suricatasc -c reload-rules; /usr/bin/suricatasc -c 'reload-tenant 1001 /etc/suricata/tenants/tenant-test.yaml'
{"message": "done", "return": "OK"}
{"message": "reload tenant failed", "return": "NOK"}
#  /usr/bin/suricatasc -c 'reload-tenant 1001 /etc/suricata/tenants/tenant-test.yaml'
{"message": "reload tenant failed", "return": "NOK"}

Related issues 1 (0 open — 1 closed)

Actions

Copy link

Updated by Antti Tönkyrä over 7 years ago

From debian sources, detect-engine.c:1228 looks to be

    PatternMatchThreadPrepare(&det_ctx->mtc, de_ctx->mpm_matcher);

Actions

Copy link

Updated by Peter Manev over 7 years ago

What does your suricata.log look like with the rapid succession reloads ?
I think I have tried that before and it was simply saying (suricata.log) that it is in the middle of reload so wont do any new one till the current one is finished - is that the case?

Actions

Copy link

Updated by Antti Tönkyrä over 7 years ago

suricata.log when run with -vv

-- RULE RELOAD --
7/4/2017 -- 15:40:51 - <Notice> - rule reload starting
7/4/2017 -- 15:40:51 - <Warning> - [ERRCODE: SC_ERR_NO_RULES_LOADED(43)] - 1 rule files specified, but no rule was loaded at all!
7/4/2017 -- 15:40:51 - <Info> - 0 signatures processed. 0 are IP-only rules, 0 are inspecting packet payload, 0 inspect application layer, 0 are decoder event
only
7/4/2017 -- 15:40:51 - <Info> - Threshold config parsed: 0 rule(s) found
7/4/2017 -- 15:40:51 - <Info> - cleaning up signature grouping structure... complete
7/4/2017 -- 15:40:51 - <Notice> - rule reload complete

-- TENANT RELOAD --
7/4/2017 -- 15:41:00 - <Info> - prefix multi-detect.1001.reload.1
7/4/2017 -- 15:41:00 - <Info> - Including configuration file /etc/suricata/tenants/tenant-base.yaml.
7/4/2017 -- 15:41:00 - <Info> - Including configuration file /etc/suricata/tenants/tenant-base.yaml.
7/4/2017 -- 15:41:00 - <Info> - Configuration node 'classification-file' redefined.
7/4/2017 -- 15:41:00 - <Info> - Configuration node 'reference-config-file' redefined.
7/4/2017 -- 15:41:00 - <Info> - Configuration node 'vars' redefined.
7/4/2017 -- 15:41:00 - <Info> - Configuration node 'default-rule-path' redefined.
7/4/2017 -- 15:41:00 - <Info> - Configuration node 'rule-files' redefined.
7/4/2017 -- 15:41:00 - <Warning> - [ERRCODE: SC_ERR_NO_RULES_LOADED(43)] - 2 rule files specified, but no rule was loaded at all!
7/4/2017 -- 15:41:00 - <Info> - 0 signatures processed. 0 are IP-only rules, 0 are inspecting packet payload, 0 inspect application layer, 0 are decoder event
only
7/4/2017 -- 15:41:00 - <Info> - cleaning up signature grouping structure... complete

-- ANOTHER TENANT RELOAD --
7/4/2017 -- 15:41:07 - <Info> - prefix multi-detect.1001.reload.2
7/4/2017 -- 15:41:07 - <Info> - Including configuration file /etc/suricata/tenants/tenant-base.yaml.
7/4/2017 -- 15:41:07 - <Info> - Including configuration file /etc/suricata/tenants/tenant-base.yaml.
7/4/2017 -- 15:41:07 - <Info> - Configuration node 'classification-file' redefined.
7/4/2017 -- 15:41:07 - <Info> - Configuration node 'reference-config-file' redefined.
7/4/2017 -- 15:41:07 - <Info> - Configuration node 'vars' redefined.
7/4/2017 -- 15:41:07 - <Info> - Configuration node 'default-rule-path' redefined.
7/4/2017 -- 15:41:07 - <Info> - Configuration node 'rule-files' redefined.
7/4/2017 -- 15:41:07 - <Warning> - [ERRCODE: SC_ERR_NO_RULES_LOADED(43)] - 2 rule files specified, but no rule was loaded at all!
7/4/2017 -- 15:41:07 - <Info> - 0 signatures processed. 0 are IP-only rules, 0 are inspecting packet payload, 0 inspect application layer, 0 are decoder event only
7/4/2017 -- 15:41:07 - <Info> - cleaning up signature grouping structure... complete

-- RAPID RULE-TENANT-RELOAD -- (rule reload; tenant reload)
7/4/2017 -- 15:41:16 - <Notice> - rule reload starting
7/4/2017 -- 15:41:16 - <Warning> - [ERRCODE: SC_ERR_NO_RULES_LOADED(43)] - 1 rule files specified, but no rule was loaded at all!
7/4/2017 -- 15:41:16 - <Info> - 0 signatures processed. 0 are IP-only rules, 0 are inspecting packet payload, 0 inspect application layer, 0 are decoder event only
7/4/2017 -- 15:41:16 - <Info> - Threshold config parsed: 0 rule(s) found
7/4/2017 -- 15:41:16 - <Info> - cleaning up signature grouping structure... complete
7/4/2017 -- 15:41:16 - <Notice> - rule reload complete
7/4/2017 -- 15:41:16 - <Info> - prefix multi-detect.1001.reload.3
7/4/2017 -- 15:41:16 - <Info> - Including configuration file /etc/suricata/tenants/tenant-base.yaml.

-- TENANT IS NOW STUCK, TENANT RELOAD --
7/4/2017 -- 15:41:33 - <Info> - prefix multi-detect.1001.reload.3
7/4/2017 -- 15:41:33 - <Info> - Including configuration file /etc/suricata/tenants/tenant-base.yaml.
7/4/2017 -- 15:41:33 - <Info> - Configuration node 'classification-file' redefined.
7/4/2017 -- 15:41:33 - <Info> - Configuration node 'reference-config-file' redefined.
7/4/2017 -- 15:41:33 - <Info> - Configuration node 'vars' redefined.
7/4/2017 -- 15:41:33 - <Info> - Configuration node 'default-rule-path' redefined.
7/4/2017 -- 15:41:33 - <Info> - Configuration node 'rule-files' redefined.

-- ANOTHER TENANT RELOAD ATTEMPT --
7/4/2017 -- 15:41:44 - <Info> - prefix multi-detect.1001.reload.3
7/4/2017 -- 15:41:44 - <Info> - Including configuration file /etc/suricata/tenants/tenant-base.yaml.
7/4/2017 -- 15:41:44 - <Info> - Configuration node 'classification-file' redefined.
7/4/2017 -- 15:41:44 - <Info> - Configuration node 'reference-config-file' redefined.
7/4/2017 -- 15:41:44 - <Info> - Configuration node 'vars' redefined.
7/4/2017 -- 15:41:44 - <Info> - Configuration node 'default-rule-path' redefined.
7/4/2017 -- 15:41:44 - <Info> - Configuration node 'rule-files' redefined.

Actions

Copy link

Updated by Andreas Herz over 7 years ago

Assignee set to OISF Dev
Target version set to TBD

Actions

Copy link

Updated by Victor Julien almost 7 years ago

Antti, are you able to try again with the current git master (ffc847db01fbf81df8a647d7a794d99894e4939d or later)? I did some changes that I think may be related.

Actions

Copy link

Updated by Victor Julien over 6 years ago

Possibly related:

=================================================================
==110497==ERROR: AddressSanitizer: attempting double-free on 0x613000037e80 in thread T3 (DL#03):
    #0 0x7f9b7ffaf2ca in __interceptor_free (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x982ca)
    #1 0x1316ee4 in SCThresholdConfDeInitContext /home/victor/devel/eidps/src/util-threshold-config.c:263
    #2 0x1316b15 in SCThresholdConfInitContext /home/victor/devel/eidps/src/util-threshold-config.c:220
    #3 0xa8d077 in SigLoadSignatures /home/victor/devel/eidps/src/detect-engine-loader.c:363
    #4 0x90a25c in DetectEngineMultiTenantLoadTenant /home/victor/devel/eidps/src/detect-engine.c:2774
    #5 0x90b4ca in DetectLoaderFuncLoadTenant /home/victor/devel/eidps/src/detect-engine.c:2854
    #6 0xa8fa0a in DetectLoader /home/victor/devel/eidps/src/detect-engine-loader.c:586
    #7 0x113563d in TmThreadsManagement /home/victor/devel/eidps/src/tm-threads.c:719
    #8 0x7f9b7e36f6b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9)
    #9 0x7f9b7d03441c in clone (/lib/x86_64-linux-gnu/libc.so.6+0x10741c)

0x613000037e80 is located 0 bytes inside of 357-byte region [0x613000037e80,0x613000037fe5)
freed by thread T2 (DL#02) here:
    #0 0x7f9b7ffaf2ca in __interceptor_free (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x982ca)
    #1 0x1316ee4 in SCThresholdConfDeInitContext /home/victor/devel/eidps/src/util-threshold-config.c:263
    #2 0x1316b15 in SCThresholdConfInitContext /home/victor/devel/eidps/src/util-threshold-config.c:220
    #3 0xa8d077 in SigLoadSignatures /home/victor/devel/eidps/src/detect-engine-loader.c:363
    #4 0x90a25c in DetectEngineMultiTenantLoadTenant /home/victor/devel/eidps/src/detect-engine.c:2774
    #5 0x90b4ca in DetectLoaderFuncLoadTenant /home/victor/devel/eidps/src/detect-engine.c:2854
    #6 0xa8fa0a in DetectLoader /home/victor/devel/eidps/src/detect-engine-loader.c:586
    #7 0x113563d in TmThreadsManagement /home/victor/devel/eidps/src/tm-threads.c:719
    #8 0x7f9b7e36f6b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9)

previously allocated by thread T3 (DL#03) here:
    #0 0x7f9b7ffaf602 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x98602)
    #1 0x7f9b7dee4641 in pcre_compile2 (/lib/x86_64-linux-gnu/libpcre.so.3+0xb641)

Thread T3 (DL#03) created by T0 (Suricata-Main) here:
    #0 0x7f9b7ff4d253 in pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x36253)
    #1 0x113e2db in TmThreadSpawn /home/victor/devel/eidps/src/tm-threads.c:1895
    #2 0xa9011b in DetectLoaderThreadSpawn /home/victor/devel/eidps/src/detect-engine-loader.c:628
    #3 0x90c82f in DetectEngineMultiTenantSetup /home/victor/devel/eidps/src/detect-engine.c:2955
    #4 0x111f541 in PostConfLoadedDetectSetup /home/victor/devel/eidps/src/suricata.c:2493
    #5 0x1122b8d in main /home/victor/devel/eidps/src/suricata.c:2874
    #6 0x7f9b7cf4d82f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)

Thread T2 (DL#02) created by T0 (Suricata-Main) here:
    #0 0x7f9b7ff4d253 in pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x36253)
    #1 0x113e2db in TmThreadSpawn /home/victor/devel/eidps/src/tm-threads.c:1895
    #2 0xa9011b in DetectLoaderThreadSpawn /home/victor/devel/eidps/src/detect-engine-loader.c:628
    #3 0x90c82f in DetectEngineMultiTenantSetup /home/victor/devel/eidps/src/detect-engine.c:2955
    #4 0x111f541 in PostConfLoadedDetectSetup /home/victor/devel/eidps/src/suricata.c:2493
    #5 0x1122b8d in main /home/victor/devel/eidps/src/suricata.c:2874
    #6 0x7f9b7cf4d82f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)

Fix incoming.

Actions

Copy link