Actions
Bug #2090
closedRule-reload in multi-tenancy is buggy
Affected Versions:
Effort:
Difficulty:
Label:
Description
Observed version 3.2.1-1 (Debian Stretch)
I have observed several failure cases when performing rule reloads in multi-tenancy setup which may or may not be the result of same underlying bug:
Case 1: Suricata needs rule-reload before tenants can be reloaded safely
# systemctl start suricata # gdb attach --pid=suripid ( in another window ) # /usr/bin/suricatasc -c 'reload-tenant 1001 /etc/suricata/tenants/tenant-test.yaml' Thread 15 "US" received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7f5a20ff9700 (LWP 12865)] 0x0000000000000000 in ?? () (gdb) bt #0 0x0000000000000000 in ?? () #1 0x000055fa66f56ef9 in ThreadCtxDoInit (de_ctx=de_ctx@entry=0x55fa6955ec30, det_ctx=det_ctx@entry=0x7f5a04002320) at detect-engine.c:1228 #2 0x000055fa66f57dc6 in DetectEngineThreadCtxInitForReload (tv=tv@entry=0x55fa69571f80, new_de_ctx=new_de_ctx@entry=0x55fa6955ec30, mt=mt@entry=1) at detect-engine.c:1398 #3 0x000055fa66f5b296 in DetectEngineReloadThreads (new_de_ctx=0x55fa6955ec30) at detect-engine.c:460 #4 0x000055fa66f5e6e8 in DetectEngineMTApply () at detect-engine.c:2454 #5 0x000055fa67016c7e in UnixSocketReloadTenant (cmd=<optimized out>, answer=0x7f5a04000f10, data=<optimized out>) at runmode-unix-socket.c:691 #6 0x000055fa6704d576 in UnixCommandExecute ...
However, if reload happens beforehand everything is fine and tenant can be reloaded many times
systemctl start suricata
# /usr/bin/suricatasc -c reload-rules
{"message": "done", "return": "OK"}
# /usr/bin/suricatasc -c 'reload-tenant 1001 /etc/suricata/tenants/tenant-test.yaml'
{"message": "reloading tenant succeeded", "return": "OK"}
# /usr/bin/suricatasc -c 'reload-tenant 1001 /etc/suricata/tenants/tenant-test.yaml'
{"message": "reloading tenant succeeded", "return": "OK"}
# /usr/bin/suricatasc -c 'reload-tenant 1001 /etc/suricata/tenants/tenant-test.yaml'
{"message": "reloading tenant succeeded", "return": "OK"}
# /usr/bin/suricatasc -c 'reload-tenant 1001 /etc/suricata/tenants/tenant-test.yaml'
{"message": "reloading tenant succeeded", "return": "OK"}
However, in rapid succession of rule-reload and tenant reload, the tenant breaks:
# systemctl restart suricata
# /usr/bin/suricatasc -c reload-rules
{"message": "done", "return": "OK"}
# /usr/bin/suricatasc -c 'reload-tenant 1001 /etc/suricata/tenants/tenant-test.yaml'
{"message": "reloading tenant succeeded", "return": "OK"}
# /usr/bin/suricatasc -c 'reload-tenant 1001 /etc/suricata/tenants/tenant-test.yaml'
{"message": "reloading tenant succeeded", "return": "OK"}
# /usr/bin/suricatasc -c reload-rules; /usr/bin/suricatasc -c 'reload-tenant 1001 /etc/suricata/tenants/tenant-test.yaml'
{"message": "done", "return": "OK"}
{"message": "reload tenant failed", "return": "NOK"}
# /usr/bin/suricatasc -c reload-rules; /usr/bin/suricatasc -c 'reload-tenant 1001 /etc/suricata/tenants/tenant-test.yaml'
{"message": "done", "return": "OK"}
{"message": "reload tenant failed", "return": "NOK"}
# /usr/bin/suricatasc -c 'reload-tenant 1001 /etc/suricata/tenants/tenant-test.yaml'
{"message": "reload tenant failed", "return": "NOK"}
Updated by Antti Tönkyrä about 7 years ago
From debian sources, detect-engine.c:1228 looks to be
PatternMatchThreadPrepare(&det_ctx->mtc, de_ctx->mpm_matcher);
Updated by Peter Manev about 7 years ago
What does your suricata.log look like with the rapid succession reloads ?
I think I have tried that before and it was simply saying (suricata.log) that it is in the middle of reload so wont do any new one till the current one is finished - is that the case?
Updated by Antti Tönkyrä about 7 years ago
suricata.log when run with -vv
-- RULE RELOAD -- 7/4/2017 -- 15:40:51 - <Notice> - rule reload starting 7/4/2017 -- 15:40:51 - <Warning> - [ERRCODE: SC_ERR_NO_RULES_LOADED(43)] - 1 rule files specified, but no rule was loaded at all! 7/4/2017 -- 15:40:51 - <Info> - 0 signatures processed. 0 are IP-only rules, 0 are inspecting packet payload, 0 inspect application layer, 0 are decoder event only 7/4/2017 -- 15:40:51 - <Info> - Threshold config parsed: 0 rule(s) found 7/4/2017 -- 15:40:51 - <Info> - cleaning up signature grouping structure... complete 7/4/2017 -- 15:40:51 - <Notice> - rule reload complete -- TENANT RELOAD -- 7/4/2017 -- 15:41:00 - <Info> - prefix multi-detect.1001.reload.1 7/4/2017 -- 15:41:00 - <Info> - Including configuration file /etc/suricata/tenants/tenant-base.yaml. 7/4/2017 -- 15:41:00 - <Info> - Including configuration file /etc/suricata/tenants/tenant-base.yaml. 7/4/2017 -- 15:41:00 - <Info> - Configuration node 'classification-file' redefined. 7/4/2017 -- 15:41:00 - <Info> - Configuration node 'reference-config-file' redefined. 7/4/2017 -- 15:41:00 - <Info> - Configuration node 'vars' redefined. 7/4/2017 -- 15:41:00 - <Info> - Configuration node 'default-rule-path' redefined. 7/4/2017 -- 15:41:00 - <Info> - Configuration node 'rule-files' redefined. 7/4/2017 -- 15:41:00 - <Warning> - [ERRCODE: SC_ERR_NO_RULES_LOADED(43)] - 2 rule files specified, but no rule was loaded at all! 7/4/2017 -- 15:41:00 - <Info> - 0 signatures processed. 0 are IP-only rules, 0 are inspecting packet payload, 0 inspect application layer, 0 are decoder event only 7/4/2017 -- 15:41:00 - <Info> - cleaning up signature grouping structure... complete -- ANOTHER TENANT RELOAD -- 7/4/2017 -- 15:41:07 - <Info> - prefix multi-detect.1001.reload.2 7/4/2017 -- 15:41:07 - <Info> - Including configuration file /etc/suricata/tenants/tenant-base.yaml. 7/4/2017 -- 15:41:07 - <Info> - Including configuration file /etc/suricata/tenants/tenant-base.yaml. 7/4/2017 -- 15:41:07 - <Info> - Configuration node 'classification-file' redefined. 7/4/2017 -- 15:41:07 - <Info> - Configuration node 'reference-config-file' redefined. 7/4/2017 -- 15:41:07 - <Info> - Configuration node 'vars' redefined. 7/4/2017 -- 15:41:07 - <Info> - Configuration node 'default-rule-path' redefined. 7/4/2017 -- 15:41:07 - <Info> - Configuration node 'rule-files' redefined. 7/4/2017 -- 15:41:07 - <Warning> - [ERRCODE: SC_ERR_NO_RULES_LOADED(43)] - 2 rule files specified, but no rule was loaded at all! 7/4/2017 -- 15:41:07 - <Info> - 0 signatures processed. 0 are IP-only rules, 0 are inspecting packet payload, 0 inspect application layer, 0 are decoder event only 7/4/2017 -- 15:41:07 - <Info> - cleaning up signature grouping structure... complete -- RAPID RULE-TENANT-RELOAD -- (rule reload; tenant reload) 7/4/2017 -- 15:41:16 - <Notice> - rule reload starting 7/4/2017 -- 15:41:16 - <Warning> - [ERRCODE: SC_ERR_NO_RULES_LOADED(43)] - 1 rule files specified, but no rule was loaded at all! 7/4/2017 -- 15:41:16 - <Info> - 0 signatures processed. 0 are IP-only rules, 0 are inspecting packet payload, 0 inspect application layer, 0 are decoder event only 7/4/2017 -- 15:41:16 - <Info> - Threshold config parsed: 0 rule(s) found 7/4/2017 -- 15:41:16 - <Info> - cleaning up signature grouping structure... complete 7/4/2017 -- 15:41:16 - <Notice> - rule reload complete 7/4/2017 -- 15:41:16 - <Info> - prefix multi-detect.1001.reload.3 7/4/2017 -- 15:41:16 - <Info> - Including configuration file /etc/suricata/tenants/tenant-base.yaml. -- TENANT IS NOW STUCK, TENANT RELOAD -- 7/4/2017 -- 15:41:33 - <Info> - prefix multi-detect.1001.reload.3 7/4/2017 -- 15:41:33 - <Info> - Including configuration file /etc/suricata/tenants/tenant-base.yaml. 7/4/2017 -- 15:41:33 - <Info> - Configuration node 'classification-file' redefined. 7/4/2017 -- 15:41:33 - <Info> - Configuration node 'reference-config-file' redefined. 7/4/2017 -- 15:41:33 - <Info> - Configuration node 'vars' redefined. 7/4/2017 -- 15:41:33 - <Info> - Configuration node 'default-rule-path' redefined. 7/4/2017 -- 15:41:33 - <Info> - Configuration node 'rule-files' redefined. -- ANOTHER TENANT RELOAD ATTEMPT -- 7/4/2017 -- 15:41:44 - <Info> - prefix multi-detect.1001.reload.3 7/4/2017 -- 15:41:44 - <Info> - Including configuration file /etc/suricata/tenants/tenant-base.yaml. 7/4/2017 -- 15:41:44 - <Info> - Configuration node 'classification-file' redefined. 7/4/2017 -- 15:41:44 - <Info> - Configuration node 'reference-config-file' redefined. 7/4/2017 -- 15:41:44 - <Info> - Configuration node 'vars' redefined. 7/4/2017 -- 15:41:44 - <Info> - Configuration node 'default-rule-path' redefined. 7/4/2017 -- 15:41:44 - <Info> - Configuration node 'rule-files' redefined.
Updated by Andreas Herz almost 7 years ago
- Assignee set to OISF Dev
- Target version set to TBD
Updated by Victor Julien about 6 years ago
Antti, are you able to try again with the current git master (ffc847db01fbf81df8a647d7a794d99894e4939d or later)? I did some changes that I think may be related.
Updated by Victor Julien about 6 years ago
Possibly related:
=================================================================
==110497==ERROR: AddressSanitizer: attempting double-free on 0x613000037e80 in thread T3 (DL#03):
#0 0x7f9b7ffaf2ca in __interceptor_free (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x982ca)
#1 0x1316ee4 in SCThresholdConfDeInitContext /home/victor/devel/eidps/src/util-threshold-config.c:263
#2 0x1316b15 in SCThresholdConfInitContext /home/victor/devel/eidps/src/util-threshold-config.c:220
#3 0xa8d077 in SigLoadSignatures /home/victor/devel/eidps/src/detect-engine-loader.c:363
#4 0x90a25c in DetectEngineMultiTenantLoadTenant /home/victor/devel/eidps/src/detect-engine.c:2774
#5 0x90b4ca in DetectLoaderFuncLoadTenant /home/victor/devel/eidps/src/detect-engine.c:2854
#6 0xa8fa0a in DetectLoader /home/victor/devel/eidps/src/detect-engine-loader.c:586
#7 0x113563d in TmThreadsManagement /home/victor/devel/eidps/src/tm-threads.c:719
#8 0x7f9b7e36f6b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9)
#9 0x7f9b7d03441c in clone (/lib/x86_64-linux-gnu/libc.so.6+0x10741c)
0x613000037e80 is located 0 bytes inside of 357-byte region [0x613000037e80,0x613000037fe5)
freed by thread T2 (DL#02) here:
#0 0x7f9b7ffaf2ca in __interceptor_free (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x982ca)
#1 0x1316ee4 in SCThresholdConfDeInitContext /home/victor/devel/eidps/src/util-threshold-config.c:263
#2 0x1316b15 in SCThresholdConfInitContext /home/victor/devel/eidps/src/util-threshold-config.c:220
#3 0xa8d077 in SigLoadSignatures /home/victor/devel/eidps/src/detect-engine-loader.c:363
#4 0x90a25c in DetectEngineMultiTenantLoadTenant /home/victor/devel/eidps/src/detect-engine.c:2774
#5 0x90b4ca in DetectLoaderFuncLoadTenant /home/victor/devel/eidps/src/detect-engine.c:2854
#6 0xa8fa0a in DetectLoader /home/victor/devel/eidps/src/detect-engine-loader.c:586
#7 0x113563d in TmThreadsManagement /home/victor/devel/eidps/src/tm-threads.c:719
#8 0x7f9b7e36f6b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9)
previously allocated by thread T3 (DL#03) here:
#0 0x7f9b7ffaf602 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x98602)
#1 0x7f9b7dee4641 in pcre_compile2 (/lib/x86_64-linux-gnu/libpcre.so.3+0xb641)
Thread T3 (DL#03) created by T0 (Suricata-Main) here:
#0 0x7f9b7ff4d253 in pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x36253)
#1 0x113e2db in TmThreadSpawn /home/victor/devel/eidps/src/tm-threads.c:1895
#2 0xa9011b in DetectLoaderThreadSpawn /home/victor/devel/eidps/src/detect-engine-loader.c:628
#3 0x90c82f in DetectEngineMultiTenantSetup /home/victor/devel/eidps/src/detect-engine.c:2955
#4 0x111f541 in PostConfLoadedDetectSetup /home/victor/devel/eidps/src/suricata.c:2493
#5 0x1122b8d in main /home/victor/devel/eidps/src/suricata.c:2874
#6 0x7f9b7cf4d82f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
Thread T2 (DL#02) created by T0 (Suricata-Main) here:
#0 0x7f9b7ff4d253 in pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x36253)
#1 0x113e2db in TmThreadSpawn /home/victor/devel/eidps/src/tm-threads.c:1895
#2 0xa9011b in DetectLoaderThreadSpawn /home/victor/devel/eidps/src/detect-engine-loader.c:628
#3 0x90c82f in DetectEngineMultiTenantSetup /home/victor/devel/eidps/src/detect-engine.c:2955
#4 0x111f541 in PostConfLoadedDetectSetup /home/victor/devel/eidps/src/suricata.c:2493
#5 0x1122b8d in main /home/victor/devel/eidps/src/suricata.c:2874
#6 0x7f9b7cf4d82f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
Fix incoming.
Updated by Andreas Herz almost 6 years ago
- Related to Bug #2518: Tenant rules reload completely broken in 4.x.x added
Updated by Victor Julien almost 6 years ago
- Status changed from New to Assigned
- Assignee changed from OISF Dev to Victor Julien
- Target version changed from TBD to 4.1rc1
Updated by Victor Julien almost 6 years ago
Can you test https://github.com/OISF/suricata/pull/3407 and report if this fixes the issue for you?
Updated by Victor Julien almost 6 years ago
- Status changed from Assigned to Closed
Actions