Bug #7819
openregister-tenant deadlocks
Description
Enabling multi tenancy and adding tenant via suricatasc deadlocks in DetectEngineMultiTenantEnabled. Here's backtrace:
(gdb) bt #0 0x00007ffff70aca40 in __lll_lock_wait () from /lib64/libc.so.6 #1 0x00007ffff70b3751 in pthread_mutex_lock@@GLIBC_2.2.5 () from /lib64/libc.so.6 #2 0x00005555555d7cbf in DetectEngineMultiTenantEnabled () at /usr/src/debug/ideco-suricata-8.0.0-1.1752669058.298.7e9ad8c1.utm.x86_64/src/detect-engine.c:3876 #3 DetectEngineThreadCtxInitForReload (tv=0x5555573739f0, new_de_ctx=<optimized out>, mt=1) at /usr/src/debug/ideco-suricata-8.0.0-1.1752669058.298.7e9ad8c1.utm.x86_64/src/detect-engine.c:3496 #4 0x0000555555740e47 in DetectEngineReloadThreads.isra.0 (new_de_ctx=<optimized out>) at /usr/src/debug/ideco-suricata-8.0.0-1.1752669058.298.7e9ad8c1.utm.x86_64/src/detect-engine.c:2361 #5 0x00005555555d757a in DetectEngineMTApply () at /usr/src/debug/ideco-suricata-8.0.0-1.1752669058.298.7e9ad8c1.utm.x86_64/src/detect-engine.c:4954 #6 0x000055555563d3af in UnixSocketRegisterTenant (cmd=<optimized out>, answer=0x7fffd8000c10, data=<optimized out>) at /usr/src/debug/ideco-suricata-8.0.0-1.1752669058.298.7e9ad8c1.utm.x86_64/src/runmode-unix-socket.c:1123 #7 0x0000555555734fd4 in UnixCommandExecute (this=<optimized out>, command=0x7fffe77fbc80 "{\"command\":\"register-tenant\",\"arguments\":{\"filename\":\"/tmp/tenant1.yaml\",\"id\":1}}", client=0x7fffd8000da0) at /usr/src/debug/ideco-suricata-8.0.0-1.1752669058.298.7e9ad8c1.utm.x86_64/src/unix-manager.c:499 #8 UnixCommandRun (client=client@entry=0x7fffd8000da0, this=<optimized out>) at /usr/src/debug/ideco-suricata-8.0.0-1.1752669058.298.7e9ad8c1.utm.x86_64/src/unix-manager.c:615 #9 0x0000555555737276 in UnixMain.constprop.0 (this=<optimized out>) at /usr/src/debug/ideco-suricata-8.0.0-1.1752669058.298.7e9ad8c1.utm.x86_64/src/unix-manager.c:666 #10 0x0000555555567990 in UnixManager (th_v=0x5555573736e0, thread_data=<optimized out>) at /usr/src/debug/ideco-suricata-8.0.0-1.1752669058.298.7e9ad8c1.utm.x86_64/src/unix-manager.c:1163 #11 0x000055555556f26e in TmThreadsManagement (td=0x5555573736e0) at /usr/src/debug/ideco-suricata-8.0.0-1.1752669058.298.7e9ad8c1.utm.x86_64/src/tm-threads.c:571 #12 0x00007ffff70b0148 in start_thread () from /lib64/libc.so.6 #13 0x00007ffff71340cc in __clone3 () from /lib64/libc.so.6
DetectEngineMTApply takes lock, but DetectEngineMultiTenantEnabled attemts to lock again.
Files
Updated by Alexander Maltsev 22 days ago
Alexander Maltsev wrote:
Enabling multi tenancy and adding tenant via suricatasc deadlocks in DetectEngineMultiTenantEnabled. Here's backtrace:
[...]DetectEngineMTApply takes lock, but DetectEngineMultiTenantEnabled attemts to lock again.
Alexander Maltsev wrote:
Enabling multi tenancy and adding tenant via suricatasc deadlocks in DetectEngineMultiTenantEnabled. Here's backtrace:
[...]DetectEngineMTApply takes lock, but DetectEngineMultiTenantEnabled attemts to lock again.
Setting mutex initializer to PTHREAD_RECURSIVE_MUTEX_INITIALIZER_NP in
https://github.com/OISF/suricata/blob/2e69e0d5c16c35ea05497c378fa03908824fe5e4/src/detect-engine.c#L99 seems to work fine. Not sure if it is the intended way, or even if recursive mutexes are supported on all platforms suricata runs on.
Updated by Jeff Lucovsky 10 days ago
Can you post your suricata.yaml and one of the tenant yaml files?
Updated by Alexander Maltsev 9 days ago
- File tenant-01K1ATJ4AZHXBZMT7P47A7ZHJZ.yaml tenant-01K1ATJ4AZHXBZMT7P47A7ZHJZ.yaml added
- File suricata.yaml suricata.yaml added
Jeff Lucovsky wrote in #note-2:
Can you post your suricata.yaml and one of the tenant yaml files?
Updated by Jeff Lucovsky 9 days ago
- Assignee changed from OISF Dev to Jeff Lucovsky
Updated by Jeff Lucovsky 9 days ago
- Status changed from New to In Review