Project

General

Profile

Actions

Bug #6835

open

BUG_ON triggered from TmThreadsInjectFlowById

Added by Jeff Lucovsky about 2 months ago. Updated 13 days ago.

Status:
Resolved
Priority:
Normal
Assignee:
Target version:
Affected Versions:
Effort:
Difficulty:
Label:

Description

Seen on a small system with 500Mbps traffic

(gdb) thread 1
[Switching to thread 1 (Thread 0x7fbf52aff640 (LWP 13680))]
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50    ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007fbfe94c671e in __GI_abort () at abort.c:79
#2  0x00007fbfe94bddfa in __assert_fail_base (fmt=0x7fbfe9618508 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
    assertion=assertion@entry=0x56068c8d18d8 "!(id <= 0 || id > (int)thread_store.threads_size)", file=file@entry=0x56068c8d11df "tm-threads.c",
    line=line@entry=2298, function=function@entry=0x56068c8d1bd0 <__PRETTY_FUNCTION__.0> "TmThreadsInjectFlowById") at assert.c:92
#3  0x00007fbfe94bde72 in __GI___assert_fail (assertion=0x56068c8d18d8 "!(id <= 0 || id > (int)thread_store.threads_size)",
    file=0x56068c8d11df "tm-threads.c", line=2298, function=0x56068c8d1bd0 <__PRETTY_FUNCTION__.0> "TmThreadsInjectFlowById") at assert.c:101
#4  0x000056068c230cd2 in TmThreadsInjectFlowById (f=0x7fbf50387380, id=0) at tm-threads.c:2298
#5  0x000056068c3265e6 in FlowForceReassemblyForFlow (f=0x7fbf50387380) at flow-timeout.c:351
#6  0x000056068c321348 in ProcessAsideQueue (td=0x7fbf51e00030, counters=0x7fbf52afcaa0) at flow-manager.c:280
#7  0x000056068c32197e in FlowTimeoutHash (td=0x7fbf51e00030, ts=..., hash_min=457352, hash_max=522688, counters=0x7fbf52afcaa0) at flow-manager.c:459
#8  0x000056068c321af5 in FlowTimeoutHashInChunks (td=0x7fbf51e00030, ts=..., hash_min=0, hash_max=653360, counters=0x7fbf52afcaa0, rows=65336,
    pos=0x7fbf52afca10) at flow-manager.c:499
#9  0x000056068c32277e in FlowManager (th_v=0x7fbfc3ac8380, thread_data=0x7fbf51e00000) at flow-manager.c:829
#10 0x000056068c22d06a in TmThreadsManagement (td=0x7fbfc3ac8380) at tm-threads.c:567
#11 0x00007fbfe8e03f3e in start_thread (arg=0x7fbf52aff640) at pthread_create.c:463
#12 0x00007fbfe958814f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(gdb)
(gdb) p f->thread_id
$6 = {0, 1}
(gdb) p/x f->flags
$7 = 0x100211
(gdb) p *f
$9 = {src = {address = {address_un_data32 = {3883608321, 0, 0, 0}, address_un_data16 = {10497, 59259, 0, 0, 0, 0, 0, 0},
      address_un_data8 = "\001){\347", '\000' <repeats 11 times>}}, dst = {address = {address_un_data32 = {3255904769, 0, 0, 0}, address_un_data16 = {10753,
        49681, 0, 0, 0, 0, 0, 0}, address_un_data8 = "\001*\021\302", '\000' <repeats 11 times>}}, {sp = 5883, icmp_s = {type = 251 '\373',
      code = 22 '\026'}, esp = {spi = 5883}}, {dp = 443, icmp_d = {type = 187 '\273', code = 1 '\001'}}, proto = 6 '\006', recursion_level = 0 '\000',
  vlan_id = {0, 0, 0}, vlan_idx = 0 '\000', {{ffr_ts = 1 '\001', ffr_tc = 0 '\000'}, ffr = 1 '\001'}, timeout_at = 1709754443, thread_id = {0, 1},
  next = 0x0, livedev = 0x7fbfe6c57120, flow_hash = 1553553381, timeout_policy = 5, lastts = {secs = 1709754438, usecs = 355472}, flow_state = 0,
  tenant_id = 0, probing_parser_toserver_alproto_masks = 0, probing_parser_toclient_alproto_masks = 0, flags = 1049105, file_flags = 2730,
  protodetect_dp = 0, parent_id = 0, m = {__data = {__lock = 1, __count = 0, __owner = 13680, __nusers = 1, __kind = 0, __spins = 0, __elision = 0, __list = {
        __prev = 0x0, __next = 0x0}}, __size = "\001\000\000\000\000\000\000\000p5\000\000\001", '\000' <repeats 26 times>, __align = 1},
  protoctx = 0x7fbfab805d80, protomap = 0 '\000', flow_end_flags = 16 '\020', alproto = 0, alproto_ts = 0, alproto_tc = 0, alproto_orig = 0,
  alproto_expect = 0, de_ctx_version = 99, min_ttl_toserver = 32 ' ', max_ttl_toserver = 32 ' ', min_ttl_toclient = 32 ' ', max_ttl_toclient = 32 ' ',
  alparser = 0x0, alstate = 0x0, sgh_toclient = 0x7fbfc2c66a00, sgh_toserver = 0x0, flowvar = 0x0, fb = 0x0, startts = {secs = 1709754438, usecs = 355472},
  todstpktcnt = 0, tosrcpktcnt = 1, todstbytecnt = 0, tosrcbytecnt = 102}
(gdb)  p *(TcpSession *)f->protoctx
$10 = {pool_id = 0, state = 2 '\002', pstate = 0 '\000', queue_len = 0 '\000', data_first_seen_dir = 0 '\000', tcp_packet_flags = 18 '\022', flags = 0,
  reassembly_depth = 1048576, server = {flags = 0, wscale = 0, os_policy = 0 '\000', tcp_flags = 18 '\022', isn = 0, next_seq = 0, last_ack = 0,
    next_win = 0, window = 65535, last_ts = 0, last_pkt_ts = 0, base_seq = 0, app_progress_rel = 0, raw_progress_rel = 0, log_progress_rel = 0,
    min_inspect_depth = 0, data_required = 0, sb = {region = {buf = 0x0, buf_size = 0, buf_offset = 0, stream_offset = 0, next = 0x0}, sbb_tree = {
        rbh_root = 0x0}, head = 0x0, sbb_size = 0, regions = 1, max_regions = 1}, seg_tree = {rbh_root = 0x0}, segs_right_edge = 0, sack_size = 0,
    sack_tree = {rbh_root = 0x0}}, client = {flags = 0, wscale = 0, os_policy = 0 '\000', tcp_flags = 0 '\000', isn = 1321748982, next_seq = 1321748983,
    last_ack = 0, next_win = 0, window = 0, last_ts = 0, last_pkt_ts = 0, base_seq = 1321748983, app_progress_rel = 0, raw_progress_rel = 0,
    log_progress_rel = 0, min_inspect_depth = 0, data_required = 0, sb = {region = {buf = 0x0, buf_size = 0, buf_offset = 0, stream_offset = 0, next = 0x0},
      sbb_tree = {rbh_root = 0x0}, head = 0x0, sbb_size = 0, regions = 1, max_regions = 1}, seg_tree = {rbh_root = 0x0}, segs_right_edge = 0, sack_size = 0,
    sack_tree = {rbh_root = 0x0}}, queue = 0x0}
(gdb) p/t f->flags
$11 = 100000000001000010001

Flow flags:
- FLOW_TO_SRC_SEEN (0)
- FLOW_TOCLIENT_IPONLY_SET (4)
- FLOW_SGH_TOCLIENT (9)
- FLOW_IPV4 (20)

Suricata stats:

{"timestamp": "2024-03-06T12:19:57.843902-0800", "event_type": "stats", "stats": {"uptime": 79, "capture": {"kernel_packets": 1861865, "kernel_drops": 0}, "decoder": {"pkts": 1861865, "bytes": 1375631276, "invalid": 430018, "ipv4": 1861865, "ipv6": 0, "ethernet": 1861865, "arp": 0, "unknown_ethertype": 0, "chdlc": 0, "raw": 0, "null": 0, "sll": 0, "tcp": 1426119, "udp": 5790, "sctp": 0, "esp": 0, "icmpv4": 0, "icmpv6": 0, "ppp": 0, "pppoe": 0, "geneve": 0, "gre": 0, "vlan": 0, "vlan_qinq": 0, "vlan_qinqinq": 0, "vxlan": 0, "vntag": 0, "ieee8021ah": 0, "teredo": 0, "ipv4_in_ipv6": 0, "ipv6_in_ipv6": 0, "mpls": 0, "avg_pkt_size": 738, "max_pkt_size": 1514, "max_mac_addrs_src": 0, "max_mac_addrs_dst": 0, "erspan": 0, "nsh": 0, "event": {"ipv4": {"pkt_too_small": 0, "hlen_too_small": 0, "iplen_smaller_than_hlen": 0, "trunc_pkt": 429956, "opt_invalid": 0, "opt_invalid_len": 0, "opt_malformed": 0, "opt_pad_required": 0, "opt_eol_required": 0, "opt_duplicate": 0, "opt_unknown": 0, "wrong_ip_version": 0, "icmpv6": 0, "frag_pkt_too_large": 0, "frag_overlap": 0, "frag_ignored": 0}, "icmpv4": {"pkt_too_small": 0, "unknown_type": 0, "unknown_code": 0, "ipv4_trunc_pkt": 0, "ipv4_unknown_ver": 0}, "icmpv6": {"unknown_type": 0, "unknown_code": 0, "pkt_too_small": 0, "ipv6_unknown_version": 0, "ipv6_trunc_pkt": 0, "mld_message_with_invalid_hl": 0, "unassigned_type": 0, "experimentation_type": 0}, "ipv6": {"pkt_too_small": 0, "trunc_pkt": 0, "trunc_exthdr": 0, "exthdr_dupl_fh": 0, "exthdr_useless_fh": 0, "exthdr_dupl_rh": 0, "exthdr_dupl_hh": 0, "exthdr_dupl_dh": 0, "exthdr_dupl_ah": 0, "exthdr_dupl_eh": 0, "exthdr_invalid_optlen": 0, "wrong_ip_version": 0, "exthdr_ah_res_not_null": 0, "hopopts_unknown_opt": 0, "hopopts_only_padding": 0, "dstopts_unknown_opt": 0, "dstopts_only_padding": 0, "rh_type_0": 0, "zero_len_padn": 0, "fh_non_zero_reserved_field": 0, "data_after_none_header": 0, "unknown_next_header": 0, "icmpv4": 0, "frag_pkt_too_large": 0, "frag_overlap": 0, "frag_invalid_length": 0, "frag_ignored": 0, "ipv4_in_ipv6_too_small": 0, "ipv4_in_ipv6_wrong_version": 0, "ipv6_in_ipv6_too_small": 0, "ipv6_in_ipv6_wrong_version": 0}, "tcp": {"pkt_too_small": 0, "hlen_too_small": 9, "invalid_optlen": 3, "opt_invalid_len": 21, "opt_duplicate": 0}, "udp": {"pkt_too_small": 29, "hlen_too_small": 0, "hlen_invalid": 0, "len_invalid": 0}, "sll": {"pkt_too_small": 0}, "ethernet": {"pkt_too_small": 0}, "ppp": {"pkt_too_small": 0, "vju_pkt_too_small": 0, "ip4_pkt_too_small": 0, "ip6_pkt_too_small": 0, "wrong_type": 0, "unsup_proto": 0}, "pppoe": {"pkt_too_small": 0, "wrong_code": 0, "malformed_tags": 0}, "gre": {"pkt_too_small": 0, "wrong_version": 0, "version0_recur": 0, "version0_flags": 0, "version0_hdr_too_big": 0, "version0_malformed_sre_hdr": 0, "version1_chksum": 0, "version1_route": 0, "version1_ssr": 0, "version1_recur": 0, "version1_flags": 0, "version1_no_key": 0, "version1_wrong_protocol": 0, "version1_malformed_sre_hdr": 0, "version1_hdr_too_big": 0}, "vlan": {"header_too_small": 0, "unknown_type": 0, "too_many_layers": 0}, "ieee8021ah": {"header_too_small": 0}, "vntag": {"header_too_small": 0, "unknown_type": 0}, "ipraw": {"invalid_ip_version": 0}, "ltnull": {"pkt_too_small": 0, "unsupported_type": 0}, "sctp": {"pkt_too_small": 0}, "esp": {"pkt_too_small": 0}, "mpls": {"header_too_small": 0, "pkt_too_small": 0, "bad_label_router_alert": 0, "bad_label_implicit_null": 0, "bad_label_reserved": 0, "unknown_payload_type": 0}, "vxlan": {"unknown_payload_type": 0}, "geneve": {"unknown_payload_type": 0}, "erspan": {"header_too_small": 0, "unsupported_version": 0, "too_many_vlan_layers": 0}, "dce": {"pkt_too_small": 0}, "chdlc": {"pkt_too_small": 0}, "nsh": {"header_too_small": 0, "unsupported_version": 0, "bad_header_length": 0, "reserved_type": 0, "unsupported_type": 0, "unknown_payload": 0}}, "too_many_layers": 0}, "tcp": {"syn": 1970, "synack": 2179, "rst": 3, "active_sessions": 234, "sessions": 360, "ssn_memcap_drop": 0, "ssn_from_cache": 123, "ssn_from_pool": 237, "pseudo": 0, "pseudo_failed": 0, "invalid_checksum": 12920, "midstream_pickups": 0, "pkt_on_wrong_thread": 538327, "ack_unseen_data": 4209, "segment_memcap_drop": 0, "segment_from_cache": 1797, "segment_from_pool": 166, "stream_depth_reached": 4, "reassembly_gap": 2571, "overlap": 1792, "overlap_diff_data": 0, "insert_data_normal_fail": 0, "insert_data_overlap_fail": 0, "memuse": 1214816, "reassembly_memuse": 5577784}, "flow": {"memcap": 0, "total": 11437, "active": 6796, "tcp": 10743, "udp": 694, "icmpv4": 0, "icmpv6": 0, "tcp_reuse": 85, "get_used": 0, "get_used_eval": 0, "get_used_eval_reject": 0, "get_used_eval_busy": 0, "get_used_failed": 0, "wrk": {"spare_sync_avg": 94, "spare_sync": 117, "spare_sync_incomplete": 31, "spare_sync_empty": 0, "flows_evicted_needs_work": 126, "flows_evicted_pkt_inject": 138, "flows_evicted": 375, "flows_injected": 116, "flows_injected_max": 9}, "end": {"state": {"new": 4641, "established": 0, "closed": 0, "local_bypassed": 0, "capture_bypassed": 0}, "tcp_state": {"none": 0, "syn_sent": 115, "syn_recv": 11, "established": 0, "fin_wait1": 0, "fin_wait2": 0, "time_wait": 0, "last_ack": 0, "close_wait": 0, "closing": 0, "closed": 0}, "tcp_liberal": 0}, "mgr": {"full_hash_pass": 2, "rows_per_sec": 65336, "rows_maxlen": 6, "flows_checked": 10154, "flows_notimeout": 5937, "flows_timeout": 4217, "flows_evicted": 4357, "flows_evicted_needs_work": 91}, "spare": 9179, "emerg_mode_entered": 0, "emerg_mode_over": 0, "recycler": {"recycled": 4266, "queue_avg": 50, "queue_max": 370}, "memuse": 46255040}, "defrag": {"ipv4": {"fragments": 0, "reassembled": 0}, "ipv6": {"fragments": 0, "reassembled": 0}, "max_frag_hits": 0}, "flow_bypassed": {"local_pkts": 1865, "local_bytes": 1924786, "local_capture_pkts": 0, "local_capture_bytes": 0, "closed": 0, "pkts": 0, "bytes": 0}, "detect": {"engines": [{"id": 0, "last_reload": "2024-03-06T12:19:33.714390-0800", "rules_loaded": 50759, "rules_failed": 29, "rules_skipped": 0}], "alert": 98, "alert_queue_overflow": 0, "alerts_suppressed": 104}, "app_layer": {"flow": {"http": 0, "ftp": 0, "smtp": 0, "tls": 0, "ssh": 0, "imap": 0, "smb": 0, "dcerpc_tcp": 0, "dns_tcp": 0, "nfs_tcp": 0, "ntp": 0, "ftp-data": 0, "tftp": 0, "ike": 0, "krb5_tcp": 0, "quic": 8, "dhcp": 0, "rfb": 0, "telnet": 0, "rdp": 0, "failed_tcp": 0, "dcerpc_udp": 0, "dns_udp": 65, "nfs_udp": 0, "krb5_udp": 0, "failed_udp": 621}, "tx": {"http": 0, "ftp": 0, "smtp": 0, "tls": 0, "ssh": 0, "imap": 0, "smb": 0, "dcerpc_tcp": 0, "dns_tcp": 0, "nfs_tcp": 0, "ntp": 0, "ftp-data": 0, "tftp": 0, "ike": 0, "krb5_tcp": 0, "quic": 7, "dhcp": 0, "rfb": 0, "telnet": 0, "rdp": 0, "dcerpc_udp": 0, "dns_udp": 291, "nfs_udp": 0, "krb5_udp": 0}, "error": {"http": {"gap": 0, "alloc": 0, "parser": 0, "internal": 0}, "ftp": {"gap": 0, "alloc": 0, "parser": 0, "internal": 0}, "smtp": {"gap": 0, "alloc": 0, "parser": 0, "internal": 0}, "tls": {"gap": 0, "alloc": 0, "parser": 0, "internal": 0}, "ssh": {"gap": 0, "alloc": 0, "parser": 0, "internal": 0}, "imap": {"gap": 0, "alloc": 0, "parser": 0, "internal": 0}, "smb": {"gap": 0, "alloc": 0, "parser": 0, "internal": 0}, "dcerpc_tcp": {"gap": 0, "alloc": 0, "parser": 0, "internal": 0}, "dns_tcp": {"gap": 0, "alloc": 0, "parser": 0, "internal": 0}, "nfs_tcp": {"gap": 0, "alloc": 0, "parser": 0, "internal": 0}, "ntp": {"gap": 0, "alloc": 0, "parser": 0, "internal": 0}, "ftp-data": {"gap": 0, "alloc": 0, "parser": 0, "internal": 0}, "tftp": {"gap": 0, "alloc": 0, "parser": 0, "internal": 0}, "ike": {"gap": 0, "alloc": 0, "parser": 0, "internal": 0}, "krb5_tcp": {"gap": 0, "alloc": 0, "parser": 0, "internal": 0}, "quic": {"gap": 0, "alloc": 0, "parser": 0, "internal": 0}, "dhcp": {"gap": 0, "alloc": 0, "parser": 0, "internal": 0}, "rfb": {"gap": 0, "alloc": 0, "parser": 0, "internal": 0}, "telnet": {"gap": 0, "alloc": 0, "parser": 0, "internal": 0}, "rdp": {"gap": 0, "alloc": 0, "parser": 0, "internal": 0}, "failed_tcp": {"gap": 0}, "dcerpc_udp": {"alloc": 0, "parser": 0, "internal": 0}, "dns_udp": {"alloc": 0, "parser": 0, "internal": 0}, "nfs_udp": {"alloc": 0, "parser": 0, "internal": 0}, "krb5_udp": {"alloc": 0, "parser": 0, "internal": 0}}, "expectations": 0}, "memcap_pressure": 0, "memcap_pressure_max": 0, "http": {"memuse": 0, "memcap": 0}, "ftp": {"memuse": 0, "memcap": 0}, "file_store": {"open_files": 0}}, "host": "sensor_192_168_22_229"}


Subtasks 1 (0 open1 closed)

Bug #6863: BUG_ON triggered from TmThreadsInjectFlowById (7.0.x backport)ClosedJeff LucovskyActions

Related issues 1 (0 open1 closed)

Related to Suricata - Bug #6957: Assert: BUG_ON(id <= 0 || id > (int)thread_store.threads_size);ClosedJeff LucovskyActions
Actions #1

Updated by Jeff Lucovsky about 2 months ago

  • Label Needs backport to 7.0 added
Actions #2

Updated by Jeff Lucovsky about 2 months ago

  • Status changed from New to In Progress
  • Assignee changed from OISF Dev to Jeff Lucovsky
Actions #3

Updated by Jeff Lucovsky about 2 months ago

The problem occurs when midstream pickups are enabled.

Actions #4

Updated by Jeff Lucovsky about 2 months ago

  • Status changed from In Progress to In Review
Actions #5

Updated by Jeff Lucovsky about 1 month ago

  • Affected Versions 8.0.0-beta1 added
  • Affected Versions deleted (7.0.3)
Actions #6

Updated by Jeff Lucovsky about 1 month ago

  • Target version changed from TBD to 8.0.0-beta1
  • Affected Versions 7.0.3 added
  • Affected Versions deleted (8.0.0-beta1)
Actions #7

Updated by OISF Ticketbot about 1 month ago

  • Subtask #6863 added
Actions #8

Updated by OISF Ticketbot about 1 month ago

  • Label deleted (Needs backport to 7.0)
Actions #9

Updated by Jeff Lucovsky about 1 month ago

  • Status changed from In Review to Resolved
Actions #10

Updated by Jeff Lucovsky 13 days ago

  • Related to Bug #6957: Assert: BUG_ON(id <= 0 || id > (int)thread_store.threads_size); added
Actions

Also available in: Atom PDF