Project

General

Profile

Actions

Task #570

open

tracking: memory fragmentation

Added by Victor Julien almost 9 years ago. Updated 4 months ago.

Status:
New
Priority:
Normal
Assignee:
Target version:
Effort:
Difficulty:
Label:

Description

During start up (mostly detection engine init) we alloc and free enormous amounts of small memory blocks. This leads to memory fragmentation.

This task is a place holder for tracking that. Addressing all/most of this is going to be a long term effort.


Files

fragmentation-leak.png (150 KB) fragmentation-leak.png Gianni Tedesco, 05/24/2021 11:26 PM

Subtasks 3 (1 open2 closed)

Bug #492: memory not released after live rule swapClosedAnoop Saldanha06/30/2012Actions
Optimization #564: pattern id: rewrite pattern id assignmentClosedAnoop Saldanha09/28/2012Actions
Optimization #573: reduce allocs in signature parsingNewOISF DevActions
Actions #1

Updated by Victor Julien almost 8 years ago

  • Target version set to TBD
Actions #2

Updated by Victor Julien almost 2 years ago

  • Tracker changed from Optimization to Task
  • Subject changed from memory fragmentation to tracking: memory fragmentation
Actions #3

Updated by Gianni Tedesco 4 months ago

Actually the main culprit of heap fragmentation is reassembly. Keep doing increasing realloc() as we do for tcp stream reassembly is worst-case behaviour, and it can cause several gigabytes of excess heap to be used up. The only way to get it back is to attach gdb and call malloc_trim(0).

Reassembly buffers need to be a chain of fixed size blocks to avoid this situation. You can copy the blocks out to the final (per-thread) buffer at the end, just before injecting the packet for inspection.

Actions #4

Updated by Gianni Tedesco 4 months ago

Or at least, appears to be around reassembly and state for protocols at least. Will investigate further.

Actions

Also available in: Atom PDF