Project

General

Profile

Actions

Support #4661

closed

Why calculate the MD5 value of the file at each packet

Added by yida zhang about 1 year ago. Updated 7 months ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Affected Versions:
Label:

Description

Why is the MD5 of the file calculated in the AppendData function?

static int AppendData(File *file, const uint8_t *data, uint32_t data_len)
{
    if (StreamingBufferAppendNoTrack(file->sb, data, data_len) != 0) {
        SCReturnInt(-1);
    }

#ifdef HAVE_NSS
    if (file->md5_ctx) {
        HASH_Update(file->md5_ctx, data, data_len);
    }
    if (file->sha1_ctx) {
        HASH_Update(file->sha1_ctx, data, data_len);
    }
    if (file->sha256_ctx) {
        HASH_Update(file->sha256_ctx, data, data_len);
    }
#endif
    SCReturnInt(0);
}

In this way, MD5 must be calculated for each packet, and there are many thread lock conflicts.

Is it not possible to calculate the MD5 when the file is closed?

Actions #1

Updated by Victor Julien 9 months ago

  • Tracker changed from Optimization to Support
  • Status changed from New to Closed
  • Assignee deleted (Victor Julien)

The files are processing in streaming mode, so at the end of the file we most likely won't have all the data. In recent code we switched to rust hashing which should be lockless.

Actions #2

Updated by yida zhang 7 months ago

Victor Julien wrote in #note-1:

The files are processing in streaming mode, so at the end of the file we most likely won't have all the data. In recent code we switched to rust hashing which should be lockless.

Great! Very much looking forward to it!

Actions

Also available in: Atom PDF