Feature #1201
closedFeature #2303: file-store enhancements (aka file-store v2): deduplication; hash-based naming; json metadata and cleanup tooling
file-store metadata in JSON format
Description
Currently we write metadata for filestore like so:
root@LTS-64-1:~# cat /var/log/suricata/files/file.2.meta TIME: 06/08/2014-14:15:08.392536 SRC IP: 31.186.225.23 DST IP: 10.0.2.15 PROTO: 6 SRC PORT: 80 DST PORT: 53064 HTTP URI: /a/11016/26510/105352-2.js?&cb=0.15413070828462816&tk_st=1&rf=http://edition.cnn.com/&rp_s=c&tg_i.site=cnn_international&tg_i.rollup=homepage&tg_i.pagetype=main&p_pos=btf&p_screen_res=1680x945 HTTP HOST: optimized-by.rubiconproject.com HTTP REFERER: http://ads.cnn.com/html.ng/site=cnn_international&cnn_intl_pagetype=main&cnn_intl_position=728x90_bot&cnn_intl_rollup=homepage&page.allowcompete=no¶ms.styles=fs&Params.User.UserID=53944fdb05ba670a3c6b805990008512&transactionID=14022297068343779055472671&tile=895079222045&domId=6c5b4c103152e6e3&kxid=ojke0w8tp&kxseg= HTTP USER AGENT: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:29.0) Gecko/20100101 Firefox/29.0 FILENAME: /a/11016/26510/105352-2.js MAGIC: HTML document, ASCII text, with very long lines STATE: CLOSED MD5: 2a5d49f36faaf44d1e115f01bee3f499 SIZE: 2175 root@LTS-64-1:~#
It would be beneficial if we can do JSON format logging as well for the meta files.
Updated by Victor Julien almost 10 years ago
- Assignee set to Anonymous
- Target version set to TBD
Updated by Andreas Moe over 9 years ago
Was just about to write an issue about this problem. Seeing that the general direction (with the eve-format) is trending towards JSON this would be a good thing to move to the JSON format aswell.
Updated by Andreas Moe over 9 years ago
Was looking through the source of suricata to see if this ticket could be solved quickly. Then i say the function "LogFileWriteJsonRecord" in log-file.c. Isnt this an implementation for this feature ticket?
Updated by Victor Julien over 9 years ago
LogFileWriteJsonRecord() isn't being used for creating/writing an 'meta' file. It's used to generate files-json.log lines. It could be used to generate the meta file with relatively little effort I think.
Updated by Andreas Moe over 9 years ago
I rewrote the log-filestore.c file to give a json formated output (not using the jansson json object but just changing the fprintf contents).
Updated by Andreas Moe over 9 years ago
https://github.com/inliniac/suricata/pull/1161
This is with the current regular format and with option for JSON format of the data.
Updated by Victor Julien over 9 years ago
- Status changed from New to Assigned
- Assignee changed from Anonymous to Andreas Moe
- Target version changed from TBD to 3.0RC2
Updated by Victor Julien over 8 years ago
- Target version changed from 3.0RC2 to 70
Updated by Victor Julien almost 8 years ago
- Target version changed from 70 to TBD
Updated by Victor Julien over 6 years ago
- Assignee changed from Andreas Moe to Jason Ish
- Target version changed from TBD to 70
The meta record should probably simply be the 'fileinfo' record written to a .json file?
Updated by Victor Julien about 6 years ago
- Status changed from Assigned to Closed
- Target version changed from 70 to 4.1beta1