Project

General

Profile

Actions

Bug #7812

open

TypeError in rule.py because of unhandled decode error (bare except)

Added by Walter Doekes about 23 hours ago. Updated about 13 hours ago.

Status:
New
Priority:
Normal
Assignee:
Target version:
-
Affected Versions:
Effort:
Difficulty:
Label:

Description

There is this code in suricata/update/rule.py:

def parse_fileobj(fileobj, group=None):
...
for line in fileobj:
try:
if type(line) == type(b""):
line = line.decode()
except:
pass
if line.rstrip().endswith("\\"):
buf = "%s%s " % (buf, line.rstrip()[0:-1])
continue

It's called by this code:

rules += rule_mod.parse_fileobj(io.BytesIO(entry.content), entry.filename)

So, it takes some kind of entry.content and passes it to the function as bytes.

In the function, it tries the conversion from bytes to string:

try:
if type(line) == type(b""):
line = line.decode()
except:
pass

But if it fails, it simply gives up and continues: now it fails later on because the line is a byte-string, not a unicode-string.

That bare `except` is very poor style, and hides what the real bug is.

I don't know what content caused it to fail, but likely a better line would have been:

line = line.decode('utf-8', 'replace')

This code will not fail in the case of UnicodeDecodeError()s.

Original problem:

Traceback (most recent call last):
File "/usr/bin/suricata-update", line 36, in <module>
sys.exit(main.main())
File "/usr/lib/suricata/python/suricata/update/main.py", line 1413, in main
sys.exit(_main())
File "/usr/lib/suricata/python/suricata/update/main.py", line 1259, in _main
rules += rule_mod.parse_fileobj(io.BytesIO(entry.content), entry.filename)
File "/usr/lib/suricata/python/suricata/update/rule.py", line 322, in parse_fileobj
if line.rstrip().endswith("\\"):
TypeError: endswith first arg must be bytes or a tuple of bytes, not str

Actions #1

Updated by Shivani Bhardwaj about 22 hours ago

  • Project changed from Suricata to Suricata-Update
  • Target version deleted (TBD)
  • Effort deleted (low)
  • Difficulty deleted (low)
  • Affected Versions 1.3.6 added
  • Affected Versions deleted (7.0.0, 7.0.1, 7.0.3, 7.0.4, 7.0.6, 7.0.7, 7.0.8, 7.0.9, 8.0.0-beta1, 8.0.0-rc1, 7.0.11, 8.0.0, 7.0.12)
Actions #2

Updated by Jason Ish about 16 hours ago

I'd still rather fail if unicode fails, but just error on that rule and continue, rather than aborting the program.

No idea on the input that causes this? Any suspected ruleset of doing so?

Actions #3

Updated by Walter Doekes about 15 hours ago

Jason Ish wrote in #note-2:

I'd still rather fail if unicode fails, but just error on that rule and continue, rather than aborting the program.

No idea on the input that causes this? Any suspected ruleset of doing so?

No idea. I was getting these errors occasionally from cron.

I'd have to patch the code to report the failing lines. I'm off to holiday, so I will probably not get around to add any of those in the coming weeks. Maybe you can patch the code to report it and get feedback from multiple systems in the mean time =)

Actions #4

Updated by Jason Ish about 13 hours ago

I think I've created some rules that show the issue. But in general, I've not seen this across all the public rulesets we index.

Anyways, should be fixed when you're doneholiday.

Actions

Also available in: Atom PDF