Bug #7812
openTypeError in rule.py because of unhandled decode error (bare except)
Description
There is this code in suricata/update/rule.py:
def parse_fileobj(fileobj, group=None):
...
for line in fileobj:
try:
if type(line) == type(b""):
line = line.decode()
except:
pass
if line.rstrip().endswith("\\"):
buf = "%s%s " % (buf, line.rstrip()[0:-1])
continue
It's called by this code:
rules += rule_mod.parse_fileobj(io.BytesIO(entry.content), entry.filename)
So, it takes some kind of entry.content and passes it to the function as bytes.
In the function, it tries the conversion from bytes to string:
try:
if type(line) == type(b""):
line = line.decode()
except:
pass
But if it fails, it simply gives up and continues: now it fails later on because the line is a byte-string, not a unicode-string.
That bare `except` is very poor style, and hides what the real bug is.
I don't know what content caused it to fail, but likely a better line would have been:
line = line.decode('utf-8', 'replace')
This code will not fail in the case of UnicodeDecodeError()s.
Original problem:
Traceback (most recent call last):
File "/usr/bin/suricata-update", line 36, in <module>
sys.exit(main.main())
File "/usr/lib/suricata/python/suricata/update/main.py", line 1413, in main
sys.exit(_main())
File "/usr/lib/suricata/python/suricata/update/main.py", line 1259, in _main
rules += rule_mod.parse_fileobj(io.BytesIO(entry.content), entry.filename)
File "/usr/lib/suricata/python/suricata/update/rule.py", line 322, in parse_fileobj
if line.rstrip().endswith("\\"):
TypeError: endswith first arg must be bytes or a tuple of bytes, not str
Updated by Shivani Bhardwaj about 18 hours ago
- Project changed from Suricata to Suricata-Update
- Target version deleted (
TBD) - Effort deleted (
low) - Difficulty deleted (
low) - Affected Versions 1.3.6 added
- Affected Versions deleted (
7.0.0, 7.0.1, 7.0.3, 7.0.4, 7.0.6, 7.0.7, 7.0.8, 7.0.9, 8.0.0-beta1, 8.0.0-rc1, 7.0.11, 8.0.0, 7.0.12)
Updated by Jason Ish about 12 hours ago
I'd still rather fail if unicode fails, but just error on that rule and continue, rather than aborting the program.
No idea on the input that causes this? Any suspected ruleset of doing so?
Updated by Walter Doekes about 10 hours ago
Jason Ish wrote in #note-2:
I'd still rather fail if unicode fails, but just error on that rule and continue, rather than aborting the program.
No idea on the input that causes this? Any suspected ruleset of doing so?
No idea. I was getting these errors occasionally from cron.
I'd have to patch the code to report the failing lines. I'm off to holiday, so I will probably not get around to add any of those in the coming weeks. Maybe you can patch the code to report it and get feedback from multiple systems in the mean time =)
Updated by Jason Ish about 8 hours ago
I think I've created some rules that show the issue. But in general, I've not seen this across all the public rulesets we index.
Anyways, should be fixed when you're doneholiday.