Suricata

Thinking about 3 hooks:
file:new
file:update
file:finish

New would be for checking the name.
Update for data.
Finish for size and hashes.

Since file inspection is still a bit basic, it would require a bit of refactoring of how rule inspection is done.

pass file:new any any -> any any (file.name; file.name; content:".pdf"; nocase; endswith; sid:1;)
drop file:new any any -> any any (msg:"drop non-pdf files"; sid:2;)

Perhaps a special hook would be required for the magic logic? It inspects only the first X bytes IIRC.