Add option to strip common reply/forward markers from email subject#790
Conversation
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 34fe9bee36
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| SUBJECT_MARKER_RE = re.compile(r""" | ||
| ^\s* | ||
| (?:(?:\[[^\]]+\]\s*)*) |
There was a problem hiding this comment.
Preserve non-reply bracketed subject prefixes
When marker stripping is enabled for subjects that begin with a meaningful bracketed prefix before the reply marker, such as [CASE-123] RE: malware or [TLP:AMBER] RE: incident, this subpattern consumes and drops the bracketed text along with RE:. That loses analyst-relevant context even though the option is described as removing common reply/forward markers; please avoid matching arbitrary bracketed prefixes or restrict it to known disposable markers.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
To use Codex here, create an environment for this repo.
Motivation
RE:,Fwd:,WG:) which pollute theemail-subjectattribute; the module should optionally normalize these.Description
remove_subject_markerstomoduleconfigand update themoduleinfo["features"]text to document it.dict_handlerso whenremove_subject_markersis enabled the module callsstrip_subject_markers(email_object)before producing the MISP object.SUBJECT_MARKER_RE,remove_common_subject_markers(subject)andstrip_subject_markers(email_object)helpers to repeatedly remove leading reply/forward markers and bracketed tags while avoiding replacing the subject with an empty string.email_object.get_attributes_by_relation("subject")and leaves unchanged subjects that would become empty after stripping.Testing
python -m py_compile misp_modules/modules/import_mod/email_import.pyand it succeeded.pymisp.toolsenvironment validatingremove_common_subject_markersagainst multiple example subjects andstrip_subject_markersbehavior on a mock object, and all assertions passed.git diff --check) with no reported issues.Codex Task