mulnera

Mulnera takes a URL and tells you who and what is mentioned on that page. The general design is a FastAPI application with both a web UI and a documented REST API, IP-based rate limiting, and a Docker Compose setup that brings Redis up alongside it. I mainly came up with this project as a way to learn named-entity recognition (NER), since I’d done so much in NLP but never actually touched NER. It was also good practice in using local/spaCy models.

The name comes from MULtilingual NER API.

How it works

You supply a list of URLs to check. The program scrapes the content, works out what language it’s in, pulls out the named entities and any security-relevant indicators, flags anything on a watchlist, and optionally translates the results into English. It was built with OSINT work in mind, but there’s nothing security-specific about the core of it. It could be used to monitor for any keywords using any list of URLs.

Content language is detected automatically, then mapped to a spaCy model by language code. English and German are configured out of the box (just for testing, really), and anything else falls back to spaCy’s multilingual model, which is less accurate but means unsupported languages still get processed rather than rejected. Adding a language is just a few lines of YAML and a model download. You can also register a transformer-based model as its own option if you’d rather have the accuracy than the speed, which is roughly a twenty-fold tradeoff.

Alongside the NER, it pattern-matches IOCs - IPv4 and IPv6 addresses, URLs, domains, email addresses, MD5/SHA-1/SHA-256 hashes, and CVE identifiers - and checks the text against configurable watchlists, which ship with the usual threat actors and malware families but are just a YAML map of category to terms. One thing worth knowing: the watchlists are matched against the original text, before any translation, so terms need to be written the way they’d actually appear in the source language.

Translation runs through the Google Translate API and is entirely optional; without credentials everything else still works. Results are cached in Redis, mostly to keep from paying to translate the same entity repeatedly. I guess we could put a translation step on both sides of the pipeline, so your keywords/IOCs could stay in English and be run against foreign sites. Maybe someday.

You can find the code here on GitHub.