[Paper] RedSage: A Cybersecurity Generalist LLM

Source: arXiv - 2601.22159v1

Overview

The paper introduces RedSage, an open‑source large language model (LLM) that acts as a cybersecurity‑focused assistant you can run locally. By combining massive, curated cybersecurity text with a novel “agentic augmentation” pipeline, the authors show that a model can become both a domain expert and a strong general‑purpose LLM—without relying on proprietary APIs that risk leaking sensitive data.

Key Contributions

• Massive domain‑specific corpus: 11.8 B tokens collected from 28.6 K high‑quality cybersecurity documents (frameworks, attack techniques, tools, etc.).
• Agentic augmentation pipeline: Simulates expert security analyst workflows to auto‑generate 266 K multi‑turn, cybersecurity‑oriented conversation samples for supervised fine‑tuning.
• RedSage model: An 8 B‑parameter open‑source LLM that incorporates both the curated pre‑training data and the augmented fine‑tuning data, delivering strong performance on security tasks while remaining runnable on a single GPU.
• RedSage‑Bench: A new benchmark suite (30 K multiple‑choice + 240 open‑ended Q&A) covering knowledge, skills, and tool usage in cybersecurity, plus evaluation on existing security and general‑LLM benchmarks.
• Open release: All model weights, datasets, and training code are publicly released, enabling reproducibility and community extensions.

Methodology

1. Data Curation – The team scraped the public web, applied large‑scale filtering (e.g., language detection, relevance classifiers), and manually vetted sources to assemble a clean, diverse set of cybersecurity texts.
2. Continual Pre‑training – Starting from a base open‑source LLM, they performed additional pre‑training on the 11.8 B token corpus, allowing the model to internalize security terminology and concepts.
3. Agentic Augmentation – They built a “virtual analyst” that follows scripted security workflows (threat hunting, incident response, tool configuration). The analyst interacts with the partially trained model, generating realistic multi‑turn dialogues that mimic real analyst‑assistant exchanges. These dialogues become supervised fine‑tuning data.
4. Fine‑tuning – The model is fine‑tuned on a mix of the agentic dialogues and general instruction data (e.g., Alpaca, OpenChat) to retain broad language abilities while sharpening security expertise.
5. Evaluation – RedSage is tested on RedSage‑Bench and on established security benchmarks (CTI‑Bench, CyberMetric, SECURE) as well as on standard LLM leaderboards to gauge both domain and general performance.

Results & Findings

• Cybersecurity benchmarks: RedSage outperforms baseline open‑source models by +3.2 – +5.6 points (average) across CTI‑Bench, CyberMetric, and SECURE.
• General LLM tasks: On the Open LLM Leaderboard, RedSage gains +4.8 – +5.1 points over the same baselines, indicating that domain‑specific training does not hurt—and can even improve—general reasoning.
• Ablation studies: Removing the agentic augmentation drops performance by ~2 points on security tasks, confirming its value.
• Efficiency: The 8 B‑parameter model runs comfortably on a single RTX 4090 (≈12 GB VRAM) with inference latency suitable for interactive assistants.

Practical Implications

• Secure, on‑premise assistants: Organizations can deploy RedSage inside firewalls, eliminating the data‑exfiltration risk inherent in cloud‑only APIs.
• Incident response automation: RedSage can suggest mitigation steps, generate detection queries (e.g., Splunk, Elastic), or draft CVE summaries, accelerating analyst workflows.
• Tool integration: Because the model was trained on real‑world security tool documentation, it can produce accurate command‑line snippets for tools like Nmap, Metasploit, or Wireshark.
• Education & training: Security bootcamps and CTF platforms can embed RedSage as a tutoring bot that answers technique‑specific questions without exposing proprietary data.
• Community extensibility: The open datasets and augmentation pipeline enable teams to add organization‑specific policies, internal tooling docs, or threat‑intel feeds, tailoring the assistant to their environment.

Limitations & Future Work

• Scope of knowledge – The corpus reflects publicly available information; emerging zero‑day techniques or proprietary tool internals may be missing.
• Hallucination risk – Like any LLM, RedSage can generate plausible‑but‑incorrect advice; a verification layer (e.g., rule‑based checks) is recommended for critical actions.
• Scale trade‑off – The 8 B model balances performance and hardware requirements; larger models could yield higher accuracy but demand more compute.
• Evaluation breadth – Benchmarks focus on multiple‑choice and short‑answer formats; real‑world continuous dialogue and multi‑modal inputs (e.g., logs, packet captures) remain unexplored.
• Future directions proposed by the authors include: expanding the augmentation pipeline to cover red‑team/blue‑team adversarial scenarios, integrating retrieval‑augmented generation for up‑to‑date threat intel, and exploring multi‑modal inputs (e.g., code snippets, network diagrams).

Authors

• Naufal Suryanto
• Muzammal Naseer
• Pengfei Li
• Syed Talal Wasim
• Jinhui Yi
• Juergen Gall
• Paolo Ceravolo
• Ernesto Damiani

Paper Information

• arXiv ID: 2601.22159v1
• Categories: cs.CR, cs.AI, cs.CL
• Published: January 29, 2026
• PDF: Download PDF