Calibrating when language models refuse. BLASPHEMOUS is a research toolkit for analyzing and intervening on refusal behavior in transformer models. It combines refusal-direction geometry, Bayesian search, weight ablation, and causal mediation analysis in a reproducible pipeline.
The project explores how refusal-related behavior is represented across model layers, how targeted interventions change that behavior, and how to measure the effects alongside capability-preservation metrics. Results depend on the model, prompt set, and experiment settings. BLASPHEMOUS does not guarantee a particular refusal rate or unchanged capabilities.
Given a Hugging Face model or local model directory, the pipeline:
- Analyzes refusal geometry using alignment, cone, and layer analyses.
- Builds a direction manifold from refusal and harmless activations.
- Searches intervention parameters with Optuna, tracking refusal, KL divergence, and related metrics.
- Applies the selected intervention across attention and MLP components, with iterative passes available.
- Runs causal mediation analysis to estimate which components contribute to measured refusal behavior.
- Saves the resulting model and metadata so the experiment can be inspected and benchmarked.
The main ablation methods are orthogonal projection and LoRA. The auto method selects based on analysis. Optimal transport is experimental in the CLI and currently falls back to the projection path.
- Python 3.10 or newer
- A CUDA-capable GPU is recommended for practical model loading and optimization; CPU is available but may be slow
- Enough memory for the model and the original-model KL reference copy
Install the package:
pip install -e .Optional dependencies:
pip install -e ".[quantization]" # 4-bit BitsAndBytes support
pip install -e ".[research]" # plotting and research visualization tools
pip install -e ".[all]" # all optional dependenciesRun an experiment on a local model or a Hugging Face model ID:
blasphemous ./Qwen2.5-0.5B-Instruct \
--output ./runs/qwen-experiment \
--trials 100 \
--method autoThe CLI prints analysis, optimization, and final evaluation metrics. The modified model and experiment metadata are written to the output path. Review the output and benchmark it against the original model before using it in another setting.
Analyze a model without running optimization or modifying its weights:
blasphemous ./Qwen2.5-0.5B-Instruct --analyze-only --device cudaSee the Quickstart for the documented experiment flow.
| Method | Description |
|---|---|
projection |
Orthogonal projection along selected refusal directions |
lora |
LoRA-based ablation path |
auto |
Chooses a method based on analysis |
optimal_transport |
Experimental CLI option that currently falls back to projection |
The optimizer can tune direction selection, layer weighting, intervention strength, multi-pass compensation, and KL regularization. Causal mediation analysis is enabled by default. Options such as --no-causal, --multi-pass, --target-all, --layer-tuning, --lambda-kl, and --mu-ouroboros let you adjust the experiment. Run blasphemous --help for the current CLI options.
Run the full pipeline from Python:
from blasphemous import run, benchmark_model
result = run(
model_name="./Qwen2.5-0.5B-Instruct",
output_path="./runs/qwen-experiment",
n_trials=100,
method="auto",
lambda_kl=1.0,
)
report = benchmark_model("./runs/qwen-experiment")
print(f"Refusal rate: {report.harmful_refusal_rate:.3f}")
print(f"Harmless refusal rate: {report.harmless_refusal_rate:.3f}")
print(f"KL divergence: {report.kl_divergence:.6f}")The result includes the geometry analysis, direction manifold, optimization result, final commit metrics, and elapsed time. See the benchmark guide for report fields and comparison helpers.
Analysis components are also available independently:
from blasphemous.analyze import analyze, profile_layers, measure_erasure
report = analyze(model, tokenizer, device="cuda")
profile = profile_layers(report)The benchmark reports harmful-prompt refusal, harmless-prompt refusal, a balanced score, KL divergence from the original model, and residual refusal signal. These are measurements against the project's prompt suites. They are not guarantees of general behavior or safety, and they can vary with prompt coverage and model choice.
For meaningful comparisons, record the base model, prompt set, run parameters, and saved metadata. Compare both refusal metrics and capability-preservation measures, inspect representative generations, and evaluate on held-out prompts relevant to your use case.
from blasphemous import benchmark_model, compare_reports
baseline = benchmark_model("./Qwen2.5-0.5B-Instruct")
experiment = benchmark_model("./runs/qwen-experiment")
comparison = compare_reports(baseline, experiment)More details are in the Benchmark guide and Release checklist.
core/analyze.py: refusal geometry analysiscore/extract.py: direction manifold constructioncore/optimize.py: Optuna parameter searchcore/commit.py: model intervention, causal weighting, and artifact metadatacore/benchmark.py: evaluation and report comparisondocs/: quickstart, benchmark, and release documentation
BLASPHEMOUS builds on ideas and methods developed in:
- OBLITERATUS, including iterative ablation and residual direction re-extraction.
- Heretic, including broad component targeting and continuous direction selection.
Please consult and cite the original projects when using the related methods in research.
Issues and pull requests are welcome. For research changes, include the model, prompts, settings, and evaluation evidence needed to reproduce the result.
See LICENSE for details.
