FEAT Bijection

[romanlutz]

https://github.com/[haizelabs/bijection-learning](https://github.com/haizelabs/bijection-learning) looks interesting and is covered in the MLCommons jailbreak taxonomy. If someone is interested in tackling this they should comment below to claim it and first come up with a plan to incorporate it. After we agree on the plan it's a good time to write the PR.

[sajisanchu1913-source]

Hi @romanlutz
I'd love to work on this. My proposed plan:

1. Implement a BijectionConverter that generates a random character-to-character mapping
2. Add a system prompt template that teaches the target LLM the bijection mapping before the attack
3. Add unit tests mocking the converter
4. Add an example notebook demonstrating the attack
   Would this approach work? Happy to discuss further before coding.

[Raulster24]

@romanlutz @sajisanchu1913-source, a few design notes from reading the paper (arXiv:2410.01294, Haize Labs) and the reference implementation, in case they're useful for scoping this.

Some load-bearing details from the reference code (that_good_infra/transformations.py, run_bijection_attacks.py):

1. Complexity control is the core result. The mapping keeps fixed_size letters as identity and shuffles the rest. The paper's "safety-capability" finding is that more capable models are best attacked by more complex bijections, so fixed_size is worth exposing as an explicit, tunable parameter rather than a fixed random map.
2. It is multi-turn, with two distinct phases. create_teaching_and_multiturn_convo builds a system message, then teaching demonstrations (num_teaching_shots), then real practice exchanges (num_multi_turns) as alternating user/assistant turns, both encoded, then the encoded query, then an optional assistant prefill.
3. Multiple bijection types: letter, digit (num_digits), plus tokenizer and word variants.
4. Decoding is mechanical (inverse map), and trials is a best-of-n loop with a fresh bijection per trial.

One thing worth deciding early: a converter takes one string and returns one string, so it can't by itself produce the multi-turn practice exchanges the paper relies on. FlipAttack handles a similar shape by shipping as both a FlipConverter and a separate FlipAttack, which might be a good model here:

• Converter: the mapping and encoding (bijection_type / fixed_size / num_digits), an optional teaching prompt rendered into one output (like CodeChameleonConverter), and the inverse map exposed for decoding.
• Attack: places the teaching and practice turns in prepended_conversation (as ManyShotJailbreakAttack and FlipAttack do), sweeps trials bijections, decodes each response, and scores via AttackScoringConfig.

Hope this is useful for the design.

[sajisanchu1913-source]

Thanks @Raulster24 for the detailed notes! Updated plan based on your analysis:

1. BijectionConverter — handles encoding with tunable fixed_size and bijection_type parameters
2. BijectionAttack — handles multi-turn teaching shots and practice exchanges using prepended_conversation (similar to FlipAttack)
3. Expose inverse map for decoding responses
4. Unit tests for both converter and attack
5. Example notebook
   Does this align with what you had in mind @romanlutz?

[u7k4rs6]

Quick note since the PR is now linked here: I worked on this in parallel with some discussion with @romanlutz, but didn't claim the issue here first, which was the wrong order on my part. @sajisanchu1913-source , apologies if I stepped on your work.

[romanlutz]

Technically @sajisanchu1913-source was first, @Raulster24 second and @u7k4rs6 third here. I'll let you all figure out who can take this but if you can't then it's by order of commenting. Please note that you can always suggest your own issues, too, and you don't need to wait for us to open any. I promise we do respond. It may take a few days to get back in some cases but we try to be as responsive as possible. We merge on average 6+ PRs a day so there's a considerable amount of work managing all of these workstreams. Thanks for your patience.

[sajisanchu1913-source]

Thank you @romanlutz
I am happy to take this on since I commented first.
@Raulster24 thank you for the detailed design notes, they were very helpful in refining the plan.
@u7k4rs6 no worries at all, appreciate the transparency.
I will proceed with the plan — BijectionConverter for encoding and BijectionAttack for the multi-turn teaching flow, following the FlipAttack pattern. Will post updates here before submitting a PR.

[sajisanchu1913-source]

Hi @romanlutz

Here's a quick update on my implementation:

For the converter, I built BijectionConverter which generates a random letter-to-letter mapping with tunable fixed_size and bijection_type parameters. It encodes prompts using the mapping and exposes an inverse map for decoding responses. I followed the FlipConverter pattern for this.

For the attack, I built BijectionAttack which handles the multi-turn teaching phase by placing teaching demonstrations in prepended_conversation, similar to how FlipAttack works. It takes num_teaching_shots, bijection_type, and fixed_size as parameters.

I also added 11 unit tests for the converter and 5 unit tests for the attack, along with a notebook demonstrating usage.

Would love your feedback before this gets merged! PR is up at #1942.