yara-x: Unvalidated deserialization in safe `Rules::deserialize` allows memory corruption and UB

> [!NOTE] > This finding was identified during an agentic unsafe Rust code review performed by Gemini AI, followed by human review and verification.

The Issue

The crate exports a public safe API [Rules::deserialize](https://github.com/VirusTotal/yara-x/blob/5bd1f35db783679c90a3ea1a66bd15fe4e55bef1/lib/src/compiler/rules.rs#L187-L254) accepting any generic byte sequence B: AsRef<[u8]>. It restores compiled rule structures directly from raw bytes using bincode::serde::decode_from_slice.

This decoded Rules struct contains internal lookup tables, including sub_patterns: Vec<(PatternId, SubPattern)>, atoms: Vec<SubPatternAtom>, and lit_pool: BStringPool. Subsequent safe operations assume these internal tables satisfy strict structural invariants:

• Rules::get_sub_pattern executes unsafe { self.sub_patterns.get_unchecked(sub_pattern_id.0 as usize) }. If untrusted serialized bytes contain an atom referencing an out-of-bounds SubPatternId, calling get_sub_pattern during scanning triggers an out-of-bounds memory read (Undefined Behavior).

https://github.com/VirusTotal/yara-x/blob/5bd1f35db783679c90a3ea1a66bd15fe4e55bef1/lib/src/compiler/rules.rs#L355-L360

• Metadata::next() extracts string metadata via unsafe { s.to_str_unchecked() }. If serialized bytes corrupt lit_pool indices or structural data, to_str_unchecked constructs a &str pointing to invalid UTF-8 bytes (Undefined Behavior).

https://github.com/VirusTotal/yara-x/blob/5bd1f35db783679c90a3ea1a66bd15fe4e55bef1/lib/src/models.rs#L204-L210

Because passing malformed or untrusted data to Rules::deserialize induces Undefined Behavior in subsequent safe calls (Scanner::new, Scanner::scan) without any unsafe blocks in caller code, this API is unsound.

<details><summary>Minimal Reproduction (Miri / Native Crash)</summary>

Zip file with crashing_payload: crashing_payload.zip

We have a payload crashing_payload.bin where only a single byte in the structural metadata tail is mutated (changing a SubPatternId from 1 to 248 while keeping the WebAssembly bytecode completely untouched and valid).

Below is the self-contained verification script which compiles and runs against the official unmodified yara-x v1.17.0 crate:

use yara_x::{Rules, Scanner};

fn main() {
    // Embed the crashing payload generated by the fuzzer at compile time.
    let serialized = include_bytes!("crashing_payload.bin");
    println!("Loaded embedded crashing payload, length: {}", serialized.len());

    // Deserialize. On unmodified library, this succeeds because the WASM and headers
    // are pristine and structural corruption isn't validated.
    if let Ok(deserialized) = Rules::deserialize(serialized) {
        println!("Deserialization succeeded! Running scanner...");
        let mut scanner = Scanner::new(&deserialized);
        
        // Run the standard scan, which will execute the WASM and trigger the out-of-bounds read!
        let _ = scanner.scan(b"lorem ipsum dolor sit amet");
        println!("Scanner finished.");
    } else {
        println!("Deserialization failed!");
    }
}

1. Miri Trace

NOTE: This needs to be run with 1.17.0. I haven't tested this against other versions.

Unfortunately I was able to get a miri trace, but I'm not able to reproduce it right now because of lockfile changes. If you're trying this out be sure to use MIRIFLAGS="-Zmiri-disable-stacked-borrows"

2. Segfault / panics

When run natively (without Miri or any sanitizers) on a standard Linux platform, the process immediately segfaults:

$ cargo run --bin verify
Loaded embedded crashing payload, length: 11777
Deserialization succeeded! Running scanner...
Segmentation fault (core dumped)

And with a newer compiler (which appears to have debug assertions in get_unchecked)

Deserialization succeeded! Running scanner...

thread 'main' (997442) panicked at lib/src/compiler/rules.rs:404:36:
unsafe precondition(s) violated: slice::get_unchecked requires that the index is within the slice

This indicates a bug in the program. This Undefined Behavior check is optional, and cannot be relied on for safety.
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

</details>

<details><summary>Suggested Fix</summary>

To uphold Rust soundness guarantees, either mark Rules::deserialize as pub unsafe fn deserialize with a formal /// # Safety contract documenting that callers are responsible for verifying the authenticity and structural integrity of the input bytes (e.g. via cryptographic signatures), or replace all internal get_unchecked and to_str_unchecked calls on deserialized data structures with safe bounds checks (.get()) and UTF-8 validation (std::str::from_utf8).

</details>

---