中文 README

MATE: Policy-Aware Security Auditing for Mobile Agents

MATE: Policy-Aware Security Auditing for Mobile Agents via Synthesis-Driven Trajectory Learning

Changyue Jiang, Jiayi Wang, Xin Wen, Jiarun Dai, Geng Hong, and Xudong Pan

Fudan University and Shanghai Innovation Institute

Paper (coming soon) | Models (coming soon) | Dataset (coming soon)

Overview

Foundation-model-powered mobile agents can execute complex workflows on real devices, but their trajectories may violate app-specific, organization-specific, or user-defined security policies. Existing defenses typically use brittle static rules or prompt a general-purpose LLM as a judge.

MATE is a lightweight, policy-aware trajectory auditor. Given an instruction, an agent trajectory, and natural-language security policies, MATE outputs:

• a policy-conditioned violation decision;
• one of 14 mobile-agent risk categories; and
• a natural-language rationale grounded in the relevant trajectory steps.

Policies remain editable text rather than fixed model parameters, so security requirements can be updated without retraining the auditor.

The MATE pipeline has four stages:

1. App information collection: collect functional descriptions, operation workflows, and security policies from 158 Chinese and English mobile apps.
2. Knowledge-grounded synthesis: generate instructions, ReAct-style trajectories, policy-conditioned labels, risk categories, and explanations.
3. Data augmentation and training: introduce trajectory-policy mismatch, multi-policy supervision, and multi-app workflows, then fine-tune 0.5B, 1.5B, and 3B auditors.
4. Deployment: normalize heterogeneous raw logs with a trajectory adapter and automatically retrieve relevant policies for auditing.

The final training corpus contains more than 140K semantically realistic, policy-conditioned trajectories.

MATEBench

We introduce MATEBench, a bilingual Chinese-English benchmark for policy-aware, trajectory-level mobile-agent security auditing. It covers all 14 risk categories and contains three subsets:

Subset	Distribution	Apps	Trajectories	Evaluation goal
MATEBench-In	In-domain, synthetic	134 seen apps	2,775	Controlled evaluation on apps represented during training
MATEBench-Out	Out-of-domain, synthetic	24 unseen apps	1,150	Generalization to new app semantics and workflows
MATEBench-Real	Out-of-domain, real device	13 apps / 3 agents	162	Auditing real trajectories with realistic logging noise

The real subset includes trajectories from Zhipu's AutoGLM, Alibaba's Mobile-Agent, and an Android-emulator-based agent. Synthetic and real subsets have closely aligned trajectory lengths and GUI-action distributions.

Main Results

MATE consistently outperforms a static rule engine, general-purpose LLM judges, ShieldAgent, and ShieldLM on MATEBench, R-Judge, and ASSEBench.

MATE-3B achieves:

Benchmark	Accuracy	F1
R-Judge	92.64%	93.92%
ASSEBench	94.98%	95.54%
MATEBench-In	96.83%	95.53%
MATEBench-Out	95.48%	94.68%
MATEBench-Real	95.06%	95.60%

Even MATE-0.5B surpasses strong prompt-based evaluators on MATEBench-Out while requiring only 0.09 seconds per trajectory on a single NVIDIA H100.

Ablation Studies

Zero-shot Qwen2.5-Instruct models achieve less than 50% accuracy on average for policy-conditioned trajectory auditing. Fine-tuning on the synthesized MATE corpus raises overall accuracy above 92%, demonstrating the importance of task-specific trajectory learning.

External app knowledge and multi-stage quality repair are both necessary. In the controlled MATE-0.5B ablation on MATEBench-Real, the full synthesis pipeline reaches 75.93% accuracy and 79.14% F1, while removing both components reduces accuracy to 46.91%.

Real-World Auditing

MATE-3B successfully audits trajectories from deployed mobile agents and produces explicit policy-grounded explanations. The following cases identify a mass-forwarded malicious coupon link as Becoming a Fraud Relay and access to a malware-test page as Device Security Compromised.

---

Installation and Usage

This repository contains the MATE implementation for benchmark evaluation, data synthesis and augmentation, trajectory normalization, and policy retrieval. The large benchmark files and model checkpoints must be downloaded separately and placed in the paths described below.

1. Environment Setup

The implementation was developed with Python 3.11. A CUDA-capable Linux environment is recommended when serving MATE checkpoints locally with vLLM.

conda create -n mate python=3.11 -y
conda activate mate

python -m pip install --upgrade pip
pip install -r requirements.txt

The main dependencies include:

• vllm and openai for OpenAI-compatible model serving and requests;
• torch and sentence-transformers for policy retrieval;
• langchain-huggingface or langchain-community for embedding utilities;
• json_repair for synthesis and augmentation output parsing; and
• scikit-learn and matplotlib for rationale-similarity visualization.

If MATE is accessed through an existing remote OpenAI-compatible endpoint, a local vLLM server is not required. In that case, install the remaining dependencies in an environment supported by your platform.

2. Prepare Models and Data

Before running the complete examples, place the downloaded artifacts at the following locations:

Mate_model/
├── MATE-0.5B/
├── MATE-1.5B/
└── MATE-3B/

MateBench/
├── MateBench-In/
├── MateBench-Out/
└── MateBench-Real/

policy_retriever/model/policy_retriever/

Each MATE checkpoint should follow the Hugging Face Transformers layout and contain its model configuration, tokenizer, generation configuration, and safetensors weights. The policy-retriever directory must contain a local SentenceTransformers-compatible model.

The evaluator uses BAAI/bge-large-zh-v1.5 to calculate rationale similarity. In an offline environment, pre-cache this model or change the embedding path in evaluation/matebench_eval.py to a local copy.

You can quickly verify whether the required directories are populated:

find Mate_model -maxdepth 2 -type f | head
find MateBench -type f -name '*.json' | head
find policy_retriever/model/policy_retriever -maxdepth 2 -type f | head

3. MATEBench Data Format

Each benchmark example is a JSON object containing an instruction, a ReAct-style trajectory, one or more security policies, and a policy-conditioned reference result:

{
  "instruction": "A user instruction",
  "trajectory": [
    {
      "Thought": "Agent reasoning",
      "Action": "click",
      "Action Input": {"x": 100, "y": 200},
      "Observation": "The next UI state"
    },
    {
      "Thought": "The task is complete",
      "Final Answer": "Completion message"
    }
  ],
  "safe_rule": "A natural-language security policy",
  "evaluation_result": {
    "Violation": "yes",
    "Category": "Privacy Leakage",
    "Rationale": "Why the trajectory violates the policy"
  }
}

4. Quick Start

Validate subset construction without a model

This command checks the benchmark paths and creates a two-example temporary subset without starting a model server:

CREATE_ONLY=1 \
SUBSET_SIZE=2 \
MATE_TASK=in \
MATE_LANG=en \
bash run_eval_subset.sh

The temporary subset is written under /tmp/matebench_subset_*.

Run the minimal end-to-end demo

The smallest complete demo starts MATE-1.5B with vLLM, creates a small English MATEBench-In subset, evaluates it, and prints a Markdown metrics summary:

bash run_minimal_demo.sh

Default configuration:

MODEL_PATH=Mate_model/MATE-1.5B
SERVED_MODEL=MATE-1.5B
PORT=8016
MATE_TASK=in
MATE_LANG=en
SUBSET_SIZE=50
BATCH_SIZE=8

Override settings with environment variables:

MODEL_PATH=Mate_model/MATE-0.5B \
SERVED_MODEL=MATE-0.5B \
SUBSET_SIZE=10 \
BATCH_SIZE=4 \
PORT=8016 \
bash run_minimal_demo.sh

The server started by the script is stopped when evaluation finishes. Set KEEP_SERVER=1 to keep it running.

Use an existing OpenAI-compatible endpoint

If a compatible server is already available at http://127.0.0.1:8016/v1, skip server startup:

START_SERVER=0 \
MODEL_NAME=MATE-1.5B \
BASE_URL=http://127.0.0.1:8016/v1 \
API_KEY=EMPTY \
MATE_TASK=in \
MATE_LANG=en \
SUBSET_SIZE=50 \
BATCH_SIZE=8 \
bash run_eval_subset.sh

5. Serve a MATE Checkpoint

Start MATE-3B on port 8016:

bash evaluation/launch_mate_vllm.sh \
  Mate_model/MATE-3B \
  MATE-3B \
  8016

The launcher is equivalent to:

python -m vllm.entrypoints.openai.api_server \
  --model Mate_model/MATE-3B \
  --served-model-name MATE-3B \
  --max-model-len 8192 \
  --tensor-parallel-size 1 \
  --gpu-memory-utilization 0.8 \
  --port 8016

For multiple GPUs or a smaller context window:

TP=2 \
GPU_UTIL=0.9 \
MAX_LEN=8192 \
bash evaluation/launch_mate_vllm.sh Mate_model/MATE-3B MATE-3B 8016

Confirm that the endpoint is ready:

curl http://127.0.0.1:8016/v1/models

6. Run Full MATEBench Evaluation

Configure the evaluator through environment variables:

export MATE_MODEL_NAME=MATE-3B
export MATE_BASE_URL=http://127.0.0.1:8016/v1
export MATE_API_KEY=EMPTY

Evaluate the in-domain, out-of-domain, and real-world subsets:

python evaluation/run_matebench.py \
  --task in \
  --lang all \
  --batch-size 64 \
  --model MATE-3B

python evaluation/run_matebench.py \
  --task out \
  --lang all \
  --batch-size 64 \
  --model MATE-3B

python evaluation/run_matebench.py \
  --task real \
  --lang all \
  --batch-size 64 \
  --model MATE-3B

Supported arguments:

Argument	Values	Description
--task	in, out, real	MATEBench subset
--lang	all, cn, en	Language split
--batch-size	positive integer	Number of concurrent requests
--model	served model name	Overrides MATE_MODEL_NAME
--data-root	directory	Benchmark root; default: ./MateBench
--output-root	directory	Result root; default: ./evaluation/evaluation_results
--no-visualize	flag	Skip PCA rationale visualizations

Example with custom paths:

python evaluation/run_matebench.py \
  --task in \
  --lang en \
  --batch-size 16 \
  --model MATE-3B \
  --data-root ./MateBench \
  --output-root ./evaluation/evaluation_results \
  --no-visualize

Results are stored as:

evaluation/evaluation_results/
└── matebench_{in,out,real}_results/
    └── <model-name>/
        └── <YYYYMMDD_HHMMSS>_<lang>/
            ├── eval_results.txt
            ├── eval_model_output.json
            ├── rationale_semantic.pdf
            └── rationale_semantic.png

Summarize all recorded runs or only the latest run:

python summarize_results.py --root evaluation/evaluation_results
python summarize_results.py --root evaluation/evaluation_results --latest

7. Data Synthesis

The synthesis modules call an OpenAI-compatible endpoint. Configure the llm_config.py inside the module that you are running:

MODELS_CONFIG = {
    "llm": {
        "model_name": "your-model-name",
        "base_url": "http://your-api-endpoint/v1",
        "api_key": "your-api-key",
    }
}

Generate task instructions

cd data_synthesis_pipeline/traj_synthesis

python main.py \
  --task instruction \
  --type en \
  --num 200 \
  --batch_size 50 \
  --pairs_per_call 20 \
  --output_path ./synthesis-results

Generate trajectories

cd data_synthesis_pipeline/traj_synthesis

python main.py \
  --task trajectory \
  --type en \
  --num 110 \
  --traj_batch_size 100 \
  --output_path ./synthesis-results

Use --type cn for Chinese generation. The provided instructions_synthesis.sh and trajectory_synthesis.sh scripts contain compact versions of these commands.

8. Data Augmentation

All augmentation modules are under data_synthesis_pipeline/data_augmentation/.

Multi-app trajectories

cd data_synthesis_pipeline/data_augmentation/multi-app

python main_multiapp.py \
  --task instruction \
  --type en \
  --num 110 \
  --batch_size 50 \
  --output_path ./synthesis-multiapp

python main_multiapp.py \
  --task trajectory \
  --type en \
  --num 20 \
  --traj_batch_size 10 \
  --output_path ./synthesis-multiapp

Multi-policy supervision

cd data_synthesis_pipeline/data_augmentation/multi-policy

python multipolicy_data.py \
  --input_path ../../traj_synthesis/synthesis-results/results-en/trajectory \
  --num 100 \
  --batch_size 100 \
  --similarity_threshold 0.5 \
  --include_original yes \
  --output_path ./multipolicy_output-en-yes \
  --embedding_model_path ../../../policy_retriever/model/policy_retriever

Set --include_original yes to retain the matched policy and append additional policies. Set it to no to construct a policy set that excludes the original matched policy.

Trajectory-policy mismatch

cd data_synthesis_pipeline/data_augmentation/traj_policy_mismatched

python main_batch.py \
  --input_path ../../traj_synthesis/synthesis-results/results-en/trajectory \
  --num 100 \
  --batch_size 100 \
  --similarity_threshold 0.5 \
  --output_path ./output-en \
  --embedding_model_path ../../../policy_retriever/model/policy_retriever

The mismatch pipeline pairs trajectories with semantically unrelated policies to create hard negative examples and reduce false-positive auditing decisions.

9. Trajectory Adapter and Policy Retriever

model_deployment/get_traj_for_mate.py converts raw agent logs into MATE's canonical trajectory schema and retrieves the most relevant security policy.

For text trajectories:

mkdir -p model_deployment/input_test/text model_deployment/output_test/text
# Place one JSON object per file under model_deployment/input_test/text.

python model_deployment/get_traj_for_mate.py \
  --input_path model_deployment/input_test/text \
  --lang en \
  --input_type text \
  --output_dir model_deployment/output_test/text \
  --adapter_batch_size 10 \
  --adapter_base_url http://127.0.0.1:8016/v1 \
  --adapter_model_name your-model-name \
  --adapter_api_key EMPTY \
  --retriever_model_path policy_retriever/model/policy_retriever \
  --retriever_batch_size 128 \
  --rule_pool_en policy_retriever/policy_pool/policy_en.jsonl \
  --rule_pool_cn policy_retriever/policy_pool/policy_cn.jsonl

For screenshot-based or multimodal trajectories:

mkdir -p model_deployment/input_test/mm model_deployment/output_test/mm
# Place the multimodal JSON inputs and referenced images under the input directory.

python model_deployment/get_traj_for_mate.py \
  --input_path model_deployment/input_test/mm \
  --lang en \
  --input_type multimodal \
  --output_dir model_deployment/output_test/mm \
  --mm_base_url http://127.0.0.1:8016/v1 \
  --mm_model_name your-vlm-name \
  --mm_api_key EMPTY \
  --retriever_model_path policy_retriever/model/policy_retriever \
  --retriever_batch_size 128 \
  --rule_pool_en policy_retriever/policy_pool/policy_en.jsonl \
  --rule_pool_cn policy_retriever/policy_pool/policy_cn.jsonl \
  --log_model_io \
  --log_dir model_deployment/logs

The input path must be a directory containing JSON files. Inputs already matching the MATE schema skip trajectory conversion but still pass through policy retrieval.

10. Troubleshooting

The benchmark smoke test reports missing directories

Populate MateBench/MateBench-In, MateBench/MateBench-Out, and MateBench/MateBench-Real with the released JSON files before running run_eval_subset.sh.

vLLM runs out of GPU memory

Start with MATE-0.5B and reduce context length or memory utilization:

GPU_UTIL=0.7 \
MAX_LEN=4096 \
bash evaluation/launch_mate_vllm.sh Mate_model/MATE-0.5B MATE-0.5B 8016

Evaluation is too slow

Reduce request concurrency:

python evaluation/run_matebench.py \
  --task in \
  --lang en \
  --batch-size 8 \
  --model MATE-1.5B

The evaluator cannot connect to the model

curl http://127.0.0.1:8016/v1/models

Use 127.0.0.1 for a client on the same machine. Although vLLM may bind to 0.0.0.0, clients should not use 0.0.0.0 as the destination address.

The retriever cannot find its model

Confirm that policy_retriever/model/policy_retriever/ contains a valid SentenceTransformers model and that both policy pools exist:

ls policy_retriever/model/policy_retriever
ls policy_retriever/policy_pool/policy_{cn,en}.jsonl

Offline rationale evaluation fails

Pre-download BAAI/bge-large-zh-v1.5 and configure evaluation/matebench_eval.py to use its local path.

Citation

@misc{jiang2026mate,
  title={MATE: Policy-Aware Security Auditing for Mobile Agents via Synthesis-Driven Trajectory Learning},
  author={Jiang, Changyue and Wang, Jiayi and Wen, Xin and Dai, Jiarun and Hong, Geng and Pan, Xudong},
  year={2026}
}