A New Framework for Cybersecurity Refusals in AI Agents
Eliot Krzysztof Jones, Mateusz Dziemian, Matt Fredrikson, J Zico Kolter
Matt Fredrikson
- CEO · Gray Swan AI
- Associate Professor · Carnegie Mellon University
Highlights
Publications
116 papers
2026
A New Framework for Cybersecurity Refusals in AI Agents
Eliot Krzysztof Jones, Mateusz Dziemian, Matt Fredrikson, J Zico Kolter
Multi-Rollout On-Policy Distillation via Peer Successes and Failures
Weichen Yu, Xiaomin Li, Yizhou Zhao, Xiaoze Liu, Ruowang Zhang, Haixin Wang, Yinyi Luo, Chen Henry Wu,
How Vulnerable Are AI Agents to Indirect Prompt Injections? Insights from a Large-Scale Public Competition
Mateusz Dziemian, Maxwell Lin, Xiaohan Fu, Micha Nowak, Nick Winter, Eliot Jones, Andy Zou, Lama Ahmad,
The Vision Wormhole: Latent-Space Communication in Heterogeneous Multi-Agent Systems
Xiaoze Liu, Ruowang Zhang, Weichen Yu, Siheng Xiong, Liu He, Feijie Wu, Hoin Jung, Matt Fredrikson,
SecCodePRM: A Process Reward Model for Code Security
Weichen Yu, Ravi Mangal, Yinyi Luo, Kai Hu, Jingxuan He, Corina S. Pasareanu, Matt Fredrikson
Sequential Pattern Recognition Attacks against Deployed Topic-Based Mechanisms
Saranya Vijayakumar, Norman Sadeh, Matt Fredrikson
PrivCode: When Code Generation Meets Differential Privacy
Zheng Liu, Chen Gong, Terry Yue Zhuo, Kecen Li, Weichen Yu, Matt Fredrikson, Tianhao Wang
LipNeXt: Scaling up Lipschitz-based Certified Robustness to Billion-parameter Models
Kai Hu, Haoqi Hu, Matt Fredrikson
The Trojan in the Vocabulary: Stealthy Sabotage of LLM Composition
Xiaoze Liu, Weichen Yu, Matt Fredrikson, Xiaoqian Wang, Jing Gao
Jailbreak-Zero: A Path to Pareto Optimal Red Teaming for Large Language Models
Kai Hu, Abhinav Aggarwal, Mehran Khodabandeh, David Zhang, Eric Hsin, Li Chen, Ankit Jain, Matt Fredrikson,
2025
When the Same Coefficients Reach Different Places: Asymmetric Realizability in Transplanting Tokenizers across Large Language Models
Xiaoze Liu, Weichen Yu, Matt Fredrikson, Xiaoqian Wang, Jing Gao
Comparing AI Agents to Cybersecurity Professionals in Real-World Penetration Testing
Justin W. Lin, Eliot Krzysztof Jones, Donovan Julian Jasper, Ethan Jun-shen Ho, Anna Wu, Arnold Tianyi Yang, Neil Perry, Andy Zou,
Prototype-Integrated Representation Learning for Novelty Detection
Saranya Vijayakumar, Christos Faloutsos, Matt Fredrikson
D-REX: A Benchmark for Detecting Deceptive Reasoning in Large Language Models
Satyapriya Krishna, Andy Zou, Rahul Gupta, Eliot Krzysztof Jones, Nick Winter, Dan Hendrycks, J. Zico Kolter, Matt Fredrikson,
Evaluating Language Model Reasoning about Confidential Information
Dylan Sam, Alexander Robey, Andy Zou, Matt Fredrikson, J. Zico Kolter
Security Challenges in AI Agent Deployment: Insights from a Large Scale Public Competition
Andy Zou, Maxwell Lin, Eliot Jones, Micha Nowak, Mateusz Dziemian, Nick Winter, Alexander Grattan, Valent Nathanael,
A Mixture of Linear Corrections Generates Secure Code
Weichen Yu, Ravi Mangal, Terry Zhuo, Matt Fredrikson, Corina S. Pasareanu
Adversarial Attacks on Robotic Vision Language Action Models
Eliot Krzysztof Jones, Alexander Robey, Andy Zou, Zachary Ravichandran, George J. Pappas, Hamed Hassani, Matt Fredrikson, J. Zico Kolter
Transferable Adversarial Attacks on Black-Box Vision-Language Models
Kai Hu, Weichen Yu, Li Zhang, Alexander Robey, Andy Zou, Chengming Xu, Haoqi Hu, Matt Fredrikson
Aligned LLMs Are Not Aligned Browser Agents
Priyanshu Kumar, Elaine Lau, Saranya Vijayakumar, Tu Trinh, Elaine T. Chang, Vaughn Robinson, Shuyan Zhou, Matt Fredrikson,
AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents
Maksym Andriushchenko, Alexandra Souly, Mateusz Dziemian, Derek Duenas, Maxwell Lin, Justin Wang, Dan Hendrycks, Andy Zou,
LLM Whisperer: An Inconspicuous Attack to Bias LLM Responses
Weiran Lin, Anna Gerchanovsky, Omer Akgul, Lujo Bauer, Matt Fredrikson, Zifan Wang
Safety Pretraining: Toward the Next Generation of Safe AI
Pratyush Maini, Sachin Goyal, Dylan Sam, Alex Robey, Yash Savani, Yiding Jiang, Andy Zou, Matt Fredrikson,
2024
Is Your Text-to-Image Model Robust to Caption Noise?
Weichen Yu, Ziyan Yang, Shanchuan Lin, Qi Zhao, Jianyi Wang, Liangke Gui, Matt Fredrikson, Lu Jiang
Attacks and Defenses for Large Language Models on Coding Tasks
Chi Zhang, Zifan Wang, Ruoshi Zhao, Ravi Mangal, Matt Fredrikson, Limin Jia, Corina S. Pasareanu
Refusal-Trained LLMs Are Easily Jailbroken As Browser Agents
Priyanshu Kumar, Elaine Lau, Saranya Vijayakumar, Tu Trinh, Scale Red Team, Elaine Chang, Vaughn Robinson, Sean Hendryx,
Improving Alignment and Robustness with Circuit Breakers
Andy Zou, Long Phan, Justin Wang, Derek Duenas, Maxwell Lin, Maksym Andriushchenko, Rowan Wang, Zico Kolter,
VeriSplit: Secure and Practical Offloading of Machine Learning Inferences across IoT Devices
Han Zhang, Zifan Wang, Mihir Dhamankar, Matt Fredrikson, Yuvraj Agarwal
Sales Whisperer: A Human-Inconspicuous Attack on LLM Brand Recommendations
Weiran Lin, Anna Gerchanovsky, Omer Akgul, Lujo Bauer, Matt Fredrikson, Zifan Wang
Efficient LLM Jailbreak via Adaptive Dense-to-sparse Constrained Optimization
Kai Hu, Weichen Yu, Yining Li, Kai Chen, Tianjun Yao, Xiang Li, Wenhe Liu, Lijun Yu,
A Recipe for Improved Certifiable Robustness
Kai Hu, Klas Leino, Zifan Wang, Matt Fredrikson
2023
Grounding Neural Inference with Satisfiability Modulo Theories
Zifan Wang, Saranya Vijayakumar, Kaiji Lu, Vijay Ganesh, Somesh Jha, Matt Fredrikson
Transfer Attacks and Defenses for Large Language Models on Coding Tasks
Chi Zhang, Zifan Wang, Ravi Mangal, Matt Fredrikson, Limin Jia, Corina Pasareanu
Is Certifying Robustness Still Worthwhile?
Ravi Mangal, Klas Leino, Zifan Wang, Kai Hu, Weicheng Yu, Corina Pasareanu, Anupam Datta, Matt Fredrikson
Representation Engineering: A Top-Down Approach to AI Transparency
Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin,
Universal and Transferable Adversarial Attacks on Aligned Language Models
Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, Matt Fredrikson
On the Perils of Cascading Robust Classifiers
Ravi Mangal, Zifan Wang, Chi Zhang, Klas Leino, Corina Pasareanu, Matt Fredrikson
Unlocking Deterministic Robustness Certification on ImageNet
Kai Hu, Andy Zou, Zifan Wang, Klas Leino, Matt Fredrikson
Learning Modulo Theories
Matt Fredrikson, Kaiji Lu, Saranya Vijayakumar, Somesh Jha, Vijay Ganesh, Zifan Wang
2022
Black-Box Audits for Group Distribution Shifts
Marc Juarez, Samuel Yeom, Matt Fredrikson
Degradation Attacks on Certifiably Robust Neural Networks
Klas Leino, Chi Zhang, Ravi Mangal, Matt Fredrikson, Bryan Parno, Corina S. Pasareanu
Enhancing the Insertion of NOP Instructions to Obfuscate Malware via Deep Reinforcement Learning
Daniel Gibert, Matt Fredrikson, Carles Mateu, Jordi Planes, Quan Le
Self-Correcting Neural Networks For Safe Classification
Klas Leino, Aymeric Fromherz, Ravi Mangal, Matt Fredrikson, Bryan Parno, Corina Păsăreanu
Robust Models Are More Interpretable Because Attributions Look Normal
Zifan Wang, Matt Fredrikson, Anupam Datta
TEO: ephemeral ownership for IoT devices to provide granular data control
Han Zhang, Yuvraj Agarwal, Matt Fredrikson
MobiSys '22: Proceedings of the 20th Annual International Conference on Mobile Systems, Applications and Services·Jun· pdfProtecting user data through ephemeral ownership of IoT devices
Han Zhang, Yuvraj Agarwal, Matt Fredrikson
Faithful Explanations for Deep Graph Models
Zifan Wang, Yuhang Yao, Chaoran Zhang, Han Zhang, Youjie Kang, Carlee Joe-Wong, Matt Fredrikson, Anupam Datta
Selective Ensembles for Consistent Predictions
Emily Black, Klas Leino, Matt Fredrikson
Consistent Counterfactuals for Deep Models
Emily Black, Zifan Wang, Matt Fredrikson, Anupam Datta
Privacy-Preserving Case-Based Explanations: Enabling Visual Interpretability by Protecting Privacy
Helena Montenegro, Wilson Silva, Alex Gaudio, Matt Fredrikson, Asim Smailagic, Jaime S. Cardoso
2021
Exploring Conceptual Soundness with TruLens
Anupam Datta, Matt Fredrikson, Klas Leino, Kaiji Lu, Shayak Sen, Ricardo Shih, Zifan Wang
Machine Learning Explainability and Robustness: Connected at the Hip
Anupam Datta, Matt Fredrikson, Klas Leino, Kaiji Lu, Shayak Sen, Zifan Wang
Self-Repairing Neural Networks: Provable Safety for Deep Networks via Dynamic Repair
Klas Leino, Aymeric Fromherz, Ravi Mangal, Matt Fredrikson, Bryan Parno, Corina S. Pasareanu
Relaxing Local Robustness
Klas Leino, Matt Fredrikson
Automating Audit with Policy Inference
Abhishek Bichhawat, Matt Fredrikson, Jean Yang
Fast Geometric Projections for Local Robustness Certification
Aymeric Fromherz, Klas Leino, Matt Fredrikson, Bryan Parno, Corina Păsăreanu
The Design of the User Interfaces for Privacy Enhancements for Android
Jason I. Hong, Yuvraj Agarwal, Matt Fredrikson, Mike Czapik, Shawn Hanna, Swarup Sahoo, Judy Chun, Won-Woo Chung,
Boundary Attributions Provide Normal (Vector) Explanations
Zifan Wang, Matt Fredrikson, Anupam Datta
Leave-one-out Unfairness
Emily Black, Matt Fredrikson
Globally-Robust Neural Networks
Klas Leino, Zifan Wang, Matt Fredrikson
Capture: Centralized Library Management for Heterogeneous IoT Devices
Han Zhang, Abhijith Anilkumar, Matt Fredrikson, Yuvraj Agarwal
Netter: Probabilistic, Stateful Network Models
Han Zhang, Chi Zhang, Arthur Azevedo de Amorim, Yuvraj Agarwal, Matt Fredrikson, Limin Jia
2020
Contextual and Granular Policy Enforcement in Database-backed Applications
Abhishek Bichhawat, Matt Fredrikson, Jean Yang, Akash Trehan
Stolen Memories: Leveraging Model Memorization for Calibrated White-Box Membership Inference
Klas Leino, Matt Fredrikson
Reconciling noninterference and gradual typing
Arthur Azevedo de Amorim, Matt Fredrikson, Limin Jia
Overfitting, robustness, and malicious algorithms: A study of potential causes of privacy risk in machine learning
Samuel Yeom, Irene Giacomelli, Alan Menaged, Matt Fredrikson, Somesh Jha
Smoothed Geometry for Robust Attribution
Zifan Wang, Haofan Wang, Shakul Ramkumar, Matt Fredrikson, Piotr Mardziel, Anupam Datta
FlipTest: Fairness Testing via Optimal Transport
Emily Black, Samuel Yeom, Matt Fredrikson
Influence Paths for Characterizing Subject-Verb Number Agreement in LSTM Language Models
Kaiji Lu, Piotr Mardziel, Klas Leino, Matt Fredrikson, Anupam Datta
Learning Fair Representations for Kernel Models
Zilong Tan, Samuel Yeom, Matt Fredrikson, Ameet Talwalkar
Interpreting Interpretations: Organizing Attribution Methods by Criteria
Zifan Wang, Piotr Mardziel, Anupam Datta, Matt Fredrikson
Individual Fairness Revisited: Transferring Techniques from Adversarial Robustness
Samuel Yeom, Matt Fredrikson
2019
FlipTest: Fairness Auditing via Optimal Transport.
Emily Black, Samuel Yeom, Matt Fredrikson
Jun· pdfFeature-Wise Bias Amplification
Klas Leino, Emily Black, Matt Fredrikson, Shayak Sen, Anupam Datta
2018
Why Are They Collecting My Data?: Inferring the Purposes of Network Traffic in Mobile Apps
Haojian Jin, Minyi Liu, Kevan Dodhia, Yuanchun Li, Gaurav Srivastava, Matthew Fredrikson, Yuvraj Agarwal, Jason I. Hong
ESTRELA: Automated Policy Enforcement Across Remote APIs.
Abhishek Bichhawat, Akash Trehan, Jean Yang, Matt Fredrikson
Nov· pdfEstrela: Automated, Contextual Enforcement of Data Flow Policies across APIs
Abhishek Bichhawat, Akash Trehan, Jean Yang, Matt Fredrikson
Nov· pdfHunting for Discriminatory Proxies in Linear Regression Models
Samuel Yeom, Anupam Datta, Matt Fredrikson
Privacy Risk in Machine Learning: Analyzing the Connection to Overfitting
Samuel Yeom, Irene Giacomelli, Matt Fredrikson, Somesh Jha
Quantitative underpinnings of secure, graceful degradation: poster
Ryan Wagner, David Garlan, Matt Fredrikson
Supervising Feature Influence
Shayak Sen, Piotr Mardziel, Anupam Datta, Matthew Fredrikson
Influence-Directed Explanations for Deep Convolutional Networks
Klas Leino, Shayak Sen, Anupam Datta, Matt Fredrikson, Linyi Li
Verifying and Synthesizing Constant-Resource Implementations with Types
Van Chan Ngo, Mario Dehesa-Azuara, Matthew Fredrikson, Jan Hoffmann
2017
Case Study: Explaining Diabetic Retinopathy Detection Deep CNNs via Integrated Gradients
Linyi Li, Matt Fredrikson, Shayak Sen, Anupam Datta
PrivacyStreams
Yuanchun Li, Fanglin Chen, Toby Jia-Jun Li, Yao Guo, Gang Huang, Matt Fredrikson, Yuvraj Agarwal, Jason Hong
The Unintended Consequences of Overfitting: Training Data Inference Attacks.
Samuel Yeom, Matt Fredrikson, Somesh Jha
Sep· pdfPrivacyProxy: Leveraging Crowdsourcing and In Situ Traffic Analysis to Detect and Mitigate Information Leakage
Gaurav Srivastava, Kunal Bhuwalka, Swarup Kumar Sahoo, Saksham Chitkara, Kevin Ku, Matt Fredrikson, Jason Hong, Yuvraj Agarwal
Proxy Non-Discrimination in Data-Driven Systems
Anupam Datta, Matt Fredrikson, Gihyuk Ko, Piotr Mardziel, Shayak Sen
An Advanced Persistent Threat Exemplar
R. Harrison Wagner, Matt Fredrikson, David Garlan
Jul· pdfUse Privacy in Data-Driven Systems
Anupam Datta, Matt Fredrikson, Gihyuk Ko, Piotr Mardziel, Shayak Sen
Use Privacy in Data-Driven Systems: Theory and Experiments with Machine Learnt Programs
Anupam Datta, Matthew Fredrikson, Gihyuk Ko, Piotr Mardziel, Shayak Sen
2016
A Methodology for Formalizing Model-Inversion Attacks
Xi Wu, Matt Fredrikson, Somesh Jha, Jeffrey F. Naughton
2015
Revisiting Differentially Private Regression: Lessons From Learning Theory and their Consequences
Xi Wu, Matthew Fredrikson, Wentao Wu, Somesh Jha, Jeffrey F. Naughton
The Limitations of Deep Learning in Adversarial Settings
Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z. Berkay Celik, Ananthram Swami
Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures
Matt Fredrikson, Somesh Jha, Thomas Ristenpart
Surreptitiously Weakening Cryptographic Systems.
Bruce Schneier, Matt Fredrikson, Tadayoshi Kohno, Thomas Ristenpart
2014
MoRePriv
Drew Davidson, Matt Fredrikson, Benjamin Livshits
On the practical exploitability of dual EC in TLS implementations
Stephen Checkoway, Matt Fredrikson, Ruben Niederhagen, Adam Everspaugh, Matthew Green, Tanja Lange, Thomas Ristenpart, Daniel J. Bernstein,
ZØ: an optimizing distributing zero-knowledge compiler
Matt Fredrikson, Benjamin Livshits
Privacy in Pharmacogenetics: An End-to-End Case Study of Personalized Warfarin Dosing.
Matt Fredrikson, Éric Lantz, Somesh Jha, Simon Lin, David Page, Thomas Ristenpart
Satisfiability modulo counting
Matt Fredrikson, Somesh Jha
The Future of Crypto: Getting from Here to Guarantees
Daniel J. Bernstein, Matt Blaze, Tanja Lange, Tanzirul Azim, Shuo Chen, Jaeyeon Jung, Suman Nath, Rui Wang,
Jan·
2012
Efficient Runtime Policy Enforcement Using Counterexample-Guided Abstraction Refinement
Matt Fredrikson, Richard Joiner, Somesh Jha, Thomas Reps, Phillip Porras, Hassen Saı̈di, Vinod Yegneswaran
2011
Dynamic Behavior Matching: A Complexity Analysis and New Approximation Algorithms
Matt Fredrikson, Mihai Christodorescu, Somesh Jha
RePriv: Re-Envisioning In-Browser Privacy
Matt Fredrikson, Ben Livshits
May· pdfVerified Security for Browser Extensions
Arjun Guha, Matt Fredrikson, Benjamin Livshits, Nikhil Swamy
RePriv: Re-imagining Content Personalization and In-browser Privacy
Matt Fredrikson, Benjamin Livshits
End-to-End Software Diversification of Internet Services
Mihai Christodorescu, Matt Fredrikson, Somesh Jha, Jonathon Giffin
2010
Private Information Retrieval
Somesh Jha, Vitaly Shmatikov, Matt Fredrikson
Dec· pdfAutomatic generation of remediation procedures for malware infections
Roberto Paleari, Lorenzo Martignoni, Emanuele Passerini, Drew Davidson, Matt Fredrikson, Jon Giffin, Somesh Jha
Synthesizing Near-Optimal Malware Specifications from Suspicious Behaviors
Matt Fredrikson, Somesh Jha, Mihai Christodorescu, Reiner Sailer, Xifeng Yan
Mining Large Information Networks by Graph Summarization
Chen Chen, Cindy Xide Lin, Matt Fredrikson, Mihai Christodorescu, Xifeng Yan, Jiawei Han
2009
A Declarative Framework for Intrusion Analysis
Matt Fredrikson, Mihai Christodorescu, Jonathon Giffin, Somesh Jhas
Mining graph patterns efficiently via randomized summaries
Chen Chen, Cindy Xide Lin, Matt Fredrikson, Mihai Christodorescu, Xifeng Yan, Jiawei Han
Cyber SA: Situational Awareness for Cyber Defense
Paul Barford, Marc Daciér, Thomas G. Dietterich, Matt Fredrikson, Jon Giffin, Sushil Jajodia, Somesh Jha, Jason Li,
2008
A Layered Architecture for Detecting Malicious Behaviors
Lorenzo Martignoni, Elizabeth Stinson, Matt Fredrikson, Somesh Jha, John C. Mitchell
Students & postdocs
Current
- Kai Hu
- Saranya Vijaykumar
- Weichen Yu
Former
- Emily Black
- Klas Leino
- Zifan Wang
- Ryan Wagner
- Andy Zou
- Samuel Yeom
- Han Zhang
- Arthur Azevedo de Amorim· postdoc
- Abhishek Bichhawat· postdoc