Chengrun Yang

Research Scientist, Google DeepMind New York, USA Contact: chengrunyang at gmail dot com | | |

About Me

I am a research scientist at Google DeepMind, working on model self-improvement and agentic post-training. I believe that powerful models can autonomously improve and evaluate themselves, with or without human supervision, and this self-driven approach is the most promising path toward building even stronger models. Some of my previous work towards this direction include:

• LLMs as optimizers [1]: LLMs can recursively improve prompts and math solutions
• LLM agents as AI researchers: search for better ideas and code implementations with LLM agents, found better RL recipe for Gemini 3 Deep Think
• model self-evaluation [2]: using LLM agents to measure model factuality
• model selection and architecture search [3, 4]

I obtained my Ph.D. in the School of Electrical and Computer Engineering at Cornell University in May 2022, where my Ph.D. thesis focused on resource-constrained automated machine learning (AutoML). I was advised by Prof. Madeleine Udell and had Prof. Thorsten Joachims and Prof. Kilian Q. Weinberger on my committee. From Summer 2021 to Spring 2022, I was a student researcher at Google Brain. I received a B.S. degree in physics from Fudan University in 2016.

Selected Papers

Show All Abstracts — Hide All Abstracts

• Long-Form Factuality in Large Language Models
  Jerry Wei*, Chengrun Yang*, Xinying Song*, Yifeng Lu*, Nathan Hu, Jie Huang, Dustin Tran, Daiyi Peng, Ruibo Liu, Da Huang, Cosmo Du, Quoc V. Le
  Neural Information Processing Systems (NeurIPS), 2024
  [abstract] [arXiv] [code] [bib]
  

  Large language models (LLMs) often generate content that contains factual errors when responding to fact-seeking prompts on open-ended topics. To benchmark a model's long-form factuality in open domains, we first use GPT-4 to generate LongFact, a prompt set comprising thousands of questions spanning 38 topics. We then propose that LLM agents can be used as automated evaluators for long-form factuality through a method which we call Search-Augmented Factuality Evaluator (SAFE). SAFE utilizes an LLM to break down a long-form response into a set of individual facts and to evaluate the accuracy of each fact using a multi-step reasoning process comprising sending search queries to Google Search and determining whether a fact is supported by the search results. Furthermore, we propose extending F1 score as an aggregated metric for long-form factuality. To do so, we balance the percentage of supported facts in a response (precision) with the percentage of provided facts relative to a hyperparameter representing a user's preferred response length (recall).
  Empirically, we demonstrate that LLM agents can achieve superhuman rating performance - on a set of ~16k individual facts, SAFE agrees with crowdsourced human annotators 72% of the time, and on a random subset of 100 disagreement cases, SAFE wins 76% of the time. At the same time, SAFE is more than 20 times cheaper than human annotators. We also benchmark thirteen language models on LongFact across four model families (Gemini, GPT, Claude, and PaLM-2), finding that larger language models generally achieve better long-form factuality. LongFact, SAFE, and all experimental code are available at https://github.com/google-deepmind/long-form-factuality.
  @article{wei2024long,
  title={Long-Form Factuality in Large Language Models},
  author={Wei, Jerry and Yang, Chengrun and Song, Xinying and Lu, Yifeng and Hu, Nathan and Huang, Jie and Tran, Dustrin and Peng, Daiyi and Liu, Ruibo and Huang, Da and Du, Cosmo and Le, Quoc V},
  journal={arXiv preprint arXiv:2403.18802},
  year={2024}
  }
  *Lead contribution

• Large Language Models as Optimizers
  Chengrun Yang*, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, Xinyun Chen*
  International Conference on Learning Representations (ICLR), 2024
  [abstract] [arXiv] [code] [bib]
  

  Optimization is ubiquitous. While derivative-based algorithms have been powerful tools for various problems, the absence of gradient imposes challenges on many real-world applications. In this work, we propose Optimization by PROmpting (OPRO), a simple and effective approach to leverage large language models (LLMs) as optimizers, where the optimization task is described in natural language. In each optimization step, the LLM generates new solutions from the prompt that contains previously generated solutions with their values, then the new solutions are evaluated and added to the prompt for the next optimization step. We first showcase OPRO on linear regression and traveling salesman problems, then move on to prompt optimization where the goal is to find instructions that maximize the task accuracy. With a variety of LLMs, we demonstrate that the best prompts optimized by OPRO outperform human-designed prompts by up to 8% on GSM8K, and by up to 50% on Big-Bench Hard tasks.
  @article{yang2023large,
  title={Large Language Models as Optimizers},
  author={Yang, Chengrun and Wang, Xuezhi and Lu, Yifeng and Liu, Hanxiao and Le, Quoc V and Zhou, Denny and Chen, Xinyun},
  journal={arXiv preprint arXiv:2309.03409},
  year={2023}
  }
  *Equal contribution

• TabNAS: Rejection Sampling for Neural Architecture Search on Tabular Datasets
  Chengrun Yang, Gabriel Bender, Hanxiao Liu, Pieter-Jan Kindermans, Madeleine Udell, Yifeng Lu, Quoc V. Le, Da Huang
  Neural Information Processing Systems (NeurIPS), 2022
  [abstract] [arXiv] [code] [poster] [bib]
  

  The best neural architecture for a given machine learning problem depends on many factors: not only the complexity and structure of the dataset, but also on resource constraints including latency, compute, energy consumption, etc. Neural architecture search (NAS) for tabular datasets is an important but under-explored problem. Previous NAS algorithms designed for image search spaces incorporate resource constraints directly into the reinforcement learning (RL) rewards. However, for NAS on tabular datasets, this protocol often discovers suboptimal architectures. This paper develops TabNAS, a new and more effective approach to handle resource constraints in tabular NAS using an RL controller motivated by the idea of rejection sampling. TabNAS immediately discards any architecture that violates the resource constraints without training or learning from that architecture. TabNAS uses a Monte-Carlo-based correction to the RL policy gradient update to account for this extra filtering step. Results on several tabular datasets demonstrate the superiority of TabNAS over previous reward-shaping methods: it finds better models that obey the constraints.
  @article{yang2022tabnas,
  title={TabNAS: Rejection Sampling for Neural Architecture Search on Tabular Datasets},
  author={Yang, Chengrun and Bender, Gabriel and Liu, Hanxiao and Kindermans, Pieter-Jan and Udell, Madeleine and Lu, Yifeng and Le, Quoc and Huang, Da},
  booktitle={Advances in Neural Information Processing Systems},
  year={2022}
  }

• AutoML Pipeline Selection: Efficiently Navigating the Combinatorial Space
  Chengrun Yang, Jicong Fan, Ziyang Wu, Madeleine Udell
  ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), 2020
  [abstract] [pdf] [code] [ACM] [ACM version errata] [bib]
  

  Data scientists seeking a good supervised learning model on a dataset have many choices to make: they must preprocess the data, select features, possibly reduce the dimension, select an estimation algorithm, and choose hyperparameters for each of these pipeline components. With new pipeline components comes a combinatorial explosion in the number of choices! In this work, we design a new AutoML system TensorOboe to address this challenge: an automated system to design a supervised learning pipeline. TensorOboe uses low rank tensor decomposition as a surrogate model for efficient pipeline search. We also develop a new greedy experiment design protocol to gather information about a new dataset efficiently. Experiments on large corpora of real-world classification problems demonstrate the effectiveness of our approach.
  @inproceedings{yang2020automl,
  title={{AutoML} Pipeline Selection: Efficiently Navigating the Combinatorial Space},
  author={Yang, Chengrun and Fan, Jicong and Wu, Ziyang and Udell, Madeleine},
  booktitle={Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery \& Data Mining},
  pages={1446--1456},
  year={2020}
  }

• OBOE: Collaborative Filtering for AutoML Model Selection
  Chengrun Yang, Yuji Akimoto, Dae Won Kim, Madeleine Udell
  ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), 2019
  Oral Presentation
  Preliminary version at NeurIPS 2018 Workshop on Meta-Learning
  [abstract] [arXiv] [pdf] [code] [ACM] [poster] [bib]
  

  Algorithm selection and hyperparameter tuning remain two of the most challenging tasks in machine learning. Automated machine learning (AutoML) seeks to automate these tasks to enable widespread use of machine learning by non-experts. This paper introduces OBOE, a collaborative filtering method for time-constrained model selection and hyperparameter tuning. OBOE forms a matrix of the cross-validated errors of a large number of supervised learning models (algorithms together with hyperparameters) on a large number of datasets, and fits a low rank model to learn the low-dimensional feature vectors for the models and datasets that best predict the cross-validated errors. To find promising models for a new dataset, OBOE runs a set of fast but informative algorithms on the new dataset and uses their cross-validated errors to infer the feature vector for the new dataset. OBOE can find good models under constraints on the number of models fit or the total time budget. To this end, this paper develops a new heuristic for active learning in time-constrained matrix completion based on optimal experiment design. Our experiments demonstrate that OBOE delivers state-of-the-art performance faster than competing approaches on a test bed of supervised learning problems. Moreover, the success of the bilinear model used by OBOE suggests that AutoML may be simpler than was previously understood.
  @inproceedings{yang2019oboe,
  title={{OBOE}: Collaborative Filtering for {AutoML} Model Selection},
  author={Yang, Chengrun and Akimoto, Yuji and Kim, Dae Won and Udell, Madeleine},
  booktitle={Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery \& Data Mining},
  pages={1173--1183},
  year={2019}
  }

• SETS: Leveraging Self-Verification and Self-Correction for Improved Test-Time Scaling
  Jiefeng Chen, Jie Ren, Xinyun Chen, Chengrun Yang, Ruoxi Sun, Sercan Ö Arık
  [abstract] [arXiv] [bib]
  

  Recent advancements in Large Language Models (LLMs) have created new opportunities to enhance performance on complex reasoning tasks by leveraging test-time computation. However, conventional approaches such as repeated sampling with majority voting or reward model scoring, often face diminishing returns as test-time compute scales, in addition to requiring costly task-specific reward model training. In this paper, we present Self-Enhanced Test-Time Scaling (SETS), a novel method that leverages the self-verification and self-correction capabilities of recent advanced LLMs to overcome these limitations. SETS integrates sampling, self-verification, and self-correction into a unified framework, enabling efficient and scalable test-time computation for improved capabilities at complex tasks. Through extensive experiments on challenging planning and reasoning benchmarks, compared to the alternatives, we demonstrate that SETS achieves significant performance improvements and more favorable test-time scaling laws.
  @article{chen2025sets,
  title={SETS: Leveraging Self-Verification and Self-Correction for Improved Test-Time Scaling},
  author={Chen, Jiefeng and Ren, Jie and Chen, Xinyun and Yang, Chengrun and Sun, Ruoxi and Ar{\i}k, Sercan {\"O}},
  journal={arXiv preprint arXiv:2501.19306},
  year={2025}
  }

Selected Frameworks

• Simply [code]: Minimal Code for End-to-End Frontier LLM Research

• Langfun [code]: Object-Oriented Prompting for LLM steerability and agent development