Build Computing Olympiad Agents with LangGraph

In this tutorial, we create Olympiad programming agents using LangGraph, drawing upon the techniques and benchmark dataset introduced in the paper "Can Language Models Solve Olympiad Programming?" by Quan Shi, Michael Tang, Karthik Narasimhan, and Shunyu Yao. #AI #LangGraph #llm

Throughout the tutorial, we learn how to enhance the agent's performance by incorporating three key techniques:

1. Reflection: In the first part, we develop a zero-shot tool calling agent and prompt it to reflect on the test case results, enabling it to correct its initial errors. This agent is comparable to the one reported in the paper, achieving a pass rate of 12.38 on the USACO benchmark.

2. Retrieval: In the second part, we introduce an initial retrieval step, functioning as an "episodic memory" for the agent. This step retrieves high-quality few-shot examples from our corpora of programming problems, assisting in solving the bronze level question. The agent created in this part is similar to the one benchmarked at 20.2 in the paper.

3. Human-in-the-loop: In the third part, we utilize the `interrupt_after` feature to allow the user to copilot the agent, guiding it towards a better answer. The benchmark performance in this scenario is limited only by the competitiveness of the human collaborator.

By the end of this tutorial, you will have a solid understanding of how to build agents in LangGraph, leveraging advanced techniques such as reflection, retrieval, and human-in-the-loop interaction.

Chapters:

00:00 Introduction
01:30 Overview of the USACO Dataset and Benchmark Performance
02:38 Introduce the 3 techniques
06:01 Part 1: Zero-Shot Agent with Reflection Capabilities
12:27 Review Zero-Shot Results
13:33 Part 2: Add Episodic Memory Retrieval
18:48 Review Results of Augmented Agent
21:06 Part 3: Add Human-in-the-Loop
26:43 Review Copilot Results
27:44 Conclusion

Additional Resources:
- Research Paper: https://arxiv.org/abs/2404.10952v1
- Tutorial Code: https://langchain-ai.github.io/langgraph/tutorials/usaco/usaco/
- Reflection Agents: https://www.youtube.com/watch?v=v5ymBTXNqtk


Presenter: https://twitter.com/WHinthorn