โ๏ธAWS Machine Learning BlogโขStalecollected in 4m
Improve agent tool-calling with SFT and DPO on SageMaker
๐กLearn how to use SFT and DPO on SageMaker to make your AI agents more reliable at calling external tools.
โก 30-Second TL;DR
What Changed
Combine SFT and DPO to optimize SLM performance for tool-calling tasks.
Why It Matters
This approach allows developers to build more reliable AI agents by reducing hallucination and improving function-calling precision in resource-constrained environments.
What To Do Next
Follow the tutorial to set up a SageMaker training job using your own tool-calling dataset to benchmark the performance gain of DPO over standard SFT.
Who should care:Developers & AI Engineers
Key Points
- โขCombine SFT and DPO to optimize SLM performance for tool-calling tasks.
- โขUse Amazon SageMaker AI training jobs to offload infrastructure management.
- โขImplement systematic evaluation methods to compare base models against fine-tuned variants.
- โขFocus on data-driven decision-making for model quality improvements.
๐ง Deep Insight
Web-grounded analysis with 24 cited sources.
๐ Enhanced Key Takeaways
- โขThe combined SFT and DPO approach is more stable and computationally efficient than traditional Reinforcement Learning from Human Feedback (RLHF) because DPO directly optimizes the model based on preference data without needing to train a separate reward model.
- โขAmazon SageMaker AI now offers serverless customization for fine-tuning, which automatically provisions compute resources, simplifying the infrastructure management for advanced techniques like DPO, Reinforcement Learning from Verifiable Rewards (RLVR), and Reinforcement Learning from AI Feedback (RLAIF).
- โขThe evolution of AI agents has shifted from early models learning single-tool calls (e.g., Meta AI's Toolformer in 2023) to current research focusing on multi-tool orchestration, which involves dynamic tool selection, managing cross-tool dependencies, sequential/parallel scheduling, and robust failure recovery.
- โขSupervised Fine-Tuning (SFT) is crucial for teaching the small language model (SLM) the basic task format, specific tool-calling language, commands, and constraints, serving as a foundational step before DPO refines the model's preferences for better responses.
- โขEvaluating tool-calling accuracy involves metrics beyond simple correctness, such as 'slot performance' (correctly identifying and filling parameters) and 'relevance scores,' which can be improved by DPO but may sometimes show trade-offs with other capabilities like rule adherence.
๐ Competitor Analysisโธ Show
| Feature/Platform | AWS SageMaker | Google Cloud Vertex AI | Microsoft Azure ML | Databricks ML | TrueFoundry |
|---|---|---|---|---|---|
| Managed Service | Fully managed ML platform, abstracts infrastructure. | End-to-end ML platform, managed notebooks, pipelines. | Robust MLOps platform, Azure ML Studio web interface. | ML platform built on Lakehouse, managed feature stores. | Modern MLOps platform, end-to-end orchestration. |
| LLM Fine-tuning | Supports SFT, DPO, RLHF, RLVR, RLAIF; serverless options. | Offers AutoML, custom model training, hyperparameter tuning. | Supports automated ML, hyperparameter tuning, pipeline designer. | Built-in AutoML, experiment tracking via MLflow. | Native support for 250+ LLMs, advanced GenAI capabilities. |
| Tool-Calling/Agents | Focus on improving agent tool-calling accuracy with SFT/DPO. | General LLM capabilities, can be integrated for agentic tasks. | Deep dive into tool calling fine-tuning for agents. | General ML platform, agent capabilities via custom integration. | Focus on GenAI adoption, multi-cloud LLM Gateway. |
| Pricing Model | Usage-based, can be complex; serverless options aim to reduce idle costs. | Usage-based and modular (compute, storage, training, prediction). | Usage-based, integrated with Azure ecosystem. | Usage-based, tailored to compute and collaboration needs. | Focus on fast, developer-friendly, cloud-agnostic deployment. |
| Key Differentiator | Deep integration with AWS ecosystem, extensive ML services. | Strong integration with GCP services (BigQuery, Dataflow). | Ideal for Microsoft/Azure ecosystem users, enterprise features. | Combines data warehousing and ML on a single platform. | Cloud-agnostic, GitOps workflows, API-first approach. |
๐ ๏ธ Technical Deep Dive
- Supervised Fine-Tuning (SFT): Involves curating a high-quality dataset of input-output pairs that explicitly demonstrate the desired model behavior, teaching the model to recognize tool-specific language, commands, and constraints.
- Direct Preference Optimization (DPO): Refines model interactions by directly incorporating human feedback or predefined objectives into the training loop. It uses a 'like this, not like that' preference, optimizing the same goals as reinforcement learning without requiring reward functions or reward models, thereby reducing resource requirements and training time.
- DPO as Binary Classification: DPO reparameterizes the RLHF objective, treating the policy optimization task as a binary classification problem, where the model learns to prefer chosen responses over rejected ones.
- DPO Hyperparameters: A key hyperparameter for DPO is
beta, which controls how aggressively the model adopts new preferences, typically ranging from 0 to 2, with 0.1 to 0.5 being common starting points. - SFT-then-DPO Workflow: This two-stage approach is recommended for stability and enhanced performance. SFT first establishes a robust initial policy and basic capabilities, and then DPO refines subtle nuances and aligns with human preferences, leading to faster convergence and higher-quality results.
- Tool-Calling Evolution: Early LLMs like Meta AI's Toolformer (Feb 2023) pioneered self-supervised tool learning, while OpenAI's function calling (2023) introduced structured, programmable API calls with JSON parameters, enabling more predictable interaction with external systems.
๐ฎ Future ImplicationsAI analysis grounded in cited sources
AI agents will achieve greater autonomy and handle increasingly complex, multi-step tasks.
The combination of SFT and DPO, coupled with advancements in multi-tool orchestration, enables agents to move beyond simple text generation to active problem-solving and sophisticated interaction with external systems.
Direct Preference Optimization (DPO) will become a standard and widely adopted method for aligning Large Language Models (LLMs).
DPO's inherent simplicity, stability, and computational efficiency compared to traditional RLHF make it a more accessible and practical approach for aligning LLMs with human preferences.
Cloud machine learning platforms will continue to integrate and simplify advanced fine-tuning and alignment techniques through serverless and managed offerings.
AWS's recent introduction of serverless customization in SageMaker AI for DPO, RLVR, and RLAIF indicates a clear industry trend towards abstracting infrastructure complexities to accelerate LLM development and deployment.
โณ Timeline
2018
BERT introduced, providing a foundational step for Large Language Models (LLMs).
2022
ChatGPT's release popularized LLMs and highlighted the importance of tool-use capabilities.
2023-02
Meta AI introduced Toolformer, pioneering self-supervised tool learning for LLMs.
2023
OpenAI introduced function calling, enabling structured and programmable tool interaction for LLMs.
2023
The paper 'Direct Preference Optimization: Your Language Model is Secretly a Reward Model' was published, introducing DPO as an alternative to RLHF.
2026-01
AWS launched Serverless Customization in Amazon SageMaker AI, supporting DPO, RLVR, and RLAIF for streamlined LLM fine-tuning.
๐ Sources (24)
Factual claims are grounded in the sources below. Forward-looking analysis is AI-generated interpretation.
๐ฐ
Weekly AI Recap
Read this week's curated digest of top AI events โ
๐Related Updates
AI-curated news aggregator. All content rights belong to original publishers.
Original source: AWS Machine Learning Blog โ