))
Automating Scientific Discovery: ScienceAgentBench
Fetch error
Hmmm there seems to be a problem fetching this series right now.
Last successful fetch was on November 09, 2024 13:09 (
What now? This series will be checked again in the next day. If you believe it should be working, please verify the publisher's feed link below is valid and includes actual episode links. You can contact support to request the feed be immediately fetched.
Manage episode 446714679 series 3605861
Το περιεχόμενο παρέχεται από το Brian Carter. Όλο το περιεχόμενο podcast, συμπεριλαμβανομένων των επεισοδίων, των γραφικών και των περιγραφών podcast, μεταφορτώνεται και παρέχεται απευθείας από τον Brian Carter ή τον συνεργάτη της πλατφόρμας podcast. Εάν πιστεύετε ότι κάποιος χρησιμοποιεί το έργο σας που προστατεύεται από πνευματικά δικαιώματα χωρίς την άδειά σας, μπορείτε να ακολουθήσετε τη διαδικασία που περιγράφεται εδώ https://el.player.fm/legal.
A scientific paper exploring the development and evaluation of language agents for automating data-driven scientific discovery. The authors introduce a new benchmark called ScienceAgentBench, which consists of 102 diverse tasks extracted from peer-reviewed publications across four disciplines: Bioinformatics, Computational Chemistry, Geographical Information Science, and Psychology & Cognitive Neuroscience. The benchmark evaluates the performance of language agents on individual tasks within a scientific workflow, aiming to provide a more rigorous assessment of their capabilities than solely focusing on end-to-end automation. The paper's experiments test five language models across three frameworks: direct prompting, OpenHands CodeAct, and self-debug, revealing that even the best-performing agent, Claude-3.5-Sonnet with self-debug, can only independently solve 32.4% of the tasks and 34.3% with expert-provided knowledge. The results highlight the limited capacities of current language agents in automating scientific tasks and underscore the need for further development to improve their ability to process scientific data, utilize expert knowledge, and handle complex tasks.
Read more: https://arxiv.org/pdf/2410.05080
71 επεισόδια
Μοίρασέ το
