))
Automating Scientific Discovery: ScienceAgentBench
Αρχειοθετημένη σειρά ("Ανενεργό feed" status)
When?
This feed was archived on May 02, 2025 14:13 (
Why? Ανενεργό feed status. Οι διακομιστές μας δεν ήταν σε θέση να ανακτήσουν ένα έγκυρο podcast feed για μια παρατεταμένη περίοδο.
What now? You might be able to find a more up-to-date version using the search function. This series will no longer be checked for updates. If you believe this to be in error, please check if the publisher's feed link below is valid and contact support to request the feed be restored or if you have any other concerns about this.
Manage episode 446714679 series 3605861
Το περιεχόμενο παρέχεται από το Brian Carter. Όλο το περιεχόμενο podcast, συμπεριλαμβανομένων των επεισοδίων, των γραφικών και των περιγραφών podcast, μεταφορτώνεται και παρέχεται απευθείας από τον Brian Carter ή τον συνεργάτη της πλατφόρμας podcast. Εάν πιστεύετε ότι κάποιος χρησιμοποιεί το έργο σας που προστατεύεται από πνευματικά δικαιώματα χωρίς την άδειά σας, μπορείτε να ακολουθήσετε τη διαδικασία που περιγράφεται εδώ https://el.player.fm/legal.
A scientific paper exploring the development and evaluation of language agents for automating data-driven scientific discovery. The authors introduce a new benchmark called ScienceAgentBench, which consists of 102 diverse tasks extracted from peer-reviewed publications across four disciplines: Bioinformatics, Computational Chemistry, Geographical Information Science, and Psychology & Cognitive Neuroscience. The benchmark evaluates the performance of language agents on individual tasks within a scientific workflow, aiming to provide a more rigorous assessment of their capabilities than solely focusing on end-to-end automation. The paper's experiments test five language models across three frameworks: direct prompting, OpenHands CodeAct, and self-debug, revealing that even the best-performing agent, Claude-3.5-Sonnet with self-debug, can only independently solve 32.4% of the tasks and 34.3% with expert-provided knowledge. The results highlight the limited capacities of current language agents in automating scientific tasks and underscore the need for further development to improve their ability to process scientific data, utilize expert knowledge, and handle complex tasks.
Read more: https://arxiv.org/pdf/2410.05080
71 επεισόδια
Μοίρασέ το
