{"id":77549,"date":"2024-10-30T03:29:30","date_gmt":"2024-10-30T07:29:30","guid":{"rendered":"https:\/\/blog.cyberconservices.com\/?p=77549"},"modified":"2024-10-27T14:37:16","modified_gmt":"2024-10-27T18:37:16","slug":"how-anthropics-new-computer-use-ability-could-further-ai-automation","status":"publish","type":"post","link":"https:\/\/blog.cyberconservices.com\/index.php\/2024\/10\/30\/how-anthropics-new-computer-use-ability-could-further-ai-automation\/","title":{"rendered":"How Anthropic’s new ‘computer use’ ability could further AI automation"},"content":{"rendered":"

By Anirban Ghoshal <\/a>\u00a0– Anthropic\u2019s Claude 3.5 Sonnet large language model has gained a new ability: operating a computer.<\/p>\n

The new ability, which the company is calling \u201ccomputer use,\u201d is currently in beta test. It enables developers to instruct Claude 3.5 Sonnet, through the Anthropic API<\/a>, to read and interpret what\u2019s on the display, type text, move the cursor, click buttons, and switch between windows or applications \u2014 much as today\u2019s\u00a0robotic process automation<\/a>\u00a0(RPA) tools can be instructed \u2014 much more laboriously \u2014 to do.<\/p>\n

\n
\n
\n

To apply its ability to use a computer, Claude 3.5 Sonnet starts from a prompt defining its goal, identifies the steps necessary to reach that goal, and then scans screenshots much as a human would look at the screen of a computer to figure out how to perform those steps.<\/p>\n<\/div>\n<\/div>\n<\/div>\n

\n
\n
\n

Key to that is Claude 3.5 Sonnet\u2019s new-found ability to return the coordinates of a feature in an image, enabling it to position the cursor on a button or in a text box on screen.<\/p>\n

Claude 3.5 Sonnet needs definitions of the tools and software on the computer it will operate, and authorization to access them. It then sends requests to use the tools, and examines the response to see if has succeeded or whether it needs to continue using the tool to complete its task.\u00a0 Read On:<\/a><\/p>\n<\/div>\n<\/div>\n<\/div>\n

 <\/p>\n","protected":false},"excerpt":{"rendered":"

Anthropic\u2019s has upgraded its Claude 3.5 Sonnet LLM with a new ability, computer use, opening up new opportunities in robotic process automation (RPA) and more.<\/p>\n Continue reading →<\/span><\/a>","protected":false},"author":1,"featured_media":77552,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"advanced_seo_description":"","jetpack_seo_html_title":"","jetpack_seo_noindex":false,"jetpack_seo_schema_type":"","_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[674,628],"tags":[675,629],"class_list":["post-77549","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-robotic-process-automation-processs-automation","category-rpa","tag-robotic-process-automation","tag-rpa"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"https:\/\/i0.wp.com\/blog.cyberconservices.com\/wp-content\/uploads\/2024\/10\/Photograph-of-a-floating-brain-with-waves-2.png?fit=1024%2C1024&ssl=1","jetpack-related-posts":[{"id":46435,"url":"https:\/\/blog.cyberconservices.com\/index.php\/2022\/02\/01\/a-dirty-secret-rpa-is-not-designed-for-process-automation\/","url_meta":{"origin":77549,"position":0},"title":"A dirty secret: RPA is not designed for process automation","author":"Rick Backus","date":"February 1, 2022","format":false,"excerpt":"Maybe it is not precisely automation and more macro-esk, but if a user can do something else while an RPA program runs I'm good calling automation. By\u00a0Mike Fitzmaurice\u00a0 - Robotic Process Automation (RPA) has had a lot of deserved success in businesses seeking to carry out basic, repetitive tasks ordinarily\u2026","rel":"","context":"In "Robotic Process Automation"","block_context":{"text":"Robotic Process Automation","link":"https:\/\/blog.cyberconservices.com\/index.php\/category\/robotic-process-automation-processs-automation\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":59754,"url":"https:\/\/blog.cyberconservices.com\/index.php\/2022\/10\/25\/what-is-robotic-process-automation-and-how-can-it-save-you-time\/","url_meta":{"origin":77549,"position":1},"title":"What Is Robotic Process Automation and How Can It Save You Time?","author":"Rick Backus","date":"October 25, 2022","format":false,"excerpt":"Several misconceptions exist about RPA, as many people don't know what Robotic Process Automation is and how it is used within a business environment. Firstly, in the context of automation, what is a robot? In simple terms, a robot is a machine capable of automatically carrying out complex tasks. Even\u2026","rel":"","context":"In "Robotic Process Automation"","block_context":{"text":"Robotic Process Automation","link":"https:\/\/blog.cyberconservices.com\/index.php\/category\/robotic-process-automation-processs-automation\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":32911,"url":"https:\/\/blog.cyberconservices.com\/index.php\/2019\/10\/21\/robotic-process-automation-rpa-how-it-works\/","url_meta":{"origin":77549,"position":2},"title":"Robotic process automation (RPA): How it works","author":"Rick Backus","date":"October 21, 2019","format":false,"excerpt":"By\u00a0Kevin Casey - \u201cDo more with less\u201d might be a timeworn excuse for a business mantra, but\u00a0robotic process automation (RPA)\u00a0is a tool that could actually help teams do just that in the right circumstances. That\u2019s the big selling point of RPA. The phrase itself might sound complicated or scary, but\u2026","rel":"","context":"In "Robotic Process Automation"","block_context":{"text":"Robotic Process Automation","link":"https:\/\/blog.cyberconservices.com\/index.php\/category\/robotic-process-automation\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/www.cyberconservices.com\/wp-content\/uploads\/2019\/10\/robot-2697683__340.png?resize=350%2C200","width":350,"height":200},"classes":[]},{"id":31932,"url":"https:\/\/blog.cyberconservices.com\/index.php\/2019\/10\/14\/rpa-basics-what-it-is-benefits-downsides-use-cases\/","url_meta":{"origin":77549,"position":3},"title":"RPA basics: What it is, benefits, downsides, use cases","author":"Rick Backus","date":"October 14, 2019","format":false,"excerpt":"By Diann Daniel - BOSTON -- Robotic process automation is among the most talked-about technologies today. Not understanding what it means in 2019 may feel akin to not knowing what\u00a0Games of Thrones\u00a0is about or who Taylor Swift is. In truth, you're not alone. Here at the\u00a0IFS World Conference, Bob De\u2026","rel":"","context":"In "Robotic Process Automation"","block_context":{"text":"Robotic Process Automation","link":"https:\/\/blog.cyberconservices.com\/index.php\/category\/robotic-process-automation-processs-automation\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/www.cyberconservices.com\/wp-content\/uploads\/2019\/10\/robot-4363354__340.png?resize=350%2C200","width":350,"height":200},"classes":[]},{"id":46313,"url":"https:\/\/blog.cyberconservices.com\/index.php\/2021\/11\/22\/what-the-future-holds-for-robotic-process-automation-the-next-chapter\/","url_meta":{"origin":77549,"position":4},"title":"What the Future Holds for Robotic Process Automation: The Next Chapter","author":"Rick Backus","date":"November 22, 2021","format":false,"excerpt":"While RPA can facilitate integration between applications and data sets it remains limited to using methods that actual users would employ through key strokes and mouse clicks. Enabling RPA to leverage the APIs of other systems would dramatically broaden the opportunities for further (hyper-) automation. By\u00a0Dwight Davis\u00a0 - As we\u2019ve\u2026","rel":"","context":"In "API"","block_context":{"text":"API","link":"https:\/\/blog.cyberconservices.com\/index.php\/category\/api\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":76824,"url":"https:\/\/blog.cyberconservices.com\/index.php\/2023\/07\/20\/charting-a-new-course-for-robotic-process-automation-in-highly-secured-environments\/","url_meta":{"origin":77549,"position":5},"title":"Charting a New Course for Robotic Process Automation in Highly Secured Environments","author":"Rick Backus","date":"July 20, 2023","format":false,"excerpt":"In a world where companies are under immense pressure to improve their bottom line, finding ways to boost productivity while reducing costs has become a top priority. A recent study by McKinsey has revealed that the United States could grow its cumulative GDP by a staggering $10 trillion by 2030\u2026","rel":"","context":"In "RPA"","block_context":{"text":"RPA","link":"https:\/\/blog.cyberconservices.com\/index.php\/category\/rpa\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/blog.cyberconservices.com\/wp-content\/uploads\/2023\/07\/3-three-people-watching-and-pointing-at-a-2.png?fit=1024%2C1024&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/blog.cyberconservices.com\/wp-content\/uploads\/2023\/07\/3-three-people-watching-and-pointing-at-a-2.png?fit=1024%2C1024&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/blog.cyberconservices.com\/wp-content\/uploads\/2023\/07\/3-three-people-watching-and-pointing-at-a-2.png?fit=1024%2C1024&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/blog.cyberconservices.com\/wp-content\/uploads\/2023\/07\/3-three-people-watching-and-pointing-at-a-2.png?fit=1024%2C1024&ssl=1&resize=700%2C400 2x"},"classes":[]}],"jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/blog.cyberconservices.com\/index.php\/wp-json\/wp\/v2\/posts\/77549","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.cyberconservices.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.cyberconservices.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.cyberconservices.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.cyberconservices.com\/index.php\/wp-json\/wp\/v2\/comments?post=77549"}],"version-history":[{"count":1,"href":"https:\/\/blog.cyberconservices.com\/index.php\/wp-json\/wp\/v2\/posts\/77549\/revisions"}],"predecessor-version":[{"id":77553,"href":"https:\/\/blog.cyberconservices.com\/index.php\/wp-json\/wp\/v2\/posts\/77549\/revisions\/77553"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.cyberconservices.com\/index.php\/wp-json\/wp\/v2\/media\/77552"}],"wp:attachment":[{"href":"https:\/\/blog.cyberconservices.com\/index.php\/wp-json\/wp\/v2\/media?parent=77549"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.cyberconservices.com\/index.php\/wp-json\/wp\/v2\/categories?post=77549"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.cyberconservices.com\/index.php\/wp-json\/wp\/v2\/tags?post=77549"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}