ChatGPT Can Now Read Some Mac Desktop Apps

At the moment, OpenAI’s ChatGPT is able to collaborate with other applications that are already installed on your personal computer

On Thursday, the company made the announcement that the ChatGPT desktop application for MacOS is now capable of reading code from a select group of developer-oriented coding applications. These applications include Visual Studio Code, Xcode, TextEdit, Terminal, and iTerm2.

Developers will no longer be required to copy and paste their code into ChatGPT, which has become a frequent method of utilizing the chatbot. This would be a significant change.

Now, when the feature is activated, OpenAI will automatically communicate the piece of code that you are working on through its chatbot as context, together with the prompt that you have provided.

On the other hand, in contrast to widespread artificial intelligence coding tools like Cursor or GitHub Copilot, ChatGPT is not currently able to write code straight into developer applications on your behalf.

Work with Apps is a feature that is not even close to being an artificial intelligence agent; but, according to OpenAI, getting ChatGPT to comprehend other applications is a “key building block” towards the development of agentic systems.

As opposed to responding to instructions or their own responses, one of the most difficult issues that artificial intelligence agents are now confronting is getting them to grasp the rest of the screen on your computer.

OpenAI has stated that it will initially concentrate on coding applications; this is most likely due to the fact that AI coding helpers have become one of the most prominent use cases for linear learning machines (LLMs).

The feature is currently accessible to users of Plus and Teams, and it will later be made available to users of Enterprise and Edu within the next few weeks.

OpenAI has stated that ChatGPT will be able to collaborate with other kinds of applications in the future, notably text-based applications that might be utilized for projects including writing.

During a demonstration with TechCrunch, an employee of OpenAI opened the ChatGPT app and an Xcode environment that contained a straightforward project modeling the solar system. However, the project did not include the Earth.

An Xcode tab was picked by the employee within ChatGPT, which instructed the artificial intelligence chatbot to examine the application. The employee then instructed the chatbot to “add the missing planets.”

As a result of the chatbot’s ability to successfully accomplish the assignment, a line of code was written to represent the Earth in a format that was consistent with the rest of the project. Despite this, they were still required to copy and paste ChatGPT’s response back into their environment.

Alexander Embiricos, who is in charge of the desktop product for OpenAI, stated that the majority of OpenAI’s reliance on the MacOS Accessibility API when it comes to reading text and translating it to ChatGPT is for the purpose of reading various applications.

MacOS’s screen reader, which is responsible for ensuring that Apple’s VoiceOver functionality functions properly, has been around for almost twenty years. In general, it is considered to be fairly trustworthy for the majority of mainstream applications, but not for everything.

Users of Work with Apps are required to install a specialized extension in order to query material for certain applications, such as Visual Studio Code from Microsoft.

And because Apple’s screen reader can only read text, it is unable to assist ChatGPT in comprehending visual features such as photographs, the orientation of objects, or movies. This is because the name of the screen reader says as much.

Utilize applications with Send the latest two hundred lines of code you have written through ChatGPT, along with each prompt for specific applications. On the other hand, the chatbot will take in all of the code that is now displayed in the window that is topmost.

To assist ChatGPT in concentrating on the appropriate portion of the project, you can highlight specific sections of code or text; however, ChatGPT will also include the text that surrounds the highlighted region. It appears that a significant number of input tokens will be required for this.

Uncertainty exists regarding OpenAI’s intentions for the expansion of this feature to other applications that are not compatible with Apple’s screen reader.

In order to comprehend and make use of other applications, Anthropic, which is one of OpenAI’s competitors, has built an artificial intelligence system that examines screenshots of a user’s desktop.

To tell you the truth, the method that Anthropic is taking gives a lot of room for improvement in its current state: it is slow and it makes a lot of mistakes.

The AI agent, on the other hand, is a more general-purpose version that does not rely on application programming interfaces (APIs) and is capable of doing more than merely reading text in another window.

During a briefing with TechCrunch, Embiricos stated, “This is not meant to be an agent; rather, it is a way to collaborate with coding tools to begin with, and there will be additional tools coming soon.”

I believe that this is a really important building component from the perspective of the agents. This concept that ChatGPT is able to comprehend or work with all of the stuff that you have so that it can assist you with it.

It is especially noteworthy that this move toward agents has been taken in light of recent claims that OpenAI is getting close to releasing a general purpose artificial intelligence agent that has been nicknamed “Operator,” as reported by Bloomberg.

It is anticipated that the tool will be available in the early years of 2025. It would compete with other early attempts at general-purpose artificial intelligence agents, such as Anthropic’s Computer usage or Google’s rumored “Jarvis” agent.

The initial release of these functionalities will be made available on MacOS by OpenAI, just a few weeks before Apple will debut an integration with ChatGPT in December.

It is not known when Work with Apps would be made available for Windows, which is the operating system that was developed by Microsoft, which is OpenAI’s main backer.