Organize | Tu Min Zheng Liyuan
Produced | CSDN (ID: CSDNnews)
“Scattered flowers gradually become charming eyes”, in the new day, OpenAI once again throws out the “king bomb”: ChatGPT launched a plug-in function, which can not only connect to the Internet, but also open the search, and can also execute code and run calculations…
With one move, not only the effort of turning over the book and opening the calculator is saved, but now you can also use ChatGPT to directly obtain the experience of the search engine providing the latest news.
From web browsers, code interpreters, to third-party tools from OpenAI
OpenAI said, “Plugins are tools designed specifically for language models, with security as a core principle, and help ChatGPT access the latest information, run calculations, or use third-party services.”
Specifically, the scope of its plug-ins includes third-party tools such as WolframAlpha, OpenTable, and Slack, as well as officially launched web browser plug-ins and code interpreters.
Based on these, ordinary users can also generate code, run code, upload and download files from csv data to images, and evaluate the output results within the ChatGPT interface.
In addition, OpenAI has also released the code of the knowledge base retrieval plugin (https://github.com/openai/chatgpt-retrieval-plugin).
Web browser
The explosion of ChatGPT is that it is almost “omnipotent”: coding, writing planning, writing novels, being a customer service… But as a language model based on historical data, ChatGPT has a problem that has been criticized: its training The data is as of September 2021, and it cannot accurately answer the updated information data.
Today, this “seal” is finally lifted – OpenAI has launched a web browser plug-in, that is, ChatGPT can be connected to the Internet!
“Inspired by past work (including our own WebGPT, as well as GopherCite, BlenderBot2, LaMDA2, and others), we now allow language models to read from the Internet, strictly expanding their scope beyond the training corpus to obtain The latest information of the day.”
OpenAI shows an example of ChatGPT already retrieving this year’s latest Oscar information through a web browser plugin:
According to OpenAI, web browsers call New Bing’s search API to retrieve content from the web, thus inheriting a lot of Microsoft’s work on security: (1) the reliability and authenticity of information sources; (2) preventing the retrieval of problematic content “Safe Mode”. Web browsers can display visited websites and cite their sources in ChatGPT’s replies.
In addition, OpenAI clearly states that the plugin is limited to retrieving information and does not include “transactional” operations such as form submissions. It will also run on a separate server, so ChatGPT’s browsing activity is separate from its infrastructure.
Code Interpreter
“An experimental ChatGPT model that can use Python to handle uploads and downloads,” is OpenAI’s introduction to its latest code interpreter.
In simple terms, OpenAI provides a working Python interpreter in a sandboxed, firewalled execution environment, and some temporary disk space. The code run by the code interpreter plugin is evaluated in a persistent session that is valid for the duration of the chat session (i.e. has a live, will timeout), and subsequent calls can build on top of each other. Currently, this feature supports uploading files to the current session workspace, as well as downloading work results.
In addition to generating code, the code interpreter (CI) also supports the following functions:
-
Solve math problems, including quantitative and qualitative math problems
-
Perform data analysis and visualization
-
Convert files between different formats
In addition, OpenAI also said that it is inviting users to try the code interpreter and find other useful functions.
To this end, foreign user Andrew Mayne tried to find that the code interpreter can also analyze the output and use it in another function. This means that you can string together different parts of the code so that the output of one becomes the input of another.
He does this by having CI use an algorithm to generate a maze, convert the maze into blocks, use an algorithm to find the exit, make it look like Pac-Man, and generate a GIF.
Previously, when developers used ChatGPT to create code, it involved testing the output in another environment. Now you can do a lot of development within ChatGPT without leaving the user interface. It can be said that the programming efficiency of developers has been greatly improved.
Through some actual tests by Andrew Mayne, we might as well witness the power of the code interpreter:
Using OpenCV for face finding
Using OpenCV and a simple face search algorithm (Haar Cascade Classifier), CI was able to create a function to find faces in this image:
Planet Orbits
Using the Matplotlib library, ChatGPT was able to simulate the orbits of the inner planets and save them as gif images.
Create a machine learning model
While ChatGPT can’t load any machine learning libraries in CI (for now), we can do some basic statistical text generation with n-grams. In this example, Andrew Mayne gave it a book title, asked it to create a prediction algorithm, and then predicted the next word in a sequence of text. The results won’t worry GPT-4 anytime soon, but they’re interesting nonetheless.
Drawing
Ask ChatGPT to draw a cat and it creates something resembling a cat.
Draw a cat with a blue top hat and a pipe in its mouth.
Search
OpenAI has open sourced a ChatGPT retrieval plugin (https://github.com/openai/chatgpt-retrieval-plugin), which allows ChatGPT to search the content of a vector database, such as from (Milvus, Pinecone, Qdrant, Redis, Weaviate or Zilliz) as an index search and add the best results to ChatGPT conversations, provided, of course, that the database content has the permission of the individual or organization.
This allows developers to add the content they are authorized to use in search plugins and ask questions or express needs through natural language to get the most relevant document snippets from their data sources (such as files, comments, emails or public documents).
Third-party plug-ins
For third-party plug-ins, OpenAI stated that developers who have passed the shortlist can build plug-ins for ChatGPT by themselves, and gave the relevant steps:
1. Establish an endpoint API that you want the language model to call (could be a new API, an existing API, or a wrapper around an existing API designed specifically for LLM).
2. Create an OpenAPI specification that documents the API, and a manifest file that links to the OpenAPI specification and includes some plugin-specific metadata.
When starting a conversation on chat.openai.com, users can choose which third-party plugins they wish to enable. Documentation about enabling plugins is presented to the language model as part of the dialog content, enabling the model to implement functionality based on the plugin APIs that need to be called.
At present, ChatGPT has officially announced the first batch of third-party plug-ins, mainly created by Expedia, FiscalNote, Instacart, KAYAK, Klarna, Milo, OpenTable, Shopify, Slack, Speak, Wolfram and Zapier.
Among them, the computational knowledge engine Wolfram|Alpha excitedly said: “Although it is still in the early stages, this is already very impressive – people can start to see how amazingly powerful what we call ‘ChatGPT + Wolfram’ is (even possibly is revolutionary).”
For example, when ChatGPT was asked about the number of livestock in Turkey last month, it simply made up a plausible but false figure. Now, after calling the Wolfram plugin, ChatGPT can give a “good and authoritative” answer and visualize it:
Available now?
Based on free plugins and third-party services, never worry about ChatGPT nonsense anymore.
OpenAI also highlights this in terms of security and risk, by incorporating explicit access to external data-such as up-to-date information on the web, code-based calculations, or information retrieved by custom plugins-language models can be validated through evidence-based references to enhance the quality of their responses.
These references not only improve the utility of the model, but also allow the user to assess the trustworthiness of the model output and cross-check its accuracy, potentially mitigating the risks associated with over-reliance.
So for developers and users, when can it be used?
In fact, since the release of large-scale model tools and products such as ChatGPT, Bard, New Bing, and Wenxin Yiyan, the simple word that many users are most afraid of is “waitlist”. As expected, this time is no exception (https:/ /openai.com/blog/chatgpt-plugins).
OpenAI said that currently only a small number of users (preferably a small number of developers and ChatGPT Plus users) can start using the plug-in function, and it plans to gradually roll out larger-scale access after learning more information (for plug-in developers, ChatGPT users , and after the alpha period, API users who want to integrate the plugin into their product).
References:
https://openai.com/blog/chatgpt-plugins
ChatGPT + Code Interpreter = Magic
?Java 20 released! ?Li Yanhong: Ten years later, he will not be able to write prompt words or be eliminated; GitHub released Copilot X, which can generate code by moving the mouth; ?Feishu's "365 Possibilities"