ChatGPT Plugins - First Impressions

Posted on Edited on In

Shiny New Toy

TL;DR: ChatGPT is a fun toy that is getting more powerful by the day. The plugin framework is a shiny new toy that promises to make ChatGPT a powerful tool for automating tasks and connecting to the internet. I'm excited to see where it goes.

Enter Stage Right: ChatGPT's initial plugin framework release promises to dazzle by connecting it to the internet via REST APIs. This is a big deal, and I'm excited to see where it goes. I've been tinkering with the plugin framework for a few days now, and I'm impressed with the simplicity of the implementation and the power of the results. I'm also impressed with the speed at which OpenAI is iterating on the framework, and I'm excited to see what the future holds. I'm not sure if I'm more excited about the possibilities for ChatGPT or the possibilities for the plugin framework itself. I'm sure both will be amazing.

OpenAI plugins connect ChatGPT to third-party applications. These plugins enable ChatGPT to interact with APIs defined by developers, enhancing ChatGPT's capabilities and allowing it to perform a wide range of actions.

OpenAI, Chat Plugins [30 April 2023]

Availability

At the moment, plugin development is a Limited Alpha behind a wait-list and plugins can be shared with up to 15 fellow developers (who also have access to plugins). It's early times for sure, but if the pace of ChatGPT over the past year is any indication we are sure to have a full marketplace of both free and paid plugins within weeks of my writing this blog post.

Architecture

ChatGPT plugins are constructed with:

  1. One or more REST API endpoints implemented in any language and/or framework you prefer: Node.js, Deno, ASP .NET Controllers or Minimal API, Azure Functions, Ruby on Rails, Rust, Go, Bip Functions, AWS Lamdas, Wolfram or whatever else you fancy (COBOL diehards rejoice!). Coded APIs aren't the only option either; you can just a easily use a low-code/no-code platform like Power Apps, Power Automate, Azure APIM, Azure Logic App, Data API Builder, or OutSystems. The only requirement is that your API is accessible via the internet.
    • OpenAI provides a Python-based "TODO API" plugin to get you started with local experimentation. It seems pretty useless until you realize that by tinkering with the inputs and outputs, descriptions, even parameter names, you will begin to witness the power of ChatGPT's language processing capabilities.
  2. A manifest file (yourdomain.com/.well-known/ai-plugin.json) describing the plugin to ChatGPT. The contents and structure of this file are likely to change, but the most critical bits are:
    • Plugin name and description (for display in the ChatGPT UI)
    • Plugin description (for the model to understand at a high level what your plugin does)
    • A logo URL
    • Your contact and legal information
    • Authentication mechanism, if applicable to your API
  3. An OpenAPI specification (YAML) file. ChatGPT's language processing is a "natural" 🤣 at parsing this contract to understand what each API endpoint can do (descriptions, route methods) as well as how the input & output models are shaped (component schemas). OpenAPI isn't to be confused with OpenAI; this specification predates OpenAI and is used for all sorts of things unrelated to AI. It's a well-established standard that is used by many API providers, including Microsoft, Google, and Amazon. This alone is the most exciting/terrifying part - public REST APIs around the globe are already in a position to offer knowledge and actions to ChatGPT, merely by having an OpenAPI specification available. What's more, the same specification supports describing that most common authentication patterns, so you can offer ChatGPT users the ability to authenticate with your API and perform actions on their behalf.

Plugin Flow (Production)

sequenceDiagram
  autonumber
  actor User
  participant Client Browser
  participant OpenAI Servers
  participant Plugin Provider API
  User->>Client Browser: New ChatGPT Chat
  Client Browser->>OpenAI Servers: Register Plugin with User's ChatGPT profile
  OpenAI Servers-->>Client Browser: Inject compact plugin context 
(description & endpoints)
into hidden prompt message
  Client Browser-->>User: Render UI
  User->>Client Browser: Prompt
  activate Client Browser
  Client Browser->>OpenAI Servers: User Prompt + Plugin context
  activate OpenAI Servers
  OpenAI Servers-->>OpenAI Servers: AI Reasoning
  OpenAI Servers->>Plugin Provider API: Perform commands and/or queries
against Plugin API
  activate Plugin Provider API
  Plugin Provider API-->>Plugin Provider API: Plugin logic
  Plugin Provider API-->>OpenAI Servers: Text, Links or Markdown 
  deactivate Plugin Provider API
  OpenAI Servers-->>OpenAI Servers: Incorporate Plugin API results into response
  OpenAI Servers-->>Client Browser: Text & Meta
  Client Browser-->>User: Render response
  deactivate OpenAI Servers
  deactivate Client Browser

Plugin Flow (Local Development)

To enable local development of a plugin, ChatGPT turns the tables and calls your API from the browser. This changes the flow dramatically:

sequenceDiagram
  autonumber
  participant Plugin Provider API
  actor User
  participant Client Browser
  participant OpenAI Servers
  User->>Client Browser: New ChatGPT Chat
  Client Browser->>OpenAI Servers: Register Plugin with User's ChatGPT profile
  OpenAI Servers-->>Client Browser: Inject compact plugin context 
(description & endpoints)
into hidden prompt message
  Client Browser-->>User: Render UI
  User->>Client Browser: Prompt
  activate Client Browser
  Client Browser->>OpenAI Servers: User Prompt + Plugin context
  activate OpenAI Servers
  OpenAI Servers-->>OpenAI Servers: AI Reasoning
  OpenAI Servers-->>Client Browser: Request Plugin API call
  Client Browser->>Plugin Provider API: Perform commands and/or queries
against Plugin API
  activate Plugin Provider API
  Plugin Provider API-->>Plugin Provider API: Plugin logic
  Plugin Provider API-->>Client Browser: Text, Links or Markdown 
  deactivate Plugin Provider API
  Client Browser-->>OpenAI Servers: Relay response to Server
  OpenAI Servers-->>OpenAI Servers: Incorporate Plugin API results into response
  OpenAI Servers-->>Client Browser: Text & Meta
  Client Browser-->>User: Render response
  deactivate OpenAI Servers
  deactivate Client Browser

Local "Quickstart" Demo

OpenAI provides a sample TODO API on GitHub so that you can clone or download, and try it out with ChatGPT in a few minutes time. It's implemented in Python, so if you don't already have a local Python development environment setup you'll need to do that.

There are numerous ways of getting Python running locally. For beginners, the article below walks you through setting up Python 3 on your operating system of choice:

Python 3 Installation & Setup Guide

Running the Plugin (API) Locally

Once you have Python set up, execute the TODO API with python main.py and you'll see the server spin up and await calls from ChatGPT. The output should look something like this:

1
2
3
4
5
6
7
8
 
> python main.py

 * Serving Quart app 'main'
 * Environment: production
 * Please use an ASGI server (e.g. Hypercorn) directly in production
 * Debug mode: True
 * Running on http://0.0.0.0:5003 (CTRL + C to quit)
[2023-04-30 17:56:58 -0400] [4776] [INFO] Running on http://0.0.0.0:5003 (CTRL + C to quit)

Connect ChatGPT

Getting ChatGPT connected to the local TODO API is pretty straightforward. In a nutshell:

  1. From any ChatGPT chat, change your Model to Plugins and then use the dropdown to navigate to the Plugin Store
  2. Click Develop your own plugin
  3. Supply the URL with port number (such as http://localhost:5003)

Once connected properly, you'll see the logo of your plugin displayed in the "Plugins" dropdown.

Test Drive!

By installing your plugin in the previous step, you're providing ChatGPT with the manifest and OpenAPI specification, both of which inform the model of what your plugin API can do and how to interact.

Let's get the show on the road. I'd like to know what's on my TODO list (it should be empty at the moment), and I'll add some items afterward:

2023-05-15T224510

Okay that's neat. What happens if I try to add more than one item at a time?

2023-05-21T132555

I'm mildly impressed that this prompt did all these things, but I'm less impressed that it required 7 API calls to add 7 items. Why on earth didn't ChatGPT just send them all in one request? Because our API doesn't support that in the request model, and ChatGPT is being a good consumer of the API.

We can refactor the Python code pretty quickly to support this. We'll also need to change the request model to accept a list of items, and then we'll need to change the code to iterate over the list and add each item individually. We'll also need to change the response model to return a list of items. Let's do it...

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
 
# main.py

# ...
@app.post("/todos/<string:username>")
async def add_todo(username):
    request = await quart.request.get_json(force=True)
    if username not in _TODOS:
        _TODOS[username] = []

    _TODOS[username].append(request["todo"])
    return quart.Response(response='OK', status=200)

# ... becomes:
@app.post("/todos/<string:username>")
async def add_todos(username):
    request = await quart.request.get_json(force=True)
    if username not in _TODOS:
        _TODOS[username] = []
    
    if 'todos' in request and isinstance(request['todos'], list):
      todos = request['todos']; # extract the list of todos
      for todo in todos:
        if todo not in _TODOS[username]: # don't add duplicates
          _TODOS[username].append(todo)
      return quart.Response(response='OK', status=200)
    else
      return quart.Response(response='Bad Request, todos missing', status=400)
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
 
# openapi.yaml

# ...
components:
  schemas:
    getTodosResponse:
      # ...
    addTodoRequest:
      type: object
      required:
      - todo
      properties:
        todo:
          type: string
          description: The todo to add to the list.
          required: true

# ... becomes:

    addTodoRequest:
      type: object
      required:
      - todos
      properties:
        todos:
          type: array
          items:
            type: string
          description: The todos to add to the list.
          required: true

Let's summarize the changes we made:

  • We changed the code to iterate over the list of todos
  • We tweaked the code to disallow duplicate todos in the datastore (unplanned changed, but a good one). Also unplanned, we added a check to make sure the request contained a list of todos.
  • We changed the request model to accept a list of todos
  • We updated the specification's request model definition and description

And now we can test it out:

2023-05-21T132639

Only one request. Nice! Let's flex our new-found muscles and add support for deleting multiple todos. Our changes will be very similar:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
 
# main.py

# ...
@app.delete("/todos/<string:username>")
async def delete_todo(username):
    request = await quart.request.get_json(force=True)
    todo_idx = request["todo_idx"]
    # fail silently, it's a simple plugin
    if 0 <= todo_idx < len(_TODOS[username]):
        _TODOS[username].pop(todo_idx)
    return quart.Response(response='OK', status=200)

# ... becomes:

@app.delete("/todos/<string:username>")
async def delete_todos(username):
    request = await quart.request.get_json(force=True)
    if 'todos_idx' in request and isinstance(request['todos_idx'], list):
        todos_idx = request["todos_idx"]
        for todo_idx in sorted(todos_idx, reverse=True):    
            if 0 <= todo_idx < len(_TODOS[username]):
                _TODOS[username].pop(todo_idx)
        return quart.Response(response='OK', status=200)
    else:
        return quart.Response(response='Bad Request, todos_idx missing', status=400)
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
 
# openapi.yaml
    delete:
      operationId: deleteTodo
      summary: Delete a todo from the list
      # ...
      requestBody:
        required: true
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/deleteTodoRequest'
      # ...
# ...
    delete:
      operationId: deleteTodos
      summary: Delete todos from the list
      # ...
      requestBody:
        required: true
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/deleteTodosRequest'
# ...
components:
  schemas:
    # ...
    deleteTodoRequest:
      type: object
      required:
      - todo_idx
      properties:
        todo_idx:
          type: integer
          description: The index of the todo to delete.
          required: true
# ... becomes:
    deleteTodosRequest:
      type: object
      required:
      - todos_idx
      properties:
        todos_idx:
          type: array
          items:
            type: integer
          description: The indexes of the todos to delete.
          required: true

Let's summarize the changes we made:

  • We changed the code to iterate over the list of todo indexes
  • We changed the request model to accept a list of todo indexes
  • We updated the specification's response model definition and description

Let's clear out our TODO list all at once:

2023-05-21T132724

I'm pretty forgetful. For instance, I've been known to duplicate items on the grocery shopping list. What would be the outcome with this ChatGPT TODO plugin as it's currently implemented?

2023-05-21T132827

Ugh oh. Our plugin code disregards the duplicate entry, but as far as ChatGPT is concerned all is well up until the point that it loads again from the plugin and compares to its own internal list. We need to provide some clarity to the AI.

The solution is for the plugin to respond to duplicate entry attempts with an informative message for ChatGPT to take into its context:

2023-05-21T132921

ChatGPT understands that dentist can't be added twice, but I'm not happy with its mischaracterization "...was already on your list, so I didn't add it again." Lies! ChatGPT did indeed try to add dentist a second time and was merely prohibited.

I can resolve this by informing ChatGPT duplicates should not be added. This might however result in ChatGPT querying all TODOs before each addition, causing a lot of unnecessary chatter. Instead, I'll tweak the plugin again, this time instructing ChatGPT to update its internal list when conflicts such as this occur:

2023-05-21T132951

The Plugin Store

ChatGPT's Plugin Store already has a wide variety of offerings. In the few days it's taken me to write this blob post, I've seen a lot of new plugins pop up. At the time of this writing, there are 0 plugins available to me as a registered developer:

Wrap-Up

What I've demonstrated here is small potatoes compared to what's possible. From within ChatGPT's UI, we can now make dinner reservations, get prices on flights, and even get a list of apartments for rent in a city of our choice. And that's just the beginning.

OpenAI's ChatGPT plugin framework shows a lot of promise, and it requires us to entertain many new ideas about how AI will interact with us. I've heard people draw comparisons to previous technological advancements, but I have a hard time doing so. This is a new thing, and it's going to take some time to figure out how to use it.

Returning from the philosophical, I see this use of plugins as a way to make ChatGPT more useful in the short term. It's a way to get more out of the platform without having to wait for OpenAI to build it. And it's a way to get more out of the platform by using existing REST APIs that we're already familiar with.

There's a gap, though. The plugins are only available from within the ChatGPT UI. If you're using the OpenAI's REST API or one of the library abstractions, you're out of luck for now. Share your voice on the matter and stay tuned to updates from the OpenAI team to community feedback.

References

  1. OpenAI Platform docs site is a must-read on all matters of interacting with OpenAI's API.
    • Several more plugin examples with source code. Check here for use of OAuth and other authentication methods and giving ChatGPT more memory of conversations as well as semantic search (aka "question answering").
    • Usage Policies includes a section on plugins. It's not mentioned in the enclosed changelog, so either it's been there all along or it's a new addition without notice.
  2. OpenAI Community Forum: "Context length VS Max token VS Maximum length"