add ChatCompletionStream Usage return

[nasa1024]

add ChatCompletionStream Usage return

[codecov]

Codecov Report

❗ No coverage uploaded for pull request base (master@8e3a046). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master     #215   +/-   ##
=========================================
  Coverage          ?   71.94%           
=========================================
  Files             ?       21           
  Lines             ?      581           
  Branches          ?        0           
=========================================
  Hits              ?      418           
  Misses            ?      124           
  Partials          ?       39           

Impacted Files	Coverage Δ
chat_stream.go	88.00% <ø> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[sashabaranov]

Thank you for the PR!

[collinvandyck]

Thank you this is what I was looking for!

[collinvandyck]

I'm not actually seeing any usage data come back in the streaming response. I added a roundtripper to log request/response data and it looks like this (formatting mine):

POST https://api.openai.com/v1/chat/completions
HTTP/2.0 200 OK
data: {
  "id": "chatcmpl-70smbYYzwcwpb19BkzLb3NFlvkIsP",
  "object": "chat.completion.chunk",
  "created": 1680444941,
  "model": "gpt-3.5-turbo-0301",
  "choices": [
    {
      "delta": {
        "role": "assistant"
      },
      "index": 0,
      "finish_reason": null
    }
  ]
}
data: {
  "id": "chatcmpl-70smbYYzwcwpb19BkzLb3NFlvkIsP",
  "object": "chat.completion.chunk",
  "created": 1680444941,
  "model": "gpt-3.5-turbo-0301",
  "choices": [
    {
      "delta": {
        "content": "How"
      },
      "index": 0,
      "finish_reason": null
    }
  ]
}

Is there something I need to do to signal to openai that usage data should be returned?

[sashabaranov]

@collinvandyck please upgrade to 1.5.8 go get -u github.com/sashabaranov/go-openai@v1.5.8

[collinvandyck]

Hey @sashabaranov ! I'm seeing this behavior on 1.5.8, go.mod:

github.com/sashabaranov/go-openai v1.5.8-0.20230401160622-b542086cbb22

After giving it a spin I saw that usage was always blank in the resp returned by Recv, so that's why I added the request/response logging to verify it wasn't actually coming back in the http response.

I'm kicking off the chat completion stream like this:

req := openai.ChatCompletionRequest{
    Model:    c.model,
    Messages: messages,
    Stream:   true,
}
resp, err := c.openai.CreateChatCompletionStream(ctx, req)
if err != nil {
    return nil, fmt.Errorf("stream: %w", err)
}

[collinvandyck]

Aha, I found this:

https://github.com/openai/openai-cookbook/blob/main/examples/How_to_stream_completions.ipynb

[nasa1024]

too embarrassed
@collinvandyck yes, /chat/completions when using stream=True will not return Usage, i will fix it

[collinvandyck]

It would be neat for go-openai to include a tokenizer to do this, but it seems like a pretty big lift. It looks like most of the folks that are doing this are calling out to a rust tokenizer since there's not an official go tokenizer. Not suggesting that you should do this, but it was surprising to me that there wasn't any official way to do this using openai's libraries in Go.

[nasa1024]

It would be neat for go-openai to include a tokenizer to do this, but it seems like a pretty big lift. It looks like most of the folks that are doing this are calling out to a rust tokenizer since there's not an official go tokenizer. Not suggesting that you should do this, but it was surprising to me that there wasn't any official way to do this using openai's libraries in Go.

sir, see 223