Summarise long chats before sending to server?

[wzol]

To be honest I stole this idea fro another GUI, but I like it. So it does a summarise on the previous conversation, so this way there is a big API saving when you keep on doing a long conversation. I have no idea how it handles coding, or games, so I just share it, maybe it is good for inspiration.

[awiggins12]

I certainly need some way to handle the max length situations.

I'm thinking I might add a "pruning" setting when the output reaches a certain token length the chat would automatically summarize the oldest X number of messages/responses and place them back at the top. X would be configurable.

With the way I have the code right now it would display in the UI the summarized versions and save them to the filesystem when they are replaced, which I'm not so keen on. Maybe I can make that configurable as well.

In any case, I'll need to implement the methods that calculate token length.

Unfortunately, my real life job as had me working late so I've had little time to put into this.

[wzol]

Maybe simplifying it a bit would help in time? The idea at the other GUI for token length was 1 token / 4 chars. No idea, where this comes from, but might worth a check.

That prune idea seems fine to me. The saving might be easier with a new, modified version of the file, so 2303100610-ouput then 2303100610-output-2303100614-prune1 and 2303100610-output-2303100620-prune2 so you'd create and continue the output in a new file everytime a prune happened.

[msampson3d]

I have submitted a pull request to allow the user to trim the number of messages sent to the api, allowing them to continue the conversation even after it extends beyond the limit. This could help in the meantime before more automatic measures are put in place.

[awiggins12]

Merged it, thanks for doing that.

[awiggins12]

I just added the options for models and now the max context length displayed will be wrong. Oh well.

[msampson3d]

Eh, easy to patch if needed, plus the long term fix will be to tokenize, count and trim for the user eventually anyway.