Skip to content

Count LLM tokens in Azure AI Foundry models #1067

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
waldekmastykarz opened this issue Mar 19, 2025 · 3 comments · May be fixed by #1167
Open

Count LLM tokens in Azure AI Foundry models #1067

waldekmastykarz opened this issue Mar 19, 2025 · 3 comments · May be fixed by #1167
Assignees
@waldekmastykarz
Labels
enhancement New feature or request work in progress

Comments

@waldekmastykarz
Copy link
Collaborator

Build a new plugin that allows customers to count the number of LLM tokens per recorded session. For each recorded request, check if it has a response, and if it's an LLM response that contains tokens information (sent and received). Calculate the total of tokens per type (prompt, completion, total). Group total per model. Show totals per type and a grand total for the session.

Initially, the plugin will support Azure AI Foundry URLs for groupings per model.

Originally proposed by @tomorgan
@waldekmastykarz waldekmastykarz added enhancement New feature or request needs peer review Issue needs review from other team members labels Mar 19, 2025
@waldekmastykarz
Copy link
Collaborator Author

@dotnet/dev-proxy-maintainers any feedback?

@waldekmastykarz waldekmastykarz changed the title Count LLM tokens Count LLM tokens in Azure AI Foundry models Mar 20, 2025
@waldekmastykarz waldekmastykarz removed the needs peer review Issue needs review from other team members label Mar 20, 2025
@waldekmastykarz
Copy link
Collaborator Author

When we get to work with other AI platforms (Ollama, OAI, etc), we'll implement them in separate plugins.

@waldekmastykarz waldekmastykarz added the help wanted We'd appreciate your help label Apr 2, 2025
@waldekmastykarz waldekmastykarz self-assigned this May 6, 2025
@waldekmastykarz waldekmastykarz added work in progress and removed help wanted We'd appreciate your help labels May 6, 2025
@waldekmastykarz
Copy link
Collaborator Author

Researching integration with OpenLIT which has ready-to-use dashboards for LLM observability including pricing

waldekmastykarz added a commit to waldekmastykarz/dev-proxy that referenced this issue May 8, 2025
@waldekmastykarz
Adds logging LLM usage information using OpenTelemetry. Closes dotnet…
e5f9d93 
…#1067
waldekmastykarz added a commit to waldekmastykarz/dev-proxy that referenced this issue May 8, 2025
@waldekmastykarz
Adds logging LLM usage information using OpenTelemetry. Closes dotnet…
6fc992d 
…#1067
waldekmastykarz added a commit to waldekmastykarz/dev-proxy that referenced this issue May 8, 2025
@waldekmastykarz
Adds logging LLM usage information using OpenTelemetry. Closes dotnet…
8cc5ca4 
…#1067
@waldekmastykarz waldekmastykarz linked a pull request May 8, 2025 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@waldekmastykarz waldekmastykarz

Labels
enhancement New feature or request work in progress
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Adds logging LLM usage information using OpenTelemetry. Closes #1067 waldekmastykarz/dev-proxy
1 participant
@waldekmastykarz