Add telemetry to diagnose intermittent NES model switching#308755
Draft
Add telemetry to diagnose intermittent NES model switching#308755
Conversation
- Add 'nesModelChanged' event that fires when the selected NES model changes mid-session, capturing source, model list, and token state - Include model source in provideinlineedit's modelConfig JSON via new selectedModel() method that returns config+source atomically - This helps diagnose whether switches are caused by /models endpoint changes, experiment flips, or fallback to hardcoded defaults
Contributor
There was a problem hiding this comment.
Pull request overview
Adds diagnostic telemetry around Next Edit Suggestions (NES) model selection to help identify the cause of rare mid-session model switching, and updates the inline edits model service API to return model + source atomically.
Changes:
- Add a new
nesModelChangedMSFT telemetry event emitted when the selected NES model name changes. - Include a
sourcefield in the serializedmodelConfigtelemetry for eachprovideInlineEditflow. - Replace
selectedModelConfiguration()withselectedModel()returning{ config, source }to avoid config/source races.
Show a summary per file
| File | Description |
|---|---|
| extensions/copilot/src/platform/inlineEdits/node/inlineEditsModelService.ts | Adds nesModelChanged telemetry via an autorun and introduces the new selectedModel() implementation. |
| extensions/copilot/src/platform/inlineEdits/common/inlineEditsModelService.ts | Updates the service interface to selectedModel(): SelectedModel. |
| extensions/copilot/src/extension/xtab/node/xtabProvider.ts | Plumbs model source into modelConfig telemetry JSON and switches to selectedModel(). |
| extensions/copilot/src/extension/xtab/test/node/xtabProvider.spec.ts | Updates mocks and tests to use selectedModel() instead of selectedModelConfiguration(). |
Copilot's findings
- Files reviewed: 4/4 changed files
- Comments generated: 2
extensions/copilot/src/platform/inlineEdits/node/inlineEditsModelService.ts
Outdated
Show resolved
Hide resolved
…ranch - Move hasCopilotToken/isFreeUser to measurements bag as numbers (1/0) instead of string properties, matching the isMeasurement GDPR annotation - Add modelSource to forceUseDefaultModel branch return value and update determineModelConfiguration return type to always include modelSource so source is never undefined in the serialized modelConfig JSON
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Investigation of
provideinlineedittelemetry revealed that ~0.04% of sessions with successful NES fetches experience intermittent model switching — the NES model bounces between two different models (e.g.,nes-callisto-003→nes-callisto→nes-callisto-003) mid-session.We couldn't determine the root cause from existing telemetry because the
modelConfigproperty didn't include how the model was selected (fetched from/models, experiment config, hardcoded default, etc.).Changes
1.
nesModelChangedtelemetry eventFires whenever
_currentModelObsproduces a different model than the previous one. Captures:previousModel/newModel— model names before and afternewModelSource— how the model was picked (fetched,expConfig,expDefaultConfig,localConfig,hardCodedDefault)modelListNames/modelListSources— full snapshot of the aggregated model listfetchedNesModels— raw NES models from latest/modelsresponsehasCopilotToken/isFreeUser— token state at switch time2.
sourcefield inmodelConfigJSON on everyprovideInlineEditeventThe existing
modelConfigtelemetry property now includes asourcefield, so every NES request self-documents how its model was chosen. Query via:3.
selectedModel()API onIInlineEditsModelServiceReplaces the old
selectedModelConfiguration()withselectedModel()returning{ config, source }atomically from a single observable read — avoids a race between config and source.Possible causes under investigation
/modelsendpoint returning different NES models across token refreshes/modelsdata falling through to hardcoded defaultsThis telemetry will confirm which.