Refactor context benchmark #249
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jakethekoenig
approved these changes
Nov 6, 2023
| @@ -48,6 +49,7 @@ def _get_all_features( | |||
| abs_path.is_dir() | |||
| or not is_file_text_encoded(abs_path) | |||
| or abs_path in ignore_files | |||
| or os.path.getsize(abs_path) > max_chars | |||
There was a problem hiding this comment.
Curious why you added this. Was there some file in the repo you were trying to exclude?
There was a problem hiding this comment.
Ya, there's a .ipynb that's massive in one of the repos I was working with, and a huge .json file in another.
| 2. If an 'Expected Edits' list is provided to the code-selection LLM, it *must* include the lines which are expected to be edited. This is reflected in the scores below as 'Recall'. \ | ||
| 3. To also identify relevant context to the query, such as the type-definitions of variables which will be edited, or functions which would be directly affected by the edits. \ | ||
| 4. To NOT select irrelevant files or lines of code. \ | ||
| 5. It's critical respond to this with a JSON-parsable list of strings (one for each prompt). \ |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I've tweaked the prompt somewhat and setup this workflow:
(not new) Generate transcripts with
pytest -s tests/benchmarks/git_log_to_transcripts.py --benchmark. This will addedited_features(directly from diff) andselected_features(output of our LLMSelector). The selected_features don't reliably include all the edits so they're kept separate.Evaluate the LLMSelector on all benchmarks with
pytest -s tests/benchmarks/context_benchmark.py --benchmark. This will cycle through all the benchmarks found and calculate recall and precision for 3 cases:UseExpected=False, LLM=False: Just the embeddings-based preselector
UseExpected=False, LLM=True: The LLM selector with the prompt and no expected edits
UseExpected=True, LLM=True: The LLM selector, given known
expected_editsfrom benchmarkResults are hit or miss - roughly 1/3 of the time it's perfect (recall=1.0), 1/3 it's useless (recall=0), and the rest in between.
I added another script,
evolve_llm_feature_selector.pywhich I played around with, hasn't been super useful yet but I think is worth hanging on to.