-
Notifications
You must be signed in to change notification settings - Fork 230
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Refactor context benchmark #249
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The prompt mutator looks like a lot of fun. Hopefully we get use out of it later. In my own testing it looks like using an llm improves recall a fair bit.
@@ -48,6 +49,7 @@ def _get_all_features( | |||
abs_path.is_dir() | |||
or not is_file_text_encoded(abs_path) | |||
or abs_path in ignore_files | |||
or os.path.getsize(abs_path) > max_chars |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Curious why you added this. Was there some file in the repo you were trying to exclude?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ya, there's a .ipynb that's massive in one of the repos I was working with, and a huge .json file in another.
2. If an 'Expected Edits' list is provided to the code-selection LLM, it *must* include the lines which are expected to be edited. This is reflected in the scores below as 'Recall'. \ | ||
3. To also identify relevant context to the query, such as the type-definitions of variables which will be edited, or functions which would be directly affected by the edits. \ | ||
4. To NOT select irrelevant files or lines of code. \ | ||
5. It's critical respond to this with a JSON-parsable list of strings (one for each prompt). \ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Cool!
I've tweaked the prompt somewhat and setup this workflow:
(not new) Generate transcripts with
pytest -s tests/benchmarks/git_log_to_transcripts.py --benchmark
. This will addedited_features
(directly from diff) andselected_features
(output of our LLMSelector). The selected_features don't reliably include all the edits so they're kept separate.Evaluate the LLMSelector on all benchmarks with
pytest -s tests/benchmarks/context_benchmark.py --benchmark
. This will cycle through all the benchmarks found and calculate recall and precision for 3 cases:UseExpected=False, LLM=False: Just the embeddings-based preselector
UseExpected=False, LLM=True: The LLM selector with the prompt and no expected edits
UseExpected=True, LLM=True: The LLM selector, given known
expected_edits
from benchmarkResults are hit or miss - roughly 1/3 of the time it's perfect (recall=1.0), 1/3 it's useless (recall=0), and the rest in between.
I added another script,
evolve_llm_feature_selector.py
which I played around with, hasn't been super useful yet but I think is worth hanging on to.