Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add LRU eviction with 1gb memory limit for PandasData #392

Merged
merged 15 commits into from
Mar 12, 2024
Merged

Add LRU eviction with 1gb memory limit for PandasData #392

Changes from 1 commit
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
e248db3
Refactor PandasData and PolygonBacktesting to support an enforce_stor…
Mar 10, 2024
93744e2
Encountered this chasing a "Not naive time" exception. Split the line…
Mar 10, 2024
83cf721
Handle empty dataframes (which occur more commonly now that get_price…
Mar 10, 2024
202d761
Take note of future warning about downcasting in dataframes.
Mar 10, 2024
6cbd2f4
Simple LRU scheme for limiting memory cache usage working.
Mar 10, 2024
05c9087
Tweak the logging message to include number of items.
Mar 10, 2024
84742f7
Merge branch 'Lumiwealth:dev' into pandas_cache
fovi-com Mar 10, 2024
59c7dfc
Make PandasData._enforce_storage_limit a static method like _set_pand…
Mar 10, 2024
31a9e52
Merge the upstream dev changes.
Mar 10, 2024
d0f2fc3
Use generator rather than list.
Mar 10, 2024
957b69c
Merged upstream and resolved conflict.
Mar 11, 2024
ab5f892
Merge upstream dev.
Mar 11, 2024
f5dadde
Merge branch 'dev' of https://github.com/fovi-llc/lumibot into pandas…
Mar 12, 2024
5b7725d
Disable pandas_data eviction by default.
Mar 12, 2024
63fb704
Correct issue link.
Mar 12, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Take note of future warning about downcasting in dataframes.
  • Loading branch information
Jim White committed Mar 10, 2024
commit 202d76165d08c3a4a1e460a6a00f55693c95e4ea
5 changes: 5 additions & 0 deletions lumibot/entities/data.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -268,6 +268,11 @@ def trim_data(self, df, date_start, date_end, trading_hours_start, trading_hours
)
return df

# ./lumibot/build/__editable__.lumibot-3.1.14-py3-none-any/lumibot/entities/data.py:280:
# FutureWarning: Downcasting object dtype arrays on .fillna, .ffill, .bfill is deprecated and will change in a future version.
# Call result.infer_objects(copy=False) instead.
# To opt-in to the future behavior, set `pd.set_option('future.no_silent_downcasting', True)`
Comment on lines +271 to +274
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

category Code Design Improvements priority 7

The warning comment about the deprecation of downcasting object dtype arrays on .fillna, .ffill, .bfill is important and should be addressed. The code should be updated to use the recommended method to avoid future issues when the behavior changes in a future version of pandas.


def repair_times_and_fill(self, idx):
# Trim the global index so that it is within the local data.
idx = idx[(idx >= self.datetime_start) & (idx <= self.datetime_end)]
Expand Down