Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add LRU eviction with 1gb memory limit for PandasData #392

Merged
merged 15 commits into from
Mar 12, 2024
Merged

Add LRU eviction with 1gb memory limit for PandasData #392

Changes from 1 commit
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
e248db3
Refactor PandasData and PolygonBacktesting to support an enforce_stor…
Mar 10, 2024
93744e2
Encountered this chasing a "Not naive time" exception. Split the line…
Mar 10, 2024
83cf721
Handle empty dataframes (which occur more commonly now that get_price…
Mar 10, 2024
202d761
Take note of future warning about downcasting in dataframes.
Mar 10, 2024
6cbd2f4
Simple LRU scheme for limiting memory cache usage working.
Mar 10, 2024
05c9087
Tweak the logging message to include number of items.
Mar 10, 2024
84742f7
Merge branch 'Lumiwealth:dev' into pandas_cache
fovi-com Mar 10, 2024
59c7dfc
Make PandasData._enforce_storage_limit a static method like _set_pand…
Mar 10, 2024
31a9e52
Merge the upstream dev changes.
Mar 10, 2024
d0f2fc3
Use generator rather than list.
Mar 10, 2024
957b69c
Merged upstream and resolved conflict.
Mar 11, 2024
ab5f892
Merge upstream dev.
Mar 11, 2024
f5dadde
Merge branch 'dev' of https://github.com/fovi-llc/lumibot into pandas…
Mar 12, 2024
5b7725d
Disable pandas_data eviction by default.
Mar 12, 2024
63fb704
Correct issue link.
Mar 12, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Simple LRU scheme for limiting memory cache usage working.
  • Loading branch information
Jim White committed Mar 10, 2024
commit 6cbd2f430a660f1b75da7486681c4c95a90c583e
9 changes: 9 additions & 0 deletions lumibot/data_sources/pandas_data.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,8 @@ class PandasData(DataSourceBacktesting):
{"timestep": "minute", "representations": ["1M", "minute"]},
]

MAX_STORAGE_BYTES = 1_000_000_000 # 1 GB

def __init__(self, *args, pandas_data=None, auto_adjust=True, **kwargs):
super().__init__(*args, **kwargs)
self.name = "pandas"
Expand Down Expand Up @@ -64,6 +66,13 @@ def _get_new_pandas_data_key(data):
return new_pandas_data

def enforce_storage_limit(self, pandas_data: OrderedDict):
storage_used = sum([data.df.memory_usage().sum() for data in pandas_data.values()])
logging.info(f"{storage_used = }")
while storage_used > self.MAX_STORAGE_BYTES:
k, d = pandas_data.popitem(last=False)
mu = d.df.memory_usage().sum()
storage_used -= mu
logging.info(f"Storage limit exceeded. Evicted LRU data: {k} used {mu:,} bytes")
return

def load_data(self):
Expand Down