Fix candle history warmup to work when no data has been imported #2112

jtran · 2018-04-20T00:21:56Z

What kind of change does this PR introduce? (Bug fix, feature, docs update, ...)

Bug fix.

What is the current behavior? (You can also link to an open issue here)

Steps to reproduce:

Choose a market you have not imported any data for.
Set candleSize to 1 and historySize to 10 in config.js.
Run live from CLI using a strategy that logs during update(). (No talib or tulip indicators. This has nothing to do with async.)

I expect the strategy to log 10 times immediately using recent historical candle data fetched from the exchange. But what actually happens is that it doesn't log immediately. I assume that it takes 10 minutes to log 10 times.

Note: if you've imported data for this exchange/currency/asset, the behavior is different and works in certain cases.

What is the new behavior (if this is a feature change)?

This branch makes it so that you do not need to import data in order to run a strategy live and have it start giving advice immediately. The recent trade/candle history is fetched from the exchange to warm up the strategy.

Other information:

I tested on GDAX. Not sure if that's relevant.

Exmoond · 2018-05-08T23:08:17Z

Hmm can you point on what did you change in the file besides comments ? cannot find anything different

xFFFFF · 2018-05-08T23:45:49Z

Is outside {} so outside else statement

jtran · 2018-05-09T00:20:52Z

The code is moved outside of the else block. This way, it always gets executed, not just in the else case. Without this change, idealExchangeStartTimeTS is computed in the other cases but never used.

xerviami · 2018-05-11T08:26:16Z

Dear,
I have the same issue (gdax or kraken).
AFAIK, the fix is not in the stable branch.
How to use this fix ?
Thanks a lot
Xavier

jtran · 2018-05-13T22:20:12Z

@xaviermiss, cherry-pick it on your clone of the repo. If it solves your problem, please report back. Keep commenting or posting about it, here, on related GitHub issues (or even a new one), in the forum, or in discord, in the hopes that someone merges it.

xerviami · 2018-05-14T15:39:41Z

Hi @jtran , I took the new datasticher.js file.
Unfortunately, it does not solve my problem...
Live identifies the local history but does not use it as a warmup -> advices come after hours of waiting.

jtran · 2018-05-15T06:44:21Z

@xaviermiss, this fix is for when you don't have local history. Please read the description and steps to reproduce.

xerviami · 2018-05-15T07:42:44Z

@jtran, the subject of this issue is:
"Fix candle history warmup to work even when no data has been imported"
As fas as I understand english, the "even" litteraly means that this issue occurs when "no data has been imported" or not (i.e., data has been iimported).

The given example is when no data is imported but it never explicitely narrows it to "non imported data".

Sorry but it is not clear.

I will test on my side with no import.

jtran · 2018-05-15T18:04:09Z

I updated the title to be clearer.

xerviami · 2018-05-16T07:33:45Z

I just tested with no history.
The begavior is not easy to understand.
Here are some logs:

2018-05-16 09:26:35 (INFO):             The trading method requests 1390 minutes of historic data. Checking availablity..
2018-05-16 09:26:35 (INFO):

2018-05-16 09:26:35 (INFO):             No usable local data available, trying to get as much as possible from the exchange..
2018-05-16 09:26:35 (INFO):             Preventing Gekko from requesting 1390 minutes of history.
2018-05-16 09:26:35 (DEBUG):            Fetching exchange data since 240 minutes ago
2018-05-16 09:26:37 (DEBUG):            Available exchange data:
2018-05-16 09:26:37 (DEBUG):                    from: 240 minutes ago
2018-05-16 09:26:37 (DEBUG):                    to: 168 minutes ago
2018-05-16 09:26:37 (DEBUG):            Unable to stitch datasets.
[...]
2018-05-16 09:26:40 (INFO):     Starting to watch the market: Kraken XBT/EUR
2018-05-16 09:26:40 (DEBUG):    scheduling ticks
2018-05-16 09:26:40 (DEBUG):    Requested XBT/EUR trade data from Kraken ...
2018-05-16 09:26:41 (DEBUG):    Processing 1000 new trades. From 2018-05-15 08:16:04 UTC to 2018-05-15 09:05:13 UTC. (an hour)
2018-05-16 09:26:49 (DEBUG):    Requested XBT/EUR trade data from Kraken ...
2018-05-16 09:26:49 (DEBUG):    Processing 1000 new trades. From 2018-05-16 06:36:34 UTC to 2018-05-16 07:26:52 UTC. (an hour)
2018-05-16 09:26:50 (ERROR):    Failed to get average tp from indicator.
2018-05-16 09:26:58 (DEBUG):    Requested XBT/EUR trade data from Kraken ...
2018-05-16 09:26:59 (DEBUG):    Processing 2 new trades. From 2018-05-16 07:26:56 UTC to 2018-05-16 07:26:56 UTC. (a few seconds)

Gekko:

does not want to fetch the needed history
fetches non continue history (from 240mn to 168mn)
then watches the market

Exmoond · 2018-05-16T07:45:40Z

point of this patch is to do not import any data and start trading right away.
ignore that 240 and 168

Dalekxy · 2018-05-17T09:05:39Z

I tried to reproduce your error, but it works for me without your commit.
Pair: XBT/EUR
Exchange: Kraken
History: 10
CandleSize: 1

2018-05-17 10:48:12 (INFO):     Setting up Gekko in realtime mode
2018-05-17 10:48:12 (INFO):
2018-05-17 10:48:12 (INFO):     Setting up:
2018-05-17 10:48:12 (INFO):              Trading Advisor
2018-05-17 10:48:12 (INFO):              Calculate trading advice
2018-05-17 10:48:12 (INFO):              Using the strategy: MACD
2018-05-17 10:48:12 (ERROR):    Failed to load indicator STC
2018-05-17 10:48:13 (INFO):             The trading method requests 10 minutes of historic data. Checking availablity..
2018-05-17 10:48:13 (INFO):

2018-05-17 10:48:13 (INFO):             No usable local data available, trying to get as much as possible from the exchange..
2018-05-17 10:48:13 (DEBUG):            Fetching exchange data since 10 minutes ago
2018-05-17 10:48:13 (DEBUG):            Available exchange data:
2018-05-17 10:48:13 (DEBUG):                    from: 10 minutes ago
2018-05-17 10:48:13 (DEBUG):                    to: 0 minutes ago
2018-05-17 10:48:13 (DEBUG):            Unable to stitch datasets.
2018-05-17 10:48:13 (INFO):             Not seeding locally available data to the trading method.
2018-05-17 10:48:13 (INFO):     Setting up:
2018-05-17 10:48:13 (INFO):              Paper Trader
2018-05-17 10:48:13 (INFO):              Paper trader that simulates fake trades.
2018-05-17 10:48:13 (INFO):

2018-05-17 10:48:13 (INFO):     Setting up:
2018-05-17 10:48:13 (INFO):              Performance Analyzer
2018-05-17 10:48:13 (INFO):              Analyzes performances of trades
2018-05-17 10:48:13 (INFO):

2018-05-17 10:48:13 (INFO):     Starting to watch the market: Kraken XBT/EUR
2018-05-17 10:48:13 (DEBUG):    scheduling ticks
2018-05-17 10:48:13 (DEBUG):    Requested XBT/EUR trade data from Kraken ...
2018-05-17 10:48:14 (DEBUG):    Processing 1000 new trades. From 2018-05-17 07:52:37 UTC to 2018-05-17 08:48:15 UTC. (an hour)
2018-05-17 10:48:33 (DEBUG):    Requested XBT/EUR trade data from Kraken ...
2018-05-17 10:48:34 (DEBUG):    Processing 29 new trades. From 2018-05-17 08:48:19 UTC to 2018-05-17 08:48:30 UTC. (a few seconds)
2018-05-17 10:48:53 (DEBUG):    Requested XBT/EUR trade data from Kraken ...
2018-05-17 10:48:54 (DEBUG):    Processing 1 new trades. From 2018-05-17 08:48:54 UTC to 2018-05-17 08:48:54 UTC. (a few seconds)
2018-05-17 10:49:13 (DEBUG):    Requested XBT/EUR trade data from Kraken ...
2018-05-17 10:49:14 (DEBUG):    Processing 1 new trades. From 2018-05-17 08:49:11 UTC to 2018-05-17 08:49:11 UTC. (a few seconds)
2018-05-17 10:49:14 (DEBUG):    calculated MACD properties for candle:
2018-05-17 10:49:14 (DEBUG):             short: 7004.22771910
2018-05-17 10:49:14 (DEBUG):             long: 7003.81034121
2018-05-17 10:49:14 (DEBUG):             macd: 0.41737788
2018-05-17 10:49:14 (DEBUG):             signal: 1.08818201
2018-05-17 10:49:14 (DEBUG):             macdiff: -0.67080412
2018-05-17 10:49:14 (DEBUG):    In downtrend since 1 candle(s)
2018-05-17 10:49:14 (INFO):     (PROFIT REPORT) original simulated balance:      7054.40000000 EUR
2018-05-17 10:49:14 (INFO):     (PROFIT REPORT) current simulated balance:       7088.89420000 EUR
2018-05-17 10:49:14 (INFO):     (PROFIT REPORT) simulated profit:                34.49420000 EUR (0.48897426%)

askmike

Apologies for the late reply. The psuedo code for the code you intend to change is:

pull latest window from DB
if(!window) { // there is no data }
else if(window is too old) { // no usable data we can stitch }
else { // we can use data! Note this fact by registering it in the config }

Only in the last situation do we actually want to work with local data. Which is why the statement right now only resides in the latest "else".

jtran · 2018-10-09T02:47:35Z

@askmike, the point is not to work with local data. The point is to fetch farther back in time from the exchange. As the long description already explains, this change is specifically for when you have not imported any local data. To see where firstFetchSince is used, see marketFetcher.js.

stale · 2019-01-13T06:39:10Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. If you feel this is very a important issue please reach out the maintainer of this project directly via e-mail: gekko at mvr dot me.

Fix candle history warmup to work even when no data has been imported

68877a9

dreammaker mentioned this pull request Apr 20, 2018

[GDAX] Strategy Execution Delayed or Never Starts #1689

Closed

xFFFFF mentioned this pull request Apr 20, 2018

bitfinex import, gekko live: Unable to stitch datasets. #2070

Closed

jtran changed the title ~~Fix candle history warmup to work even when no data has been imported~~ Fix candle history warmup to work when no data has been imported May 15, 2018

mmvasa added a commit to mmvasa/gekko that referenced this pull request Jun 18, 2018

Apply pull askmike#2112

28ec21d

askmike suggested changes Oct 8, 2018

View reviewed changes

stale bot added the wontfix label Jan 13, 2019

stale bot closed this Jan 20, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix candle history warmup to work when no data has been imported #2112

Fix candle history warmup to work when no data has been imported #2112

jtran commented Apr 20, 2018 •

edited

Loading

Exmoond commented May 8, 2018 •

edited

Loading

xFFFFF commented May 8, 2018 •

edited

Loading

jtran commented May 9, 2018

xerviami commented May 11, 2018

jtran commented May 13, 2018

xerviami commented May 14, 2018 •

edited

Loading

jtran commented May 15, 2018

xerviami commented May 15, 2018

jtran commented May 15, 2018

xerviami commented May 16, 2018 •

edited

Loading

Exmoond commented May 16, 2018

Dalekxy commented May 17, 2018

askmike left a comment

jtran commented Oct 9, 2018

stale bot commented Jan 13, 2019

Fix candle history warmup to work when no data has been imported #2112

Fix candle history warmup to work when no data has been imported #2112

Conversation

jtran commented Apr 20, 2018 • edited Loading

Exmoond commented May 8, 2018 • edited Loading

xFFFFF commented May 8, 2018 • edited Loading

jtran commented May 9, 2018

xerviami commented May 11, 2018

jtran commented May 13, 2018

xerviami commented May 14, 2018 • edited Loading

jtran commented May 15, 2018

xerviami commented May 15, 2018

jtran commented May 15, 2018

xerviami commented May 16, 2018 • edited Loading

Exmoond commented May 16, 2018

Dalekxy commented May 17, 2018

askmike left a comment

Choose a reason for hiding this comment

jtran commented Oct 9, 2018

stale bot commented Jan 13, 2019

jtran commented Apr 20, 2018 •

edited

Loading

Exmoond commented May 8, 2018 •

edited

Loading

xFFFFF commented May 8, 2018 •

edited

Loading

xerviami commented May 14, 2018 •

edited

Loading

xerviami commented May 16, 2018 •

edited

Loading