fix: batch or partition DB access and commits for user streak updation

[melvinebenezer]

fixes #3293

• Partitioned the access to list of users and updated the same
• Added more debug
• Tested on a small batch of Fake users

What is the best USER_STREAK_BATCH_SIZE ?

[andreaskoepf]

I think the current implementation could be implemented with two update statements (which would be more efficient than the loops):

# reset streak_days to 0 for all users with more than one day inactivity
update "user" set streak_days=0, streak_last_day_date=current_timestamp::date 
where (current_timestamp-last_activity_date) > interval '1 day' or streak_days is null or last_activity_date is null;

# increment streak_days when user completed tasks on consecutive days 
update "user" set streak_days=streak_days+1, streak_last_day_date=current_timestamp::date 
where current_timestamp::date != streak_last_day_date::date;

I think there was a logic error in the current code which causes the streak_days count to become too low after a some time. After incrementing it sets the streak_last_day_date = current_time .. but in this case it is clear that (current_time-streak_last_day_date) > 24h .. that means it takes >24h before each streak increment .. the effect depends on the update interval .. but let's say it is 4 hours than in the worst case it could mean a streak is incremented only all 28h ..

IMO one big problem with the current logic is: It always updates the streak_last_day_date of all users .. especially of all inactive users... maybe the whole streak-update mechanisms should be redesigned?

[melvinebenezer]

To summarise the changes

1. Update the user streak when there is user activity
2. Periodic celery task to reset streak info in case of inactivity

[andreaskoepf]

Looks good now to me!