Move HA to SQLite; parse pde yaml; prototype PDE encryption

160-pde.md

@@ -1,4 +1,31 @@
 
+☐ Prototype a PDE running on GAE and NodeJS
+
+sequence so far:  
+1) PDE filter fires in the shard  
+2) PDE server gets the message and replies with either  
+  a) a reply to the user that needs no further processing from the bot  
+  b) a modified (preprocessed) message  
+     (turning `/r jake's health + 2d2` to `/r 33 + 2d2`)  
+  c) not interested: server handler decided not to process the message  
+3) If there's a reply or a modified message, they're passed to Sidekick handler  
+4) Sidekick handler might log the PDE content and keep tabs on it  
+  It sends the reply to the Discord (allowing for HTTP keepalive)  
+
+☐ A command to register HTTP plugins  
+
+    /r pde $yaml  # cf. https://learnxinyminutes.com/docs/yaml/
+      id: $short_unique_base36  # If ID exists then the PDE is updated
+      channel: $id  # If the user knows the Discord ID then they're likely have some kind of access to the channel
+      url: $http_url
+      pk: $ct_pk  # If recipient's public key is present then Sidekick generates a key pair and shares the PDE public key
+                  # Recepient will need that public key to open the messages and to encrypt the replies
+      zstd: 3  # Compression level
+
+→ ☐ Error message if the user is not a known patron  
+
+☐ The question of moderation and of malicious content used to set the bot up  
+
 ☑ Upload WASMs into channels  
 ☑ `fn dice_io`: Load the set of WASM plugins for the channel.  
 ☑ → implement `fn wasm_plugins`  
@@ -19,62 +46,11 @@
 ☐ → Try https://glitch.com/create  
 ☐ → Try http://heroku.com/ (https://github.com/emk/heroku-buildpack-rust)  
 ☒ Communicate with Node.js via HTTP  
-☐ → Unit test connecting to Node.js  
-☐ Move shard-handler communication to HTTP, https://github.com/ArtemGr/Sidekick/issues/195  
+☑ Move shard-handler communication to HTTP, https://github.com/ArtemGr/Sidekick/issues/195  
 ☑ → Hyper server, [fast](https://www.techempower.com/benchmarks/#section=data-r18&hw=ph&test=plaintext)  
 ☑ → Port `ShardStatusPt::dispatch`  
 ☑ Move SQLite synchronization to HTTP  
-☐ Move mobile-handler communication to HTTP  
+☑ Move mobile-handler communication to HTTP  
   We should eventually use something like the https://pub.dev/packages/firebase_messaging to get notifications, in order to save battery and traffic and RAM, but for now we can just reuse the bidirectional HTTP communication, leaving the push optimization for later  
 ☑ → [Argo tunnel](https://github.com/cloudflare/cloudflared)  
-☐ → Long polling  
-☐ A command to register HTTP plugins  
-☐ Drop the `wasm_bytes` table  
-
-# Sketching deduplication
-
-plugin 1 [ v5 function [ library functions ] ]
-plugin 2 [ function [ library functions ] ]
-plugin 3 [ function 1-10 KiB [ library functions 1-10 MiB ] ]
-plugin 100th
-plugin 200th
-plugin 400th
-...
-
-Deduplication, because the same wasm can be uploaded to different channels/servers
-could] Usage statistics, to differentiate between the oft used and the unused WASMs
-could] Compress the wasm files that are used less often in order to reduce the CPU load
-could] Given the potentially large size, consider using a separate database for the WASMs.
-  Downside is having to setup in the firewall a second replication port.
-
-A layer of dictionary compression should be designed (with implementation adjourned for later)
-in hopes of uncluttering the database from the repeated WASM parts (library functions)
-and for reuse in similar scenarios.
-
-Expected queries:
-UPSERT INTO wasm (channel, server, user, name, len, hash)  -- Mind wasm_bytes removal!
-UPSERT INTO wasm_bytes (len, hash, body)
-SELECT len, hash FROM wasm WHERE channel = ?
-SELECT name FROM wasm WHERE channel = ?
-When a WASM is removed *or updated*, we should check if any links remain to the old WASM:
-SELECT channel FROM wasm WHERE len = ? AND hash = ?  -- Index on `len`?
-
-# Running WASM
-
-## Node.js
-
-Going to use Node in order not to build V8 (we're gettting it with Node instead).
-This should be more compatible with how the kind of enviroment most Rust-JS interoperations expect.
-
-Windows version installed from https://nodejs.org/en/download/current/, using the latest version in order to get the fresh V8 with it (see the table at https://nodejs.org/en/download/releases/). Allowed Node to install the extra C++ development tools.
-
-Embedding Node in Rust is, in turn, [a problem](https://github.com/nodejs/node/issues/24028), since the shared library is not released by default. So we're going to run Node as an independent process instead.
-
-The stdin / stdout pipe is optimized in Linux, AFAIK. Should work as well as shared memory RPC.
-
-In order to reduce the Rust-Node-WASM overhead we're going to send the entire bot command to WASM, getting back a fully processed result (e.g. with dice notation expressions replaced by rolls).
-This processed result should be able to mark the replaced portions as final, preventing further tempering (standard dice natation, variable substitution) on the Rust side.
-
-Should keep in mind that WASM processing ought to play well with variable substitution.
-
-P.S. There is "node.dll" in Upwork, we can probably just use it on Windows!
+☑ → Long polling  

191-mobile.md

@@ -1,3 +1,6 @@
+
+☐ Document the HTTP API and share it with Fraks
+
 ☑ Receive text input from mobile app  
 ☑ “/r bind app” on any private channel → generate a temporary (lm) bind key, return the public part  
 → ☐ If the user is not a patron, suggest getting a patron key  

221-pp.md

@@ -1,4 +1,25 @@
-https://github.com/ArtemGr/Sidekick/issues/221
+https://github.com/ArtemGr/Sidekick/issues/221  
+
+# todo list
+
+☐ Make a notifications branch  
+☐ Investigate http://www.cstr.ed.ac.uk/projects/festival/  
+☐ Investigate https://cloud.google.com/text-to-speech  
+☐ Generate a feed for https://github.com/ca333/komodoDEX/issues/728  
+→ ☐ EdDSA signatures  
+
+# outdated
+
+☐ Whitelisting; “Knowing that a user is playing Overwatch or listening to a certain artist on Spotify right now is innocuous. But, we believe the ability to track many, many users over a long period of time and profile their behaviors (not just what they play or listen to, but when they're online or offline, or at their computer vs. on mobile, or any random info they put in their custom status) is a potential issue”; “When you apply, we will ask you what features you are using the data for. We will approve or deny” - https://github.com/discord/discord-api-docs/pull/1307#issuecomment-576969633; https://blog.discordapp.com/the-future-of-bots-on-discord-4e6e050ab52e; https://support.discordapp.com/hc/en-us/articles/360040720412; “and like it or not this year there will be multiple changes that will be considered breaking changes for bots” - https://github.com/discord/discord-api-docs/pull/1307#issuecomment-581716513  
+→ ☐ ID verification at https://discordapp.com/developers/applications/209040908673482753/verification ⇒ Tried with a Russian ID, Discord sayd “We are unable to verify your identity because one or more of the documents you provided are unverified. Please try again” ⇒ With the international ID it worked… Now I need to describe the bot… https://hackmd.io/@artemciy/BJ29oy2DU ⇒ There was an error submitting all this. I've sent a support request, https://support.discordapp.com/hc/en-us/requests/7089456, https://mail.google.com/mail/u/0/#all/FMfcgxwHMjpbHlHzgfmpSRGxXcPQNKTq, https://www.patreon.com/posts/35797646 …  
+
+☐ Consider unsubscribing from the status events and using the mobile application to track the status instead  
+  This might help with solving the status abuse (case in point - MV keeping the pair programming signal for long), will allow for recipient selection (like when I want to talk with yurii but not with MV), etc etc  
+
+☐ Experimenting with Discord statuses shows that a certain number of people will “abuse” the system by setting the status and going away. This makes the Discord-based status largely unusable, just a few apples can radically change the way the LFG is experienced. The (mobile?) user interface should be protected from it, only allowing for a limited time status and hiding the AFK users.  
+☐ Similarly, we should take steps to discourage people from trying to book each other's time.  
+  cf. https://discordapp.com/channels/@me/654706300076425281/697873438097801256  
+  It's common to see messages along the lines of “If you are free later on today, could you lend me a hand?” This is completely backwards. Instead of using the non-verbal-like clues to catch the right moment the person is trying to load the rest of our day with another task for us to juggle.
 
 ☐ Public HTTP API that gives external tools access to watched status updates  
   Should be a first-class citizen, as the tools have extended information about when it's a good time to be notified of a watched status and how best to render notifications