Document checksums

[IljaN]

With 10.0 we implemented a checksum feature which checks the file-integrity on upload and download by computing a checksum after the file-transfer is finished.

Desktop Sync-Client

Checksum Algorithm Negotiation

The client queries the server capabilities after login to decide which checksum algorithm to use.
Currently SHA1 is hard-coded in the official server-release and can`t be changed by the end-user. Note that the server additionally also supports MD5 and ADLER32 but the desktop client will always use the checksum algorithm announced in the capabilities:

GET http://localhost:8000/ocs/v1.php/cloud/capabilities?format=json

{
   "ocs":{
      "meta":{
         "status":"ok",
         "statuscode":100,
         "message":"OK",
         "totalitems":"",
         "itemsperpage":""
      },
      "data":{
         "version":{
            "major":10,
            "minor":0,
            "micro":0,
            "string":"10.0.0 beta",
            "edition":"Community"
         },
         "capabilities":{
            "core":{
               "pollinterval":60,
               "webdav-root":"remote.php/webdav"
            },
            "dav":{
               "chunking":"1.0"
            },
            "files_sharing":{
               "api_enabled":true,
               "public":{
                  "enabled":true,
                  "password":{
                     "enforced":false
                  },
                  "expire_date":{
                     "enabled":false
                  },
                  "send_mail":false,
                  "upload":true
               },
               "user":{
                  "send_mail":false
               },
               "resharing":true,
               "group_sharing":true,
               "federation":{
                  "outgoing":true,
                  "incoming":true
               }
            },
            "checksums":{
               "supportedTypes":[
                  "SHA1"
               ],
               "preferredUploadType":"SHA1"
            },
            "files":{
               "bigfilechunking":true,
               "blacklisted_files":[
                  ".htaccess"
               ],
               "undelete":true,
               "versioning":true
            }
         }
      }
   }
}

Upload

A checksum is calculated with the previously negotiated algorithm by the client and sent along with the file in a HTTP-Header.
OC-Checksum: [algorithm]:[checksum]

During file upload the server computes SHA1, MD5 and ADLER32 checksums and compares one of them to the checksum supplied by the client.

On mismatch the server returns HTTP-Statuscode 400 (Bad Request) thus signaling the client that the upload failed. The upload is discarded by the server. and the file is blacklisted by the client:

<?xml version='1.0' encoding='utf-8'?>
<d:error xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns">
  <s:exception>Sabre\DAV\Exception\BadRequest</s:exception>
  <s:message>The computed checksum does not match the one received from the
client.</s:message>
</d:error>

The client retries the upload using exponential back-off. On success (matching checksum) the computed checksums are stored by the server in oc_filecache alongside the file.

Chunked Upload

Mostly same as above. The checksum of the full file is sent with every chunk, but the servery compares the checksum of the last chunk.

Download

The server sends the checksum in a http header with the file. (same format as above).
If no checksum is found in oc_filecache (freshly mounted external storage) it is computed and stored in oc_filecache on first download. The checksum is then provided on all subsequent downloads but not on the first.

[IljaN]

@guruz Please review and feel free to edit above as needed.
@settermjd Feel free to ping me if something is unclear

[guruz]

I'm too afraid to edit as there is no version control in comments (is there?)

 "version":{
            "major":10,
            "minor":0,
            "micro":0,
            "string":"10.0.0 beta",
            "edition":"Community"
         },
         "capabilities":{
            "core":{
               "pollinterval":60,
               "webdav-root":"remote.php/webdav"
            },
            "dav":{
               "chunking":"1.0"
            },
            "files_sharing":{
               "api_enabled":true,
               "public":{
                  "enabled":true,
                  "password":{
                     "enforced":false
                  },
                  "expire_date":{
                     "enabled":false
                  },
                  "send_mail":false,
                  "upload":true
               },
               "user":{
                  "send_mail":false
               },
               "resharing":true,
               "group_sharing":true,
               "federation":{
                  "outgoing":true,
                  "incoming":true
               }
            },

Remove and replace all this with "..." ?

During file upload the server computes SHA1, MD5 and ADLER32 checksums

This sounds like it computes multiple of them?

servery

spelling

The checksum of the full file is sent with every chunk, but the servery compares the checksum of the last chunk.

You mean compares AFTER the last chunk right? When merging the chunks?

Otherwise looks good to me, I wonder if this is blog material too? ( @hodyroff )

[IljaN]

This sounds like it computes multiple of them?

@guruz It does 😎

[settermjd]

@IljaN, so, what you're after is for this to be added, as-is, to the docs, and where in the docs do you want it added?