Consistent handling of last modified dates when copying/moving files

[despens]

Is your feature request related to a problem? Please describe.

Nextcloud handles last-modified metadata inconsistently when copying and moving files and folders:

Activity	Source	Destination	last modified preserved? v15	last modified preserved? v16	last modified preserved? v17	last modified preserved? v18	last modified preserved? v19
file upload > 10 MB via browser	Desktop	Primary Storage	✔️	✔️	✔️		✔️
file upload < 10 MB via browser	Desktop	Primary Storage	⛔	⛔	✔️		✔️
file upload via browser	Desktop	External Storage	⛔	⛔			⛔
sync files with Desktop client	Desktop	Primary Storage	✔️	✔️	✔️		✔️
upload files via WebDAV (@codiflow)	Desktop	Primary Storage	⛔		⛔	⛔	⛔
download files via WebDAV	Primary Storage	Desktop	✔️				⛔
upload files via rclone	Desktop	Primary Storage	✔️	✔️			✔️
upload files via rclone	Desktop	External Storage	⛔	⛔
moving files and folders within nextcloud	Primary Storage	Primary Storage	✔️	✔️	✔️		✔️
moving files and folders within nextcloud	Primary Storage	External Storage	⛔	⛔			⛔
moving files and folders within nextcloud	External Storage	Primary Storage	⛔	⛔			⛔
copy files and folders within nextcloud	Any Storage	Any Storage	⛔	⛔	⛔		⛔
copying files via cloud federation	remote nextcloud	local nextcloud	⛔				⛔
upload via file drop	Desktop	primary storage	⛔		⛔		⛔

(Primary storage refers to object storage directly managed by nextcloud, External Storage refers to externally mounted object storage.)

Describe the solution you'd like

Last-modification metadata should always be preserved, no matter where files and folders are coming from or moving to.

Describe alternatives you've considered

This should just work 😉

Additional context

For my use case, the last modification date of files and folders is critical for comparing different versions of the same item at different locations. It also is important when moving files from active usage into an archival storage (nearline bucket mounted externally).

(related #15177, #14982)

[codiflow]

I can add that uploading files into nextcloud by using WebDAV also doesn't preserve the original timestamps.

[despens]

Thx @codiflow, added info to the table, please check if this is correct. Did you try with several WebDAV clients?

[codiflow]

Thx @codiflow, added info to the table, please check if this is correct. Did you try with several WebDAV clients?

I use Krusader on Manjaro Linux and Windows Explorer on Windows 8.1 Pro. With both the timestamp will be set to now on upload. It also doesn't matter if the file is smaller or larger than 10 MB.

[CherAlban]

I can confirm the <>10MB behavior with Firefox 66.0.3 (64-bit).
Using WebDav in BeyondCompare the timestamp always gets set to now on upload. (which is very annoying for my use case)

[despens]

Using WebDAV via Gnome Shell, can confirm that last-modified dates are not preserved when moving items to nextcloud.

[despens]

Updated table again, found out that copying data from nextcloud to local computer via WebDAV preserves last modified date.

[codiflow]

I got a workaround for those who don't want to wait until this gets fixed:

1. Create a "sync" folder in Nextcloud
2. Install nextcloud client on your pc and sync this "sync" folder with to your pc
3. Move files whose timestamps should get preserved into this folder the nextcloud client is watching for changes on. Wait until your files got synced to the nextcloud server.
4. Go to the nextcloud WebUI and move the files from "sync" folder (with right timestamp) to the right destination

It's somehow complicated but prevents me from the need to sync the entire nextcloud with my PC.

[despens]

Tested with rclone and added to table.

[despens]

Tested with nextcloud 16.0.1 and added to table.

[despens]

Updated the table with some 17.0.2 tests. Great that files uploaded via browser now consistently keep their last modified info. Why is it different when using file drop though?

[wrightsinreston]

Thanks for documenting this issue. I do not use a browser or WebDAV. Rather, I connect Win-10, Win-7, and Ubuntu clients to a NC v17 server running on a Raspberry Pi using the NC client apps. All the clients use the v2.6.3 NC client app appropriate for their OS architecture. When I change a synchronized file on any client, the NC server correctly synchronizes the updated content across all hosts as expected and the NC Web Admin page and Windows hosts all report the correct last-modified time. But the Ubuntu host reports the time at which the server synchronized it, which will differ for each Ubuntu client. This really confuses some java apps I've written which rely on last-modified time to make decisions. As you have noted in your write-up: "This should just work." And, at least, it should work the same way on all platforms. I do not consider this to be an Enhancement request and believe it should be registered as a bug.

[wrightsinreston]

Here is some further information for those encountering this issue. I believe this offers a viable work-around even if the underlying issue appears to remain.

The issue appears to manifest itself only when the following three conditions are met: (1) The Nextcloud client is running on a Linux box, (2) the files in question are NOT owned by the UID of the running Nextcloud client process. (Note: Even if file permissions allow read/write, the file MUST be owned by the UID of the Nextcloud client process,) and (3) the disk volume on which the files in question reside is mounted under a UID different from that of the file owner. If all three of those criteria are met, Nextcloud will be unable to set the Last-Modified time and the time-stamp will reflect the time at which the file was synchronized.

If the disk volume is mounted under the same UID as the running Nextcloud client process (even if it resides on a different host PC) or if it is mounted as a CIFS volume, then the issue vanishes and Nextcloud can successfully set the time-stamp as expected.

This appears to be unique to Linux hosts. Windows hosts do not appear to manifest this issue regardless of file ownership or file system.

[Kuphi]

@despens can you update your table for nextcloud 18 and 19? Thank you! 🥇

[despens]

I just updated one of my nextcloud instances to version 19 and updated the above table.

As @MorrisJobke stated in #20862 (comment) the intended behavior is to display the last-modified time. But that information is easily lost or never makes it into nextcloud, depending on what kind of storage or upload mechanism is used. In many cases, last-modified time will hold uploaded time.

This issue is blocking the use of different storage providers for hot and cold storage: Hot storage are files currently in use and need fast access. This can be local disk or local block storage. Cold storage is for files that are not in active use anymore and can go into an archive, which could be on much cheaper storage offered by a specialized provider. However, when files are moved to cold storage they lose their last-modified dates and appear as recently modified.

I'm looking for a way to help nextcloud at least support moving files from one storage to another while retaining the last-modified date of each file, to enable the hot/cold storage use-case. What's the best way of doing that?

[ltguillaume]

The current Android client (currently running v3.12.1 RC1 F-Droid, with Nextcloud v19.0.0) seems to exhibit totally random behavior with regard to the modification date: using the Auto upload feature, some files are uploading while preserving the modification date, while others aren't. No distinction can be made in this between <10MB and >10MB.

[bodlin]

Hi folks,
I just tested copy via cloud federation between two freshly installed Nextcloud servers (both v19.0.0). What I did via web UIs:

1. user1@nc1 shares SrcDir for user2@nc2
2. user2@nc2 accepts invitation for the share
3. user2@nc2 views SrcDir@nc1 with all files and all timestamps correct
4. user2@nc2 performs copy of all files from SrcDir@nc1 to DestDir@nc2
5. user2@nc2 views DestDir@nc2 - timestamps of all files are less than minute ago

For photos is timestamp crucial thing.

[despens]

@bodlin, I added your result to the table above.

[codiflow]

Is there a reason why theres no NC v18 column in the table? I can confirm the issue still exists using WebDAV with NC v18.

[despens]

@codiflow The only reason is that I upgraded from 17 to 19 directly on my instances 😉. I'll add you info.

[da-anda]

I used the official Android app to sync my pictures to my freshly installed NC20 instance, and can confirm that files > 1M loose their original creation date and instead get the date of the upload/sync. This is super annoying and IMO a no-go. Timestamps should be preserved no matter what. NC is not usable for me this way.

I also do not understand why NC server is unable to extract the file creation time and add this info into the DB on it's own, but this is a different story.

edit: to me this is a clear bug and not an enhancement like this ticket is currently labeled

[maxxer]

Under what circumstances exactly @maxxer ?

I wish I knew! Unfortunately I haven't been able to figure out when it works.

I'm making more tests as I'd like to make it work, but every way is a dead end. My backend is FTP, so I mounted the storage with curlftpfs, giving rights to the user Nextcloud is running with. From command line I can do everything, I can create files and directories, I tried running a simple php put_file_contents() and the file was created correctly with date, time and size.
I add the directory in Nextcloud as external storage (local), I can create directories but they have timestamp 0, uploading a file (both via webdav and web) fails because the backend returns size 0 to Nextcloud.

[maxxer]

(Just for the record, I ended setting up everything with rclone, storage and encryption. I mount the rclone remote locally in a Nextcloud accessible directory, and then add an external storage (type local), unencrypted, in Nextcloud)

[satyamkapoor]

@maxxer can I preserve dates if I use rclone to move files from local SSD to S3? Thinking to move to WASABI as primary storage for nextcloud

[maxxer]

@maxxer can I preserve dates if I use rclone to move files from local SSD to S3? Thinking to move to WASABI as primary storage for nextcloud

I am using an S3 clone, poorer than Wasabi. But I think the answer to your question is no. You can make a test, Wasabi offers a 1 month trial, IIRC

[pgy866]

The date I uploaded the file has been changed, I hope to add an option to retain the original file attributes, or not to retain

[PVince81]

I can add that uploading files into nextcloud by using WebDAV also doesn't preserve the original timestamps.

not technically possible because the Webdav protocol doesn't include the local modification time

Nextcloud has a custom header X-OC-Mtime that the desktop client and the web UI sets to preserve it.
Third party clients would need to be modified to also pass this...

Regarding external storage not preserving mtime, this sounds like a bug.
Some external storages don't support setting a mtime (like S3), but oc_filecache still has a way to track it in the "mtime" column while the storage's own mtime is tracked in "storage_mtime".

[PVince81]

it would be good to rebuild the top post table now that NC 21 is EOL and retest with recent versions.
and add a "notes" column.

then in general, for the notes:

• if upload to external storage not working: likely bug with ext storage
• moving between ext and primary storage: likely bug with ext storage
• copying files: the local OS doesn't preserve mtime (unless using "cp -a"), so not sure if a copy should keep mtime by design, needs to be discussed
• files drop: missing X-OC-Mtime header in the upload code: https://github.com/nextcloud/server/blob/v23.0.2/apps/files_sharing/js/files_drop.js#L86
• any third party client: not possible to support due to the need of proprietary X-OC-Mtime header
• download via Webdav: might depend on the client, not sure if third party clients keep the mtime returned by server (likely not possible to support, needs research)

[despens]

Hey all I posted the original bug report in 2019 and do not have the necessary setup in place anymore to reproduce all of these cases. I'm kindly asking other users to contribute their observations in this online notebook table.

You're right that when using basic WebDAV, for instance via a davs:// URL in Gnome/Nautilus, there is no way to preserve the mtime unless Gnome would, for instance, do something special with nextcloud online accounts.

When a file is copied, the mtime should not change. This is independent from what the operating system below is set up to do. Users do not expect the modification time to change unless the content of the file is changed.

[ianrenton]

I have made a few updates to the table linked by @despens from my recent experience .This is with NextCloud v24.0.3 so I have added a column to your table, I hope that's OK.

[codiflow]

I also updated the table from @despens (added NC 25 and did some reformatting by replacing true/false with symbols which are easier to understand) and can confirm that WebDAV upload never works with linux tools unless they have a special Nextcloud integration like mentioned at #15192 (comment)

[kesselb]

We fixed a bug related to mtime and external s3 storage recently: #41062

[bigfootdevelopment]

I might add, I've had a very mixed bag of results as well. Running NC 27.1.3 This is pretty ridiculous.

It seems mostly folders are affected on my end. Most folders dont retain any modified date, nor do subfolders within those folders, yet oddly all of the files retain the date within the folders and subfolders

Then again I have a random 1 or 2 folders that retained the date, plus all the files inside.

So we know it CAN work, it just gives bizarre inconsistent results.

[KlausBlum]

Are there any news to this subject?

I work on a Windows 10 machine connected to a NC 28.0.0 instance via WebDAV.
Once a file is uploaded, the "created" timestamp is shown as the current time (now).
The "last modified" timestamp shows the correct time (in the past).

When I check back a few minutes later, there is no "created" timestamp, and the "last modified" timestamp shows when the file was uploaded via WebDAV.

I'd like to use a file sync tool (FreeFileSync), but this would require that files keep their correct time and date.

[KlausBlum]

Maybe I should add that I remember that on the same machine I did not have this issue more than a year ago with previous NC versions. So it SEEMS to be possible even via WebDAV.

Does anyone know if NC changed its file handling or if it must be due to a change in MS Windows?

[komoricodrutz]

Hi everyone!
I can confirm that the issue is still there in 27.1.5. I noticed it in 25.x also, but didn't have the time to delve deeper into it.
Using Linux Mint 20.3 and Firefox, i tried the following operations:

• When uploading files through Firefox: Files timestamps (modified) are preserved.
• When syncing from Android devices (4 different makes and brands, using the FolderSync android App): timestamps are preserved.
• When syncing with the AppImg version of the nextcloud client: timestamps are preserved
• When using WebDav connected from the integrated GNOME Online Accounts application: Timestamp is changed immediately after transferring it, straight in Nemo, so it probably has to do with the WebDav thing mentioned at some point before.
• When copying/moving files around from the browser (also tested it in chromium and brave for good measure), the timestamp is again modified. It doesn't seem to matter if the moving around happened between internal or external or any other combination of storages, the files modified date changes to the time the files were moved around.
  This is particularly frustrating, especially if you have a lot of pictures and you rely on the modified date, which is the only available date sorting option, to sort your files, because you may have different naming conventions, depending on the device taking the picture, so you can't rely on alphabetical sorting in this context.
  The interesting thing is that the file metadata does NOT show the modified date and the "Date created" entry is the correct one.
  One more thing I noticed playing around is that when editing a picture in Nextcloud, it completely removes all metadata. But I think that's a bug for another time.
  And using exiftool in this context is quite a hassle and a waste of resources, if you have to comb through all the thousands of files to copy the created date to the modified date.
  Perhaps a workaround would be to add a file created date column in the folder views and perhaps allow the Files, Photos and Memories apps to take that column into consideration, for each album in the case of the latter two.

@despens

When a file is copied, the mtime should not change. This is independent from what the operating system below is set up to do. Users do not expect the modification time to change unless the content of the file is changed.

Exactly!

As suggested by @despens I added a column with NC27 to the table and the behaviours I noiced.

[evrial]

Guys this is not nextcloud bug. You can setup any webdav server and this will happen, because the pathetic clowns listed below didn't let the server to set the mtime, only to get. RFC 2518, RFC 4918 Basically client and server need to agree on mtime header and bypass the RFC, that what NC did with server and client. And with Windows 10 or MacOS clients will never send the mtime and they don't care.
Y. Goland
Microsoft
E. Whitehead
UC Irvine
A. Faizi
Netscape
S. Carter
Novell
D. Jensen
Novell
February 1999

[da-anda]

It is a nexcloud bug. Users don't care which protocol is used in the background when they sync data using their official apps. And nobody forces nextcloud to internally use WebDAV. Also, nextcloud could extend the official WebDAV API with additional features that are not (yet) part of the standard to fix this (and other issues), like sending an additional custom header.
If you use other WebDAV clients to sync data to NC, then it's ofc a different story. But at least official apps should keep crdate etc

[evrial]

Agree that WebDAV API and RFC need an update. What doesn't work is native OS clients and web browsers because it's not a part of standard.
https://docs.nextcloud.com/server/latest/developer_manual/client_apis/WebDAV/basic.html#request-headers

[despens]

It is true that nextcloud cannot mandate external clients to support nextcloud's extension to the WebDAV protocol that transfers modtime dates. For instance, rclone supports modtime on upload with WebDAV.

However, when data is transferred in between different nextcloud instances, or using tools that are part of the nextcloud project, I would expect this critical information not be lost.

That is the case when using

• nextcloud browser upload
• nextcloud to nextcloud federation
• external storage mounted by nextcloud (unless that external storage doesn't support modtime)
• nextcloud mobile apps

[micha06de]

We have Nextcloud 27.1.7.2 here, and I was surprised to discover today that the last modified timestamp is not retained when copying within the Nextcloud web app (it is retained when moving). I seem to remember that this used to be different.

Does anyone know to confirm that this is "normal" behavior, or is there something wrong with our server configuration?

[skjnldsv]

Does anyone know to confirm that this is "normal" behavior, or is there something wrong with our server configuration?

Upgrade to 28 or 29. We fixed it as far as I know.

[emanueleriboldi]

Still doesn't work. I'm using NC 29 and when copying files from PC to Network Drive Share (via WebDAV), the last modified date is the time when I copy the file.

[lucidnx]

it is crazy that it's still not fixed after long time... this is one of important functions!

[KlausBlum]

I'm not sure the error is on the nextcloud side.
The table above mentions rclone as upload tool that retains the correct file date and time. IIUC rclone uses the WebDAV protocol plus some extra tricks to manipulate the file date which obviously is accepted by nextcloud.
I have been using a combination of rclone and FreeFileSync for the last few months now, and it works without problems (despite being significantly slower than "pure" WebDAV).

[despens]

@KlausBlum There is no single error here, but depending on tools and protocols used, nextcloud sometimes retains last modified information and sometimes not, including internal movement of data in between different storage backends, or in between nextcloud instances, where no external tool is used.