Trying to use h5l_create_external

[ggggggggg]

I'm working on a patch to provide a julia API for creating external (soft) links via H5L_create_external. I have a working example, but I'm having a hard time writing a test for it. I create a link from source_file["ext_link"] to target_file["target_group"] then add a few entries to that group. I want to open target_file and check that the entries have the correct values, but I get an error saying "file close degree doesn't match". I'm guessing I want something like "close hard" but I don't know where to look.

I close and restart julia, I can open target_file and see that it has the correct items.

Details below

In plain.jl I replace line 1974 with (I don't think H5L_create_hard_external exists)

     (:h5l_create_external, :H5Lcreate_external, Herr, (Ptr{UInt8}, Ptr{UInt8}, Hid, Ptr{UInt8}, Hid, Hid), (:target_file_name, :target_obj_name, :link_loc_id, :link_name, :lcpl_id, :lapl_id), :(error("Error creating external link ", link_name, " pointing to ", target_obj_name, " in file ", target_file_name))),

Then I run the following

using Base.Test
using HDF5
source_file = h5open(tempname(), "w")
target_file = h5open(tempname(), "w")
# create a group in the target file for the external link to point to
target_group = g_create(target_file, "target_group")
# close the target file (and group), cannot access external link while target file is open
close(target_file)

# h5l_create_external(target_file_name, target_obj_name, link_loc_id, link_name, lcpl_id, lapl_id)
# create the link
HDF5.h5l_create_external(target_file.filename, "target_group", source_file.id, "ext_link", HDF5.H5P_DEFAULT,HDF5.H5P_DEFAULT)

# Now we can use the external link to create a new group inside the 
# target group (even though the target file is closed!). The external link
# works just like a soft link.
group = g_create(source_file["ext_link"], "new_group")

# lets write a few things into this group
group["abc"]="abc"
group["1"]=1
group["1.1"]=1.1
close(source_file)

# Now lets check for those item by looking in target_file directly
target_file = h5open(target_file.filename, "r") # error occurs here
group2 = target_file["new_group"]
@test group2["abc"]=="abc"
@test group2["1"]==1
@test group2["1.1"]==1.1

And get this error

julia> target_file = h5open(target_file.filename, "r")
HDF5-DIAG: Error detected in HDF5 (1.8.13) thread 0:
  #000: H5F.c line 1594 in H5Fopen(): unable to open file
    major: File accessibilty
    minor: Unable to open file
  #001: H5F.c line 1414 in H5F_open(): file close degree doesn't match
    major: File accessibilty
    minor: Unable to initialize object
ERROR: Error opening file /var/folders/_0/25kp6h7x25v6vyjv2yjlcnkm000wrm/T/juliaSob70V
 in h5f_open at /Users/user/.julia/v0.4/HDF5/src/plain.jl:2027
 in h5open at /Users/user/.julia/v0.4/HDF5/src/plain.jl:550
 in h5open at /Users/user/.julia/v0.4/HDF5/src/plain.jl:557

[ggggggggg]

When external links are traversed, they are always opened with H5F_CLOSE_WEAK. Somehow it seems that despite closing both source_file and target_file, a reference to target_file still exists with H5F_CLOSE_WEAK. Then I get the error that file close degree doesn't matter when trying to open target_file because h5open always uses H5F_CLOSE_STRONG.

That said, I still think this may represent a weakness in HDF5.jl because following the same procedure with h5py seems to work fine. As far as I can tell h5py also uses H5F_CLOSE_STRONG by default.

import h5py
import os

source_file = h5py.File(os.tmpnam(),"w")
target_file = h5py.File(os.tmpnam(),"w")
tfname = target_file.filename
target_group = target_file.require_group("target_group")
target_file.close()

source_file["ext_link"] = h5py.ExternalLink(tfname, "target_group")
group = source_file.require_group("ext_link/new_group")
group["abc"]="abc"
group["1"]=1
group["1.1"]=1.1
source_file.close()

target_file = h5py.File(tfname,"r")

assert(target_file["target_group"]["new_group"]["abc"].value == "abc")
assert(target_file["target_group"]["new_group"]["1"].value == 1)
assert(target_file["target_group"]["new_group"]["1.1"].value == 1.1)

h5py's close function is a bit more involved than HDF5.jl's. It is pasted below, the license is BSD, so I think there is no problem with me pasting it here

    def close(self):
        """ Close the file.  All open objects become invalid """
        with phil:
            # We have to explicitly murder all open objects related to the file

            # Close file-resident objects first, then the files.
            # Otherwise we get errors in MPI mode.
            id_list = h5f.get_obj_ids(self.id, ~h5f.OBJ_FILE)
            file_list = h5f.get_obj_ids(self.id, h5f.OBJ_FILE)

            id_list = [x for x in id_list if h5i.get_file_id(x).id == self.id.id]
            file_list = [x for x in file_list if h5i.get_file_id(x).id == self.id.id]

            for id_ in id_list:
                while id_.valid:
                    h5i.dec_ref(id_)

            for id_ in file_list:
                while id_.valid:
                    h5i.dec_ref(id_)

            self.id.close()

Link to docs noting use of H5F_CLOSE_WEAK: https://www.hdfgroup.org/HDF5/doc/RM/RM_H5L.html#Link-CreateExternal