HDF5 DB Issue

[kyleabeauchamp]

I'm having issues with the HDF5 backend on OSX. Upon calling step() I get the following traceback. I presume there's some pytables / HDF5 / anconda dependency problem. I was able to successfully run tables.test(), so there's not something royally busted about my install.

~/anaconda2/lib/python2.7/site-packages/pymc/MCMC.pyc in sample(self, iter, burn, thin, tune_interval, tune_throughout, save_interval, burn_till_tuned, stop_tuning_after, verbose, progress_bar)
    277 
    278         # Run sampler
--> 279         Sampler.sample(self, iter, length, verbose)
    280 
    281     def _loop(self):

~/anaconda2/lib/python2.7/site-packages/pymc/Model.pyc in sample(self, iter, length, verbose)
    241         if length is None:
    242             length = iter
--> 243         self.db._initialize(self._funs_to_tally, length)
    244 
    245         # Put traces on objects

~/anaconda2/lib/python2.7/site-packages/pymc/database/hdf5.pyc in _initialize(self, funs_to_tally, length)
    450         for object in self.model.observed_stochastics:
    451             if object.keep_trace is True:
--> 452                 setattr(table.attrs, object.__name__, object.value)
    453 
    454        # Make sure the variables have a corresponding Trace instance.

~/anaconda2/lib/python2.7/site-packages/tables/attributeset.pyc in __setattr__(self, name, value)
    459 
    460         # Set the attribute.
--> 461         self._g__setattr(name, value)
    462 
    463         # Log new attribute addition.

~/anaconda2/lib/python2.7/site-packages/tables/attributeset.pyc in _g__setattr(self, name, value)
    401             value = stvalue[()]
    402 
--> 403         self._g_setattr(self._v_node, name, stvalue)
    404 
    405         # New attribute or value. Introduce it into the local

tables/hdf5extension.pyx in tables.hdf5extension.AttributeSet._g_setattr (tables/hdf5extension.c:7549)()

HDF5ExtError: HDF5 error back trace

  File "H5A.c", line 591, in H5Awrite
    not an attribute

End of HDF5 error back trace

Can't set attribute 'x' in node:
 /chain0/PyMCsamples (Table(0,)) 'PyMC samples'.

[kyleabeauchamp]

pytables                  3.2.2               np110py27_0  
pymc                      2.3.6               np110py27_0  

[fonnesbeck]

@kyleabeauchamp Can you post a small example that replicates the error?

[kyleabeauchamp]

Here you go. AFAIK, the key ingredient is something about the observed variable or the deterministic. Without those, things work fine in HDF.

import pymc as pm

n_vars = 50
n_observations = 10000
observed = np.random.random_integers(1, 100, size=n_observations)

z = pm.Uniform("z", 0.0, 1E8, size=n_vars)


@pm.deterministic
def detvar(z=z):
    return np.ones(n_observations)

w = pm.Normal("w", detvar, 1, observed=True,value=observed)


variables = [z, detvar, w]
mcmc = pm.MCMC(variables, db='hdf5', dbname="test.h5")


mcmc.sample(100000, thin=10, burn=1000)

[glinsky007]

I have been getting the same error. It has to do with the size of the stochastic variables that are being observed. When I reduce n_observations to 100, the error goes away. The magic number is between 8000 and 8250. I am looking into why there is that limit.