Add methods for creating a default MS

[sjperkins]

Added three methods for creating canonical MeasurementSets and their subtables.

• pyrap.tables.default_ms(name, tabdesc). This creates an MS called 'name' and associated subtables. Extra columns, hypercolumns and keywords can be specified in tabdesc which will overwrite/merge into the standard definition for the MS.
• pyrap.tables.default_ms_subtable(subtable, tabdesc). This creates an MS subtable. For e.g. default_ms_subtable("ANTENNA") will created an ANTENNA subtable. Extra columns, hypercolumns and keywords can be specified in tabdesc which will overwrite/merge into the standard definition for the subtable.
• pyrap.tables.msutils.required_ms_desc(table). Returns a dictionary table description for the specified table. required_ms_desc("MAIN") will return a description for a Measurement Set, while required_ms_desc("ANTENNA") will return a description for the ANTENNA subtable.

[sjperkins]

Still a WIP, I'll request a review when I'm ready. Depends on casacore/casacore#559.

[sjperkins]

@ludwigschwardt @o-smirnov @brentjens.

Can you think of anything other simple things that would be useful here?

[sjperkins]

The following Dockerfile allows one to play with the new functionality

FROM ubuntu:16.04
MAINTAINER simon.perkins@gmail.com

ENV DEBIAN_FRONTEND noninteractive

ENV PACKAGES \
    software-properties-common \
    python-software-properties \
    python-pip \
    git \
    build-essential \
    cmake \
    gfortran \
    g++ \
    libncurses5-dev \
    libreadline-dev \
    flex \
    bison \
    libblas-dev \
    liblapacke-dev \
    libcfitsio3-dev \
    wcslib-dev \
    libhdf5-serial-dev \
    libfftw3-dev \
    python-numpy \
    libboost-python-dev \
    libpython3.5-dev \
    libpython2.7-dev

# enable universe, multiverse, restricted with world wide mirrors
#ADD apt.sources.list /etc/apt/sources.list
RUN apt-get update && \
    apt-get upgrade -y && \
    apt-get install -y $PACKAGES && \
    add-apt-repository -y ppa:radio-astro/main && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

RUN git clone --depth 1 https://github.com/sjperkins/casacore.git -b expose-record-to-tabledesc-static-members /src/casacore && \
    mkdir -p /src/casacore/build && \
    cd /src/casacore/build && \
    cmake -DCMAKE_INSTALL_PREFIX:PATH=/usr .. && \
    make -j 8 && \
    make install

RUN pip install git+https://github.com/sjperkins/python-casacore.git@create-default-ms#egg=python-casacore

RUN pip install git+https://github.com/sjperkins/katdal.git@use-python-casacore-ms-creation#egg=katdal

ENTRYPOINT ["/bin/bash"]

[sjperkins]

don't merge this yet please

[sjperkins]

Still adding some testcases

[sjperkins]

I'd like a review on this PR please.

[sjperkins]

Hmmm test build failures.

This isn't going to build against casacore until casacore/casacore#559 makes it into an official release. Guess we'll have to wait until then

I've done a fair amount of testing with in ska-sa/katdal#75 to test this works and am using it in production.

I'm away until late March, can pick this up again then.

[sjperkins]

I've found an interesting issue related to this PR that I'd like input on resolving.

The MeasurementSet class has an addCat method that adds the CATEGORY keyword to the FLAG_CATEGORY column of an MS if it does not exist.

It has type Vector<String>, but zero length.

Now, in the functionality added in this PR, this MS column description gets translated into the python column description as:

 'FLAG_CATEGORY': {'_c_order': True,
                   'comment': 'The flag category, NUM_CAT flags for each datum',
                   'dataManagerGroup': 'FlagCategory',
                   'dataManagerType': 'TiledColumnStMan',
                   'keywords': {'CATEGORY': []},
                   'maxlen': 0,
                   'ndim': 3,
                   'option': 0,
                   'options': 4,
                   'shape': [1, 767, 4],
                   'valueType': 'boolean'},

If this column description then gets inserted back into the C++ layer, the CATEGORY keyword is converted into a Vector of some other type, which (much later) results in this kind of exception:

Exception filling the sub-tables: TableRecordRep::get_pointer - incorrect data type used for field CATEGORY

I can think of 3 options here:

1. Add a "Default" FLAG Category in the C++ layer
2. Add a 'keywords' : { 'CATEGORY' : ["Default"]' } to the FLAG_CATEGORY description returned by the C++ layer in pyrap.tables.msutils.required_ms_desc
3. Remove the CATEGORY keyword from the FLAG_CATEGORY description returned by the C++ layer.

I've test (2) and (3) and confirm they solve the get_pointer exception which arose during the splitms task.

[tammojan]

This isn't going to build against casacore until casacore/casacore#559 makes it into an official release. Guess we'll have to wait until then

That just happened in casacore 2.3.0. Given that; I'd prefer any solution that doesn't involve changing casacore.

[sjperkins]

@tammojan OK, I've pushed a fix into this branch. Tests run against casacore master branch.

[sjperkins]

@tammojan We've done an imaging run with these changes and it worked well. From my side, I'm happy for this to be merged.