Parametrize test_apt_interface and implement temp directory fixture

[laurenmarietta]

I ended up parametrizing the tests within test_apt_interface anyway in order to better debug things in #214, so I figured I might as well just submit a PR. Changes in this PR:

• Uses pytest's parametrization feature to separate each test APT program into its own test
• Reinstates the use of the temporary_directory fixture for each test

Closes #241.

[laurenmarietta]

Looks like the Travis build failed because of the Fiducial Point Override error. That should (fingers crossed) be remedied by merging #214.

[Johannes-Sahlmann]

Yes, that's what i thought too. Are you waiting for @bhilbert4 's approval to merge #214 ?

[laurenmarietta]

Yes, that was my plan, if that sounds okay?

[Johannes-Sahlmann]

I am getting a pytest failure for program 1068 when I run it locally on my machine. I am not sure how that can happen when the Travis test passes. Please find the error below. Can you please confirm that this passes for you locally?

(fpy35) nereus:~ jsahlmann$ pytest -vs /Users/jsahlmann/jwst/code/github/spacetelescope/mirage/tests/test_apt_interface.py

==================================================================================== FAILURES ====================================================================================
________________________________________ test_complete_input_generation[misc-/01068/1068_official_version.xml-niriss_point_sources.list] _________________________________________

temporary_directory = '/Users/jsahlmann/jwst/code/github/spacetelescope/mirage/tests/temp_data', instrument = 'misc', apt_file_seed = '/01068/1068_official_version.xml'
source_list_file_name = '/Users/jsahlmann/jwst/code/github/spacetelescope/mirage/tests/test_data/misc/niriss_point_sources.list'

    @pytest.mark.parametrize('instrument, apt_file_seed, source_list_file_name', complete_input_generation_parameters)
    def test_complete_input_generation(temporary_directory, instrument, apt_file_seed, source_list_file_name):
        """Exercise mirage input generation from APT files (.xml and .pointing)."""
    
        # Determine source list filename
        apt_dir = os.path.join(TEST_DATA_DIR, instrument)
        source_list_file_name = os.path.join(apt_dir, source_list_file_name)
    
        # Populate the catalogs
        catalogs = {}
        for instrument_name in 'fgs nircam niriss miri nirspec'.split():
            if instrument_name.lower() == 'nircam':
                catalogs[instrument_name.lower()] = {}
                catalogs[instrument_name.lower()]['sw'] = source_list_file_name
                catalogs[instrument_name.lower()]['lw'] = source_list_file_name
            else:
                catalogs[instrument_name.lower()] = source_list_file_name
    
        print('\n\n' + '=' * 100 + '\n')
        if 'DeepField' in apt_file_seed: # exlude until support for NircamTimeSeries is included
            return
    
        # # Remove any existing yaml files
        # obs_yaml_files = glob.glob(os.path.join(temporary_directory, 'jw*.yaml'))
        # for file in obs_yaml_files:
        #     os.remove(file)
    
        if '.xml' in apt_file_seed:
            apt_file_xml = os.path.join(apt_dir, apt_file_seed[1:])
            apt_file_pointing = os.path.join(apt_dir, apt_file_seed[1:].replace('.xml', '.pointing'))
            observation_list_file = os.path.join(temporary_directory, '{}_observation_list.yaml'.format(apt_file_seed.replace('/', '_').split('.')[0]))
    
        else:
            observation_list_file = os.path.join(temporary_directory, '{}_observation_list.yaml'.format(apt_file_seed))
            apt_file_xml = os.path.join(apt_dir, '{}.xml'.format(apt_file_seed))
            apt_file_pointing = os.path.join(apt_dir, '{}.pointing'.format(apt_file_seed))
    
        print('Processing program {}'.format(apt_file_xml))
        yam = yaml_generator.SimInput(input_xml=apt_file_xml, pointing_file=apt_file_pointing,
                                      catalogs=catalogs, observation_list_file=observation_list_file,
                                      verbose=True, output_dir=temporary_directory, simdata_output_dir=temporary_directory,
                                      offline=True)
    
>       yam.create_inputs()

jwst/code/github/spacetelescope/mirage/tests/test_apt_interface.py:140: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
jwst/code/github/spacetelescope/mirage/mirage/yaml/yaml_generator.py:311: in create_inputs
    apt.create_input_table()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <mirage.apt.apt_inputs.AptInput object at 0x113854860>, verbose = False

    def create_input_table(self, verbose=False):
        """
    
            Expansion for dithers is done upstream.
    
            Parameters
            ----------
            verbose
    
            Returns
            -------
    
            """
        # Expand paths to full paths
        # self.input_xml = os.path.abspath(self.input_xml)
        # self.pointing_file = os.path.abspath(self.pointing_file)
        if self.output_csv is not None:
            self.output_csv = os.path.abspath(self.output_csv)
        if self.observation_list_file is not None:
            self.observation_list_file = os.path.abspath(self.observation_list_file)
    
        # main_dir = os.path.split(self.input_xml)[0]
    
        # if APT.xml content has already been generated during observation list creation
        # (generate_observationlist.py) load it here
    
        if self.apt_xml_dict is None:
            raise RuntimeError('self.apt_xml_dict is not defined')
            # tab = self.apt_xml_dict
        # if self.apt_xml_dict is not None:
        # else:
        #     tab = self.apt_xml_dict
        # else:
        #     # Read in xml file
        #     readxml_obj = read_apt_xml.ReadAPTXML()
        #     tab = readxml_obj.read_xml(self.input_xml)
    
        # This affects on NIRCam exposures (right?)
        # If the number of dithers is set to '3TIGHT'
        # (currently only used in NIRCam)
        # remove 'TIGHT' from the entries and leave
        # only the number behind
        # tight = [True if 'TIGHT' in str(val) else False for val in tab['PrimaryDithers']]
        # if np.any(tight):
        #     tab = self.tight_dithers(tab)
    
        # if verbose:
        # for key in tab.keys():
        #     print('{:<25}: number of elements is {:>5}'.format(key, len(tab[key])))
    
        # xmltab = tab
    
        # Read in the pointing file and produce dictionary
        pointing_dictionary = self.get_pointing_info(self.pointing_file, propid=self.apt_xml_dict['ProposalID'][0])
    
        # Check that the .xml and .pointing files agree
        assert len(self.apt_xml_dict['ProposalID']) == len(pointing_dictionary['obs_num']),\
            ('Inconsistent table size from XML file ({}) and pointing file ({}). Something was not '
             'processed correctly in apt_inputs.'.format(len(self.apt_xml_dict['ProposalID']),
>            len(pointing_dictionary['obs_num'])))
E       AssertionError: Inconsistent table size from XML file (28) and pointing file (30). Something was not processed correctly in apt_inputs.

jwst/code/github/spacetelescope/mirage/mirage/apt/apt_inputs.py:307: AssertionError
==========================================================================
=============================================================== 1 failed, 22 passed, 4 warnings in 451.55 seconds ================================================================
(fpy35) nereus:~ jsahlmann$ 

thanks!

[laurenmarietta]

Very odd... yes, I just re-ran test_apt_interface.py on this branch on my machine, and everything passed.

It looks like your assertion error might be related to the last couple changes I made for #214, but since that has been merged into master, I don't know why that would be happening?