-
Notifications
You must be signed in to change notification settings - Fork 33
FAQ
- What image types can MIST handle
- My image filename pattern does not work
- MIST encountered an Out of Memory problem
- My stitching results look incorrect
- What if the output stitched image is too big to be saved
- I installed the CUDA toolkit, but I am getting a CUDA Exception
- The FFTW version does not work
- My stitching performance is not good
- I have a GPU in my machine but it is not showing up
- How to stitch from metadata
- Stitching Time-Slices
- Stitching Multiple Channels
- Stitching using a Macro Script
- What is Stage Repeatability
- Submitting issue or bug report
Any image type that Fiji/ImageJ can read. MIST uses the ImageJ image reader contained within Fiji/ImageJ. The relevant ImageJ image types and formats documentation can be found online in the ImageJ documentation. ImageJ Documentation on Image Types.
All image tiles being stitched must be the same size (width, height, and color depth). MIST does not support stitching images of varying sizes within the same grid.
The Filename Pattern is used to match specific image files within the Image Directory.
There are two types of Filename Pattern, Sequential and Row-Column, both of which can handle Timeslices.
MIST adopts the ImageJ/Fiji notation of using curly brackets "{" and "}" to denote a block of characters that are replaced with numbers. The block "{ppp}", where the character "p" is a placeholder for a number digit, represents numbers from 000 to 999. Each letter between two curly brackets is replaced with a digit between 0 and 9.
Sequential Filename Pattern Types have only one set of curly brackets "{}" that denotes the image's position in the image grid. The special character "p" must be used between the curly brackets. Therefore, a valid sequential filename pattern must have one set of curly brackets "{p}" with at least one "p" character between the curly brackets.
Examples:
- Img_pos001.tif = Img_pos{ppp}.tif
- Img0001.png = Img{pppp}.png
- ImageName01.tif = ImageName{pp}.tif
- If a dataset has a sequence of 789 files
- common name "Dataset01_"
- with a 6 digit numbering (ex. Dataset01_000001.tif)
- Then a valid filename pattern is: "Dataset01_000{ppp}.tif" or "Dataset01_{pppppp}.tif"
Row-column Filename Pattern Types have two sets of curly brackets "{r}" "{c}". One block denotes the image's row index within the image grid and uses the special character "r". The other block denotes the image's column index within the image grid and uses the special character "c". For a valid row-column Filename Pattern there must be one "{r}" block with at least one "r" between the curly brackets and one "{c}" block with at least one "c" between the curly brackets.
Examples:
- Img_row01_col01.tif = Img_row{rr}_col{cc}.tif
- Img_r0001_c0001.png = Img_r{rrrr}_c{cccc}.png
- ImageName_001_001.tif = ImageName_{rrr}_{ccc}.tif
The usage of "{p}" or "{r}" and "{c}" must match the Filename Pattern Type selected. If the Filename Pattern contains "{p}" the sequential Filename Pattern Type must be selected. If the Filename Pattern contains "{r}" and "{c}" then row-column Filename Pattern Type must be selected.
MIST can also handle stitching a series of independent 2D image grids. For example a time-lapse series of image grids. The time-slice stitching is controlled by an additional set of curly brackets in the Filename Pattern with the "{ttt}" special text. Setting the Timeslices field in the input tab of the MIST GUI controls which of the available time-slices will be stitched. The special text "{ttt}" must be used regardless of whether the independent 2D image grid are time slices of z stack slices.
Examples:
- Img_pos001_time01.tif = Img_pos{ppp}_time{tt}.tif
- Img_r0001_c0001_t001.png = Img_r{rrrr}_c{cccc}_t{ttt}.png
- ImageName_001_001_01.tif = ImageName_{rrr}_{ccc}_{tt}.tif
- Dataset01_000001.tif = Dataset{tt}_000{ppp}.tif
There are four different execution modes MIST can run:
The mode selection exists in the Advance options tab in the GUI. Auto mode uses the FFTW mode if it can find the required native libraries and the Java mode if not.
The FFTW and CUDA modes are more memory intensive than the Java mode.
If you are having memory problems first check to see whether you are using 32bit or 64bit computation by looking under the advanced tab and de-selecting the "Use Double Precision?" checkbox.
MIST will attempt to avoid out of memory problems by detecting the available memory and adjusting the execution accordingly. MIST will first attempt to store all the image tiles in the image grid in memory to prevent spending time re-reading. If there is not enough memory to do this, it will enable the releasing of image tiles which will result in holding just the memory pool size images in memory and re-reading image tiles as required. If there is still not enough memory, MIST will lower the number of compute threads being used to reduce the size of the required memory pool. If all these strategies fail, MIST will run the stitching using a sequential Java version which requires enough memory to hold two images and two FFT data arrays.
The memory required by Java mode can be computed using the following formula, given the following four inputs.
- w = Image Grid Width
- h = Image Grid Height
- n = Number Threads Used
- s = Image Tile Size in MB
Memory Required (MB) = 6s(min(w,h) + n + 2) + 4s
If the Java mode does not have enough memory, try increasing the amount of memory Java is allowed to use.
If MIST is being run through ImageJ/Fiji there is a menu option controlling the amount memory it is allowed to use. Select the menu option "Edit" >> "Options" >> "Memory & Threads ..." which opens a dialog box containing an option "Maximum memory:" in MB. Increase this value and select "Ok". A reasonable value is about 75% of the available system memory. Restart Fiji to apply the new memory limit and re-launch MIST to see if the memory problem persists.
The Fiji FAQ has a question related to managing memory in Fiji.
For more advanced Fiji memory management information see the Java Options page maintained by the Fiji developers.
The memory required by FFTW mode can be computed using the following formula, given the following four inputs.
- w = Image Grid Width
- h = Image Grid Height
- n = Number Threads Used
- s = Image Tile Size in MB
Memory Required (MB) = 10s(min(w,h) + n + 2) + 32sn
The FFTW mode has a memory pool that holds two types of image data, the 16bit image pixel data, and the double complex FFT data (or single complex if not using 64bit computation). The size of the memory pool is determined by the size of the image grid being stitched. Memory pool size = min(gridHeight, gridWidth) + 2 + numWorkThreads. MIST iterates over the image grid in a pattern that allows the reuse of the FFT data to prevent requiring the re-computation of any FFT. There is a potential additional numWorkThreads copies of the image pixel data if the input images are not 16bit and a conversion is required from the original format to 16bit.
Unlike the Java mode, FFTW allocates native system memory in addition to memory within the JVM. When determining the available memory, MIST relies on the available JVM memory as an estimation of the available system memory.
The memory required by CUDA mode can be computed using the following formula, given the following five inputs.
- w = Image Grid Width
- h = Image Grid Height
- n = Number Threads Used
- s = Image Tile Size in MB
- g = Number of GPUs Used
CPU Memory Required (MB) = 2s(min(w,h) + n + 2) + 24sg
GPU Memory Required (MB) = 8s(min(w,h) + n + 2) + 24s + 48s
The CUDA mode has two types of memory, pinned memory that exists on both the GPU and the CPU and GPU resident memory. In addition to the pinned memory, the CUDA version needs enough CPU side memory to hold the pixel data for each image tile in the memory pool. For pinned memory, there are two double (or singel) precision arrays the size of the images, and two integer arrays the size of the images. This memory is per GPU, not per memory pool element. The GPU resident memory required consists of four double (or single) precision arrays the same size of the images and one double (or single) complex array the size of the image per memory pool element. The size of the memory pool is defined by the size of the image grid, min(gridHeight, gridWidth) + 2. On the CPU side, one image tile (at 16bit precision) needs to be stored per memory pool element.
Unlike the Java mode, CUDA allocates native memory on the GPU in addition to memory within the JVM.
The following are possible reasons for incorrect looking stitching:
- Incorrect Input Parameters
- Unexpected Image Overlap Variability
- Bad Stage Repeatability Estimate
- Noisy Input Images
The best way to verify the input parameters is by previewing the image grid using zero percent overlap between images. This allows you to visually check that neighboring regions look like they should overlap, and that the arrangement of images is reasonable.
To preview a zero percent overlap mosaic go to the Output tab in the GUI and select Preview Mosaic With No Overlap. Visually ensure that the image grid looks properly organized. Check for corresponding image features between regions that should overlap. Common errors include an incorrect Starting Point, Direction, Grid Width, or Grid Height.
If the zero percent overlap is taking too long to generate you can define a sub-grid using the Subgrid tab and the preview will only be generated for the subgrid.
The image tiles must be arranged in a rectangular grid and overlap with each other. The advanced parameter percent Overlap Uncertainty (default: 5%) reflects the uncertainty in the overlap between image tiles.
Set the Overlap Uncertainty to reflect the variability in the overlaps between images in your dataset.
MIST estimates the Stage Repeatability of the mechanical XY-Stage used to acquire the image grid.
The estimated repeatability is printed in the Log windows as well as in the statistics file saved in the Metadata Directory.
If MIST estimates an unreasonably large Stage Repeatability it is advisable to limit it to a value you know is reasonable for your stage. Conversely, if MIST estimates an unreasonably small Stage Repeatability it is advisable to increase it, allowing MIST to search a larger space trying to find the optimal translations.
Stage Repeatability is measured in pixels.
If the images being stitched contain a high level of noise, MIST can have trouble stitching.
Three potential solutions:
-
Enable double precision computation within the Advanced tab. This is unlikely to solve the problem, but it will not decrease the stitching accuracy. Normally double precision computation is left off because it requires more memory without providing much (if any) accuracy increase.
-
Set the advanced parameter Number of FFT Peaks (default: 2) which determines how many potential translations are tested looking for an optimal one. In noisy datasets increasing this value can improve the stitching at the cost of compute time.
-
Pre-filter the images to remove the noise before stitching. Since there are many varieties of de-noising filters, the selection and application of a reasonable one is up to the user.
If pre-filtering is used it is possible to generate a stitched image from the original images using the global positions computed from the filtered images. This is done by first stitching the filtered images (to generate the global positions metadata file) and then assembling together the stitched image from the original images and the global positions from the filtered images. See the question on how to stitch from metadata for details on how to perform stitching from metadata.
The developers are interested in problematic data sets. Issues with stitching can be submitted to nist-mist@nist.gov or MIST Github Issues Page.
MIST is written in Java which has a maximum size for any array. The ImageJ image writer that MIST relies on uses a Java array to store the pixel data imposing a limit on the size of images that can be saved.
If the number of pixels in the output stitched image exceeds 2147483647 (231-1) then the stitched image cannot be saved.
In other words, the image height in pixels multiplied by the image width in pixels must be less than 2147483647.
If the resulting stitched image cannot be saved using MIST there are two ways to get around this. The first is to stitch a sub-gid using the sub-grid options tab. This will stitch a smaller region and if that region is smaller than the maximum size then that image can be saved. The other option is to stitch the full grid without saving or displaying the resulting stitched image. This will allow MIST to stitch the images and generate the global positions metadata file. This file contains the global positions of every image in the grid. It can then be parsed and assembled using another software package.
Installing the CUDA toolkit might require a restart after the installation completes. If you attempt to run the MIST CUDA stitching before performing the restart you might get a CUDA Initialization error. If this happens restart your machine to allow the CUDA device to be initialized when MIST starts.
Alternatively, ensure that you have the correct version of the CUDA toolkit installed. See the Install Guide for details.
If the FFTW Stitching Program type does not work the most likely reason is MIST cannot find the required library file.
FFTW uses compiled native libraries to perform fast Fourier Transforms. Without these native libraries the FFTW version cannot run.
Details are available by operating system:
For Windows the correct FFTW library file should have been included with the plugin installation. To confirm this navigate to the folder where Fiji is installed. Then confirm the file "libfftw3.dll" exists in the path "lib" >> "fftw" in the directory Fiji is installed. The file might not be named exactly "libfftw3.dll", however it will always end in ".dll". If that file exists then confirm that MIST is pointing at that location in the advanced parameters tab. Open MIST, select the Advanced tab, select the FFTW Stitching Program type. Verify that FFTW Library File contains the path to the "libfftw3.dll" file you confirmed exists. Ensure the path does not just point to the correct folder, make sure that it points right at the library file. For example, "C:\Fiji.app\lib\fftw\libfftw3.dll".
For OSX (Mac) you must install FFTW before you are able to use the FFTW Stitching Program type. See the FFTW installation guide for instructions on how to install FFTW.
Once FFTW in installed ensure that the FFTW Library File parameter under the MIST Advanced parameters tab points to the location where FFTW was installed. The library file should end with ".dylib". The default location for the FFTW library is in "/usr/local/lib".
For Linux you must install FFTW before you are able to use the FFTW Stitching Program type. See the FFTW installation guide for instructions on how to install FFTW.
Once FFTW in installed ensure that the "FFTW Library File" parameter under the MIST advanced parameters tab points to the location where FFTW was installed. The library file should end with ".so". The default location for the FFTW library is in "/usr/local/lib".
There are many factors that can impact the performance of the MIST stitching:
The largest factor in the stitching performance is where the images are located.
MIST is designed with the expectation that the image to be stitched are stored on a local internal hard drive on the computer performing the stitching.
When possible copy the images to be stitched to a local hard drive before performing stitching.
Other image location options are included here ranked from slowest to fastest:
- Network hard drive connected using LAN (Ethernet)
- Any external drive connected using USB
- Local hard drive
- Local SSD
If the images are not local, read time will dominate the stitching time.
If the images are stored on a local disk, the next performance bottleneck is a lack of system memory.
If the computer performing the stitching does not have enough memory to hold all image tiles in memory, tiles will be release and re-read as required slowing down the computation.
When possible MIST will trade memory usage for compute time, increasing the required memory to lower the compute time. MIST is able to stitch your data even with low memory, but it will just be slow. The ideal case is when the computer has enough memory to store all image tiles in memory to allow MIST to read each image tile once and only once.
If memory is not an issue then the next bottleneck is the CPU.
Modern CPUs often have many cores and it might take several threads of compute work to saturate the CPU. MIST has an advanced parameter that controls how many CPU Worker Threads are used. By default this value is set to the number of cores available on the computer in an attempt to saturate the CPU by providing it enough work. Increasing the number of CPU Worker Threads beyond the default is strongly discouraged. The number of CPU Worker Threads should only be used to artificially throttle the CPU.
If the fully saturated CPU is not providing the performance you require a GPU can be added to accelerate the MIST stitching computation.
One of the MIST Stitching Program types is CUDA, which utilizes an NVidia GPU to accelerate the computation. The GPU needs to be CUDA compute capable 2.0 or greater. Also only NVidia GPUs will work as MIST relies on the CUDA toolkit which only works with NVidia GPUs. It is recommended that any GPUs have at least 1GB of graphics memory, but the actual required memory depends on the size of the image grid. For larger grids, 2GB or 4GB of graphics memory might be required. See the question MIST encountered an Out of Memory problem, specifically the CUDA Mode subsection for details on the GPU memory required.
Under the MIST Advanced parameters tab select the CUDA Stitching Program type. Press the button labeled Execute Device Query to call the CUDA device query and find which GPUs the CUDA system can find connected. If you expect to see a GPU which does not show up, check that you have the proper version of the CUDA toolkit is installed. See CUDA Install Guide for directions on how to install the CUDA toolkit version that is required. If the proper toolkit is installed and the GPU does not show up, the GPU might not have the required compute capability (compute capability of 3.0, Kepler architecture or newer), or it might not operate with CUDA in the manner MIST is expecting. Debugging GPU compatibility is beyond the scope of this FAQ.
To stitch from metadata there must be a previous execution of MIST that generated the metadata the current stitching will use. In the previous run, MIST output several metadata files into the directory indicated by the Metadata Directory specified in the Output tab. To setup the new stitching from metadata, enter the new image grid information into the Input tab. Turn the Assemble From Metadata checkbox on. This will enable the Global Positions File box. Enter into Global Positions File the filepath to the global positions metadata file from which you want to stitch. The global positions file file contains the final positions of each individual image tile in the output stitched image and those positions are used to assemble the new images into a mosaic.
MIST is designed to stitch 2D image grid datasets. It does not address volumetric or 3D stitching which requires a system to identify and correlate features across a third dimension.
However, MIST has the facility for handling a sequence of independent image grids. This ability to iterate over a series of independent image grids enables both time-lapse stitching, as well as z-stack stitching. Since each stitched image grid is independent, MIST does not care whether it is iterating over time-slices, or z-stacks.
See My image filename pattern does not work for details on how to setup the Filename Pattern.
To control which time-slices are stitched there is a field in the Input tab called Timeslices which takes a comma separated list and/or number range to specify the time-slice numbers you want MIST to stitch. For example, the Timeslices input could be "1-25,35,45" which would stitch timeslices 1 through 25, 35, and 45.
Any series of independent image grids can be input into MIST as time-slices in order to stitch them.
MIST does not currently support the direct stitching of multiple image channels.
However, by taking advantage of the assemble from metadata functionality one can perform the same task. If you have a dataset with two channels that you want to stitch, first select which channel to use for stitching. Setup and run MIST for this channel. To assemble the second channel using the same translations the first channel generated, re-launch MIST. In the input tab update the Filename Pattern to reflect the second channel. Select Assemble From Metadata. This will enable the Global Positions File text box. Enter into the Global Positions File box the filepath to the global positions metadata file to use to assemble the new channel.
In the output tab keep the same Output Directory and Metadata Directory. Change the output filename prefix to something different than what was used for the first channel. Select Begin Stitching and MIST will use the global positions file selected by the Global Positions File text box to assemble the new channel into a stitched image.
ImageJ/Fiji has the ability to record and run Macro script to automate tasks. See an Introduction to Macro Programming for a general overview of macro programming in Fiji.
To create a MIST stitching macro use the macro recorder.
Select "Plugins" >> "Macros" >> "Record"
This will open a window showing the macro script version of any commands you run in Fiji.
Open the MIST plugin and setup the parameters to run your experiment and select "Begin".
The macro recorder should capture the information required to run MIST from a macro script. The captured command can be copied to a macro script and modified to run MIST from a macro.
A motorized mechanical XY-Stage is used to move a biological sample with respect to the microscope’s optical imaging system. This movement is done by two independent stepper motor linear actuators, one for each direction.
It is important to understand the mechanical limitations of a stepper motor linear actuator mainly with regards to its accuracy and repeatability.
Accuracy: Actuator motion can only be as true as the mechanical components permit. The accuracy is the actuator’s ability to provide precise increments of motion along its axis. In a mechanical system, this primarily concerns the lead screw as well as the feedback device (encoder, linear scale etc.). Lead accuracy of the screw, resolution of encoders, and ability of the controller must combine to produce the desired accuracy. In most microscopes the accuracy is very high and can be calibrated.
Repeatability: Repeatability is the ability of a device to reach a specific location multiple times. It does not take into account the trueness of the position but rather the ability to go back to that position. Many times the actuator will follow a slightly bowed or twisted path due to imperfect construction. The repeatability is the measure of uncertainty relative to a given actuator movement. The repeatability cannot be calibrated, but it can be measured for any microscope stage. Each time the stage controller requests a position value, the actual resulting position is within repeatability of the requested value.
In a grid tiling, the positions (x, y), that the stage will go to, have an uncertainty equal to the stage repeatability (x ± r, y ± r). However, translations (dx, dy) computed in the vertical or horizontal directions between consecutive tiles are differences between respective positions. Thus, the uncertainty on the computed translation values is 2*repeatability(dx ± 2r, dy ± 2r).
When computing vertical translations between two consecutive tiles, the dx and dy components will correspond to the projection of the stage displacement on the camera coordinate systems (which might have a fixed rotational angle) with an uncertainty equal to 2*repeatability. We use this information to estimate the stage repeatability from the computed translations.
The Stage Repeatability value can be entered into MIST under the advanced parameters tab of the GUI. MIST expects a Stage Repeatability values in pixels.
The code for MIST is being stored and managed on Github which has the functionality to report and track issues. To report a bug go to the MIST Github page and select "Issues" on the right hand side of the page. Or go directly to the Issues page. To create a new issue select the green "New Issue" button on the top right side of the page.
When posting in issue please be as specific as you can and if possible include the following data generated by MIST from the stitching experiment that encountered the error.
- The contents of the Log window in Fiji.
- done by clicking the menu option "File" >> "Save As" on the Log window
- The relevant metadata files
- statistics file (important)
- global positions file
- relative positions file
- relative positions no optimization file
If you have questions or issues regarding MIST feel free to email the developers at: nist-mist@nist.gov