Pi-lomar updates

[Short-bus]

A few issues that arose with the initial version have been addressed.

TO UPDATE YOUR COPY --------------------------------------------

You can copy the /src and /circuitpython folders from the Main branch of GitHub and overwrite the older copies on the RPi.

The RPi will be automatically updated as soon as you have copied the /src files across.
The Tiny2040 microcontroller will need the new copy of /circuitpython/code.py copying from the RPi onto the controller via a USB connection (remove the GPIO header connection first for safety!)
Then restart the pilomar.py program.

MICROCONTROLLER CHANGES ------------------------------------

circuitpython/code.py
Now reports the boot_out.txt file back to the RPi at startup to check which flavour and version of python is installed.
Startup LED color changed from PINK to CYAN so it does not look like an error state.
UART error messages standardised.
ADC0 pin can be left unassigned, so it's available for other purposes.
DRV8825 fault signals only reported ONCE to prevent buffer overflow and crash.
steppermotor class gains 'DriverType' attribute in case other drivers are used.
'tune complete' message has timestamp formatted instead of integer value.
Compatible versions check only considers MAJOR and MINOR elements of aa.bb.cc version numbering.
memorymanager class counts how many times the garbage collector has been triggered. Count is periodically reported back to the RPi.

RASPBERRY PI CHANGES -------------------------------------------

src/pilomarcelestrak.py
Now respects projectroot for file handling.

src/pilomarimage.py
New CalculateStarSpread() method to calculate how much of the image contains stars. Supports finetuning of the tracking function.

src/pilomarlogfile.py
New PackageSearchResult() method to extract entries from log files via grep commands and zip the result.

src/textcolor.py
Fixed fault where refresh character '?' did not refresh the menu display.

src/pilomar.py
ProjectRoot is now calculated dynamically from the location of src/pilomar.py. Makes it easier to install in different directory structures.
Fixed bad default values for MinAltitudeAngle, MaxAltitudeAngle. Includes self-correcting any existing bad values.
LocalStarsMagnitude and ConstellationStarsMagnitude parameters added to control the size of the Hipparcos catalog selected for observations.
TargetMinMagnitude parameter defaults to 7.0 which is a better starting point for the 16mm lens builds.
Removed support for the 4 LEDs seen on the development PCB, they are not part of the published project.
Workaround for crash after the initial Hipparcos catalog build. The build closes the user input stream for some reason, so now the program stops cleanly and asks for a restart.
SetMotorAngle() method now shows the current position before asking the user for a new one.

Miscellaneous Tools Menu:
Added "About" function to list the version numbers of various components and packages.
Added "Tracking status" function to list the parameters related to the tracking function, helps to identify which changes to make to improve tracking.
Added "Set local timezone" function to let the user select their local timezone from a list rather than having to research it. LocalTZ parameter is still for reference only at the moment.

Microcontroller Tools Menu:
Added "Zip comms log" function to extract UART communication messages from the log file and zip them. Easier to find/share information when debugging issues.
Added "Monitor comms" function to display in realtime the communication between the microcontroller and the Raspberry Pi over UART. To help with debugging communications.

[Short-bus]

2024-01-Issues branch now merged.

The latest round of updates is now merged into the main branch.
See: https://github.com/Short-bus/pilomar/wiki/Updating-the-software-from-GitHub for advice about updating.

OBJECTIVE

Use feedback from people who have built a pi-lomar to fix some bugs and niggles.
Make it easier to identify common mistakes and problems that people have had.

• Especially the problems with getting UART communication working.
  Remove some of the restrictions that people have found.
  Prepare for people wanting to tune the operation of pi-lomar when they start capturing images.

ISSUES ADDRESSED

#44 Issues related to CircuitPython 8.x installation on Tiny2040. (enhancement)
#51 Preview can inherit previous target as background. (bug)
#52 Move more motor parameters into parameter file. (enhancement)
#53 src/pilomarimage.py CalculateStarSpread() fails for bright skies. (bug)
#54 Tree branches confuse Astroalign. (enhancement)
#56 Target selection menu does not recognise the 'x' key. (bug)
#57 Simplify microstepping configuration for motors. (enhancement)
#58 Can pilomar run on RPi4B with Bookworm o/s? (enhancement)
#59 ProjectRoot not fully respected bug. (bug)

MICROCONTROLLER SOFTWARE

circuitpython/code.py

• The program now also runs on CircuitPython 8.x. You nolonger need to find the archived copy of CircuitPython 7.2.
• Message processing cleaned to avoid confused data if versions are out of sync.
• OptimiseMoves parameter now allows unlimited rotation in any direction. This improves performance in more Southern latitudes.
• Gearing and step signal configuration now set in motor configuration message. Set it via the parameter file.
• The old UseMicrostepping parameter has been removed because it has no effect. There are other ways to introduce microstepping. See below.
• DRV8825 fault signal handling improved.
• GPIO pins now use PULL-UP setting and fault status latches more appropriately. This is better protection from unconnected floating pins triggering random on/off fault signals.
• Multiple changes to the code to help identify communication problems.

RASPBERRY PI SOFTWARE

src/pilomarimage.py, pilomarlogfile.py, pilomartimer.py, pilomar.py, textcolor.py

• General updates and continuing cleanup of code.
• MotorsEnabled and UseMicrostepping parameters removed because they had no function anymore. Microstepping is still possible via other options. See below.
• Motor configurations now taken from parameter file rather than hardcoded values. This supports builds which use alternative motors. MinAngle,MaxAngle,GearRatio,MotorStepsPerRev,MicrostepRatio,RestAngle,BacklashAngle,Orientation. All now taken from the parameter file. Values are available for each motor separately. FastTime,SlowTime,TimeDelta now taken from parameter file, global values for both motors. This means that you can tailor these values in your build and the values will be retained if you update the software. In previous versions these values were hardcoded so you would have to re-edit the source codes each time you took an update.
• Microstepping support simplified. To activate microstepping just set the 'MicrostepRatio' parameter to the number of microsteps you want the motor to take. The program will revise all related parameters in the RPi and Microcontroller configurations. If you change MicrostepRatio you must power cycle the RPi and microcontroller to ensure that the new configurations propogate correctly.
• Cleanup of preview image. Parameters to control the number of labels added. Keeps image more clear.
• Latitudes in Southern Hemisphere now allowed. Previous 'error' message is now removed. Feedback suggests that the telescope will operate OK. See the new OptimiseMoves parameter to allow more free movement if needed.
• GOTO and HOME moves now report progress during moves. This gives some reassurance that things are happening on large moves.
• GOTO nolonger reports an error if a single attempt to move does not complete in 30 seconds. The limit is now 60 seconds, an error is only reported if all allowed retries fail. False alarms were sometimes raised if there was a backlog of microcontroller messages to handle before the move could begin. Any stale messages are now flushed at the start of the observation to reduce the processing delay.
• Code protected from empty starlists, eg if capturing daylight images. Drift tracking routines would fail if a blue-sky (or total cloud) image had no stars at all.
• NGC catalog data is now cached rather than being recalculated. This makes the program start up faster. NGC Catalog now includes the 'type' of object, this is included in Preview image labelling.
• Now an option to generate a short AVI animation of the preview images at the end of an observation. AVI is created in the session's /preview folder to show motion of the target and surrounding stars during the observation. Set the parameter GeneratePreview to False to remove this option.
• The raspistill commands to capture different types of images are now held as templates in the parameter file. You can edit these templates if you want to change the options. Your changes will be retained across future versions.
• Camera image buffers are reset at the start of each observation so previous and current target images do not get confused.
• Improvements to the 'about' and 'status' options on various menus to show more information. Making more information available to understand configuration and its effects.
• View Parameters option now highlights which parameters have been changed from their defaults.
• Timer objects now use non-blocking wait() instead of sleep() to improve performance.
• Some new parameters introduced (currently for documentation only, but may be used functionally in future versions). BoardType : A label to identify the motorcontroller PCB, so that variations can be supported. IRFilter : Indicates whether the sensor has an infrared filter (default True) PollutionFilter : Indicates whether a light pollution filter is in place (default False)
• The initial 'target selection' menu now allows you to 'x' out instead of forcing you to select a target. If you 'x' out, your last target is selected automatically. If you have not made any observations yet you MUST still choose a target before you can continue.
• More visible warnings if communication is not working as expected.
• UART teething problems seem to be common with new builds, so the program has more facilities to monitor/alert. The green 'status' display shown as the program starts up includes a check and report for suspected comms problems. Select the Status option from the main menu to view it again.
• 'Monitor communication' option added to the Microcontroller Tools menu. This shows traffic in both directions in realtime so you can see what is working. You can 'reset' the microcontroller from this display and watch the microcontroller reconfigure itself by pressing the 'r' key. You can send an example yellow flash to the microcontroller LED to prove the microcontroller is understanding and responding by pressing the 'f' key. This shows if communication is working as planned.
• Initial support for Bookworm 64bit operating system on RPi4 added.
  -- This is not heavily tested yet, functionality may still change.
  -- It ONLY works on the Raspberry Pi 4 at present.
  -- The RPi5 hardware needs further changes, the current version will NOT work on an RPi5.
  -- The RPi3 does not support Bookworm.
  -- The software uses libcamera when running under Bookworm.
  -- The software uses rapistill when running under Buster.
  -- The 64bit O/S improves overall performance.
  -- The program will check the combination of hardware and o/s at startup to be sure you have a supported combination.
  -- On RPi3 pi-lomar only works with Buster 32bit.
  -- On RPi4 pi-lomar works with Buster 32bit or Bookworm 64bit.
• The opencv routines to enhance images for tracking calculations are also now parameterised. The original version had a single hardcoded enhancement routine. The new version allows you to create and edit enhancement scripts in the parameter file. If your observation conditions require special enhancement parameters for reducing haze or making stars more clear you can now set the values without having to modify the software itself. There are two scripts included by default.
  -- (1) 'EnhanceStars': Increases contrast in the image to help drift tracking identify stars more clearly.
  -- (2) 'UrbanHaze': Removes common urban haze gradient from the image then enhances the remaining stars. This also seems to reduce tree branches in some cases.

RASPBERRY PI SCRIPTS

scripts/buildpilomar_bookworm64, scripts/buildpilomar_buster32, scripts/zip-pilomar

• You can now build pi-lomar on the BOOKWORM 64bit operating system on the RPi4. Use scripts/buildpilomar_bookworm64 when performing the initial build.
• You can build pi-lomar on the BUSTER 32bit operating system on the RPi4 and RPi3. Use scripts/buildpilomar_buster32 when performing the initial build.
• The zip-pilomar script now correctly stores the circuitpython code.

DRAFT DOCUMENTATION

docs/Pilomar Manual 2024-02-26.pdf

• Parameter list updated to new version.
• Includes an example of an observation with images of various stages of the process.

[Short-bus]

New release of the package is now merged. Many new features and a few issues fixed.
Many thanks to all the people who have contributed items and ideas so far!

JULY.2024 RELEASE NOTES

** VERSIONS IN THIS RELEASE **

• src/pilomar.py version 1.1.0
• circuitpython/code.py version 1.0.0
• PCB version 2023-12-14 (unchanged)
• 3D printed structure V13 (unchanged)

** UPGRADE NOTES **

Follow the standard upgrade notes on Pi-lomar's GitHub wiki but with the following extra items.

1. For safety consider backing up your current SD card image before upgrading.
2. You MUST update both both the RPi and the Tiny2040 with this release.
   There are changes that both devices must handle.
3. On existing BOOKWORM builds you should run the new scripts/buildpilomar_bookworm64 script to add the astropy library.
   (The new FITS image format needs this library)
4. Delete or rename /home/pi/pilomar/data/ngc.pkl on the Raspberry pi so that the new catalog can be built.
5. 'Resume last observation' and 'Repeat earlier observations' history is reset by this update.
   Before upgrading make a note of any important earlier observation details that you want to return to in the new version.
   You will need to select them again.

**CHANGE NOTES **

circuitpyhon/code.py

• Breaking change! You must update src/pilomar.py to match this version.
• Runs on CircuitPython 7.2 and 8.2.
  • Not yet running on CircuitPython 9.x
  • Current recommendation is CircuitPython 8.2
• General code refactoring to save memory.
  • Removed unused imports.
  • Removed unused methods from some classes.
  • LED task list simplified, removed unused states.
• Common UART channel config for 7.2 and 8.2 CircuitPython environments.
• Smoother motor movement if using microstepping.
  • CircuitPython clock is only accurate to whole seconds which caused movements to bunch up.
  • New solution simulates more accurate clock which spreads movement evenly.
• Trajectory now works on STEP POSITION rather than ANGLE.
  • This is a breaking change. src/pilomar.py must also support this.
• Slewmicrostepping supported.
  • You can have different microstepping settings for OBSERVATION vs GOTO/HOME moves.
  • This allows the telescope to move into position more quickly.
  • This requires the latest src/pilomar.py to support it.
• Backwards compatibility with earlier message formats removed because this is
  already a breaking change.
• RPi can send direct "pin" commands to turn microcontroller GPIO pins on/off.
  • This is a development and debugging feature.
  • The latest version of src/pilomar.py allows manually entered commands
    to trigger this.

data/ngc.json

• The NGC targets have been expanded to include the later IC targets too.
  • You need to delete any existing /data/ngc.pkl cache file if you want to use the new catalog.

src/pilomar.py

• General refactoring continues.
• Initial support for Raspberry Pi 5.
• Pilomar now runs on RPi3B, RPi4B and RPi5.
  • RPi3B must use the original BUSTER 32bit O/S.
    Some new features are not available on BUSTER builds.
  • RPi4B can still use the original BUSTER 32bit O/S.
    Or you can upgrade to BOOKWORM 64bit (significant performance improvement).
  • RPi5 must use the latest BOOKWORM 64bit O/S.
• Reduced repeated log messages during observation loops.
• New FolderHandler class replaces old FolderList.
• Observation related directory structure is now only built as needed.
  • Previously all possible directories were built when target was chosen.
• SessionHistory now uses a new sessionlist class.
  • Now stored in a JSON file. (Previous SessionHistory from older versions will be lost when you update to this)
  • The change is introduced to support planned future features.
• FastCapture and FullCapture modes introduced.
  • FullCapture is the traditional solution. Images are fully processed as soon as they are captured.
    Eg: .DNG raw data is extracted and saved before capturing the next image.
  • FastCapture is an option in the Parameter file. Images are captured as quickly as possible,
    final processing must be done separately when the observation has completed.
    Use the Process Image Files option on the Camera Tools Menu to complete the processing at the end of the observation.
• Aurora target type added.
  • Uses new FastCapture mode to gather images quickly with minimum processing during the observation.
  • Produces Keogram of the aurora activity at the end of the observation.
• When simulating images, you can now simulate light pollution, image noise and electrical field noise.
  • This is to aid with development/testing. Controlled via Parameter file.
• Dashboard displays use softer colour scheme (background now very dark gray, colors can still be changed in parameters).
• Startup warnings if OptimiseMoves or SlewMotor parameters enabled.
• Improved error detection when trying to find USB storage for images.
  • Program now issues more clear warning and lists common causes of problems.
  • Less likely to terminate.
• Camera Tools menu lets you select 16mm or 50mm lens.
  • Makes it simpler to set the parameters correctly for each lens.
  • You still need to restart the program to propagate the new settings fully.
• Timelapse feature added.
  • You can now define a delay between each image if you want to capture a timelapse sequence.
  • For example, creating a timelapse of a moonrise.
• astrolens, astrosensor and astrocamera classes moved out to separate include file.
  • Easier to exchange for alternative camera classes in the future.
• You can now send a manually typed command to the microcontroller.
  • Development/debug feature.
  • Do this from the MONITOR COMMUNICATIONS option, press the 'c' key to enter a manual command.
• Microcontroller communication thread now closes automatically if main thread fails.
• Horizon parameter added.
  • Observations and trajectories will not allow observations to continue below the 'horizon'.
  • This means that the MinAngle for the altitude motor can now be below 0degrees if you need it.
  • -ve degrees allows you to park the camera below horizontal - for example if you want to HOME the lens up against a cover or shade.
• Warning if Comet data from Minor Planet Centre is getting old.
  • Comet data should be refreshed periodically as orbit data and new comets are regularly updated.
  • Refresh is not yet automatic, you have to delete the old data file when you have an internet connection available to download a fresh copy.
• Exercise Motor options added to Motor Tools menu.
  • This automatically moves the motors through their entire range of movement. For build/dev/debug support.
• You can now adjust the exposure time DURING an observation.
  • Use the "+" and "-" keys to increase or decrease the exposure time.
  • "+" will DOUBLE the exposure time. Eg from 10 seconds to 20 seconds.
  • "-" will HALVE the exposure time. Eg from 10 seconds to 5 seconds.
• Preview image includes a RA/DEC direction compass.
  • You can see which in direction RA and DEC increase/decrease on the diagram.
• GPIO handling moved into separate pilomargpio.py library.
  • Pilomar can select between GPIO and GPIOD libraries depending upon the O/S.
  • RPi3 & RPi4 use the traditional GPIO library.
  • RPi5 uses the alternative GPIOD library.
• EnableCleanup/DisableCleanup processing improved.
  • Previous version did not separate raspistill and libcamera requirements properly.
• Basic FITS files can now be generated.
  • Some image processing software wants images in .FITS format rather than .jpg or .dng
  • On BOOKWORM O/S builds there is now an option to export as .fits
  • BUSTER builds don't support this.
• You can now decide which types of images are captured from the Camera Tools Menu instead of having to edit the parameter file.
  • JPG, DNG or FITS (See src/pilomarfits.py below for more detail)
  • pilomar will select the appropriate camera driver for the O/S and image types.
• If you edit the Parameter file from the menu, you must now restart the program for it to take effect.
  • The edit parameters option is a convenience function, but does not yet fully re-apply all parameter changes dynamically.
  • Earlier versions allowed you to reload the file during operation, but they did not safely update all instances of the new values.
  • New version will not allow any motor moves to be made until program is restarted.
• Improved support for microstepping.
  • Microstepping makes the system smoother and quieter in operation.
  • New 'Slew' mode allows larger step moves for HOME/GOTO movements. (SlewMotor parameter)
  • Options now on the motor tools menu to turn on/off and to select common/recommended microstepping configurations.
  • Program must be restarted after changing motor configurations to make sure changes propogate fully.

src/celestrak.py

• Improvements to data caching and refreshing logic.

src/pilomarcamera.py

• New library to support different camera classes.

src/pilomargpio.py

• New include file. GPIO handling is moved into this file.
  There are new classes for the classic GPIO handler and also
  classes for the newer GPIOD handler. This is to enable RPi 5 support.

src/pilomarimage.py

• Some refactoring.
• Initial EXIF tag extract available. More development to come.
• Logging functionality reworked across all modules.
• Added pilomarkeogram class. Related to mapping aurora activity.
• Removed old keograph class. Not used.

src/pilomarlogfile.py

• Nolonger flush log file writes immediately to storage. This was hitting the SD card very hard.
  • FastFlush parameter can be enabled to perform immediate flush of write buffers IF required.

src/textcolor.py

• Updated to latest version. (0.0.6)

src/pilomarfits.py

• New utility to save JPG & FITS format image files.
  • Only works on BOOKWORM operating system builds (it requires picamera2 library).
  • This feature is early in development, it probably needs improvement, but good to get some feedback if anyone uses it.
  • If saving in JPG & FITS at the same time, the .jpg format is a very simple debayered copy of the raw sensor data.
    If you want higher quality .jpg files I recommend not saving .FITS at the same time.

docs/Pilomar manual 2024-07-04.pdf

• Latest draft of manual for pi-lomar to match this version.
  Still a work in progress.