APOLLO Operator's Guide

Last updated: Nov 25, 2024 - RM

1. Hardware
  1.1 The Utah box
    1.1.1 Opening and closing Utah
    1.1.2 Left side / laser cavity
    1.1.3 Right side / receiver
    1.1.4 Top section
  1.2 The obs-level cabinet / "phone booth"
    1.2.1 Opening and closing the cabinet
    1.2.2 Top section
    1.2.3 Middle section
    1.2.4 Bottom section
  1.3 The ILE
    1.3.1 Opening and closing the ILE
    1.3.2 Right side
    1.3.3 Left side
  1.4 The ACS cabinet
  1.5 TBAD
  1.6 Interlock system and keyswitch
2. Software
  2.1 ATUI
    2.1.1 Tabs along the top
    2.1.2 Control/status buttons on left
    2.1.3 The Plots window
  2.2 housctl
  2.3 ICC
  2.4 Miscellaneous scripts
    2.4.1 TBAD monitoring
    2.4.2 Web page server
    2.4.3 Generating quarterly schedule
    2.4.4 Generating notification files
    2.4.5 Generating slew commands

3. Operations
  3.1 Periodic communications and checks
    3.1.1 Summer of odd years: LOD
    3.1.2 Quarterly: FAA and Space Command
    3.1.3 Monthly: TSIM test
  3.2 Preparation and warmup
    3.2.1 Day before: Request NOTAM
    3.2.2 Hour before: Set up observing log
    3.2.3 Hour before: Prepare ATUI and other windows
    3.2.4 Hour before: Check GPS clock
    3.2.5 Hour before: Make polynomial files
    3.2.6 Hour before: Check space block files
    3.2.7 Half hour before: Check out ACS
    3.2.8 15m before: check APD
    3.2.9 15m before: STV crosshairs
    3.2.10 5m before: warm up main laser
  3.3 Ranging
    3.3.1 Pointing and focus check on a star
    3.3.2 Pointing check on a crater
    3.3.3 Laser threshold and power check
    3.3.4 First target: Apollo 15
    3.3.5 Repeat Apollo 15 with ACS
    3.3.6 Other reflectors
    3.3.7 Logging results
    3.3.8 Finish and cooldown
4. Troubleshooting and Maintenance
  4.1 Pulse train and BUT
  4.2 Common problems
  4.3 Regular checks

1. Hardware

1.1 The Utah box

The Utah box is a box shaped like the eponymous state, attached to the rear of the telescope's mirror cell adjacent to port BC1. The box contains an optical bench which holds the main APOLLO laser (the "Leopard" laser), the optics which send the outgoing light to the telescope and receive the incoming light from the Moon, the APOLLO detector (the APD), and various electronics to support the laser and detector.

Heating and cooling systems attempt to maintain the temperature inside Utah at 20+-1 C. The temperature may get higher while the box is closed and the laser is in use, or lower if the box is open in cool weather. Laser performance will be affected if Utah temperatures go above 25 C or below 15 C.

1.1.1 Opening and closing Utah

In order to access Utah, you will need the telescope at high altitude: 80 degrees for a tall person to reach, 83-84 for a short person. The Utah box has three sections: the left side door is the most frequently accessed and also has to be opened first, the right side door is the second most common, and the top part rarely needs to be opened, which is good since it is not on a hinge and therefore is the most difficult to deal with. Each section has a number of small fiddly latches which are prone to getting out of alignment or sometimes getting trapped in between sections so that they can't be properly latched.

To open the left door, undo three latches at the top of the door, three down the middle, and three underneath the left door (one of those may be unlatched already because it is permanently out of alignment). Gently pry the door open at the middle. If you are going to be working inside Utah for more than a minute, you may want to tape the left-side door to the standing cabinet to hold it out of the way, otherwise it will fall closed again.

To open the right door, with the left door already open, undo an additional two latches at the top of the right door and two latches underneath the bottom of the right door. Be careful! There are some extra latches further underneath the right side which are for accessing the APD which you do not want to unlatch in the normal course of things. So make sure that the latches you do open are near the edge of the underside. Once the latches are undone, you can fold the right-side door all the way open and out of your way.

To open the top of Utah, first open the left and right doors, then place a small stepladder directly in front of Utah, and climb at least partway up the ladder. Undo three additional latches on the top of the right side, two latches on the center vertical portion, three latches on the top of the left portion, and three on the leftmost vertical. Pull the ell-shaped cover gently clear and lower it down to the floor out of your way.

Reclosing the top section is tricky because these latches are the most likely to get trapped in between the box and the cover. Stand on the ladder and pick up the cover, holding it with the outward side down so that gravity will open the latches. Then bring it upright and into position carefully and fasten one latch (the rightmost latch on the top section is usually a good choice). With that latch closed you can support the cover with one hand while checking that all the other latches are free, and if necessary open a small gap to free any that are trapped. Once they are all free, you can fasten the remaining latches.

Reclosing the right door is the simplest because it has the fewest latches and they are mostly well-behaved. When you close the left door, sometimes the latches along the center seam get trapped, so check those are free before you start fastening any of them. You may find that you need to press the left door closed with one hand until you have two or three latches fastened (top and center are recommended), and after that you can let go.

1.1.2 Left side / laser cavity

The left side of Utah houses the laser cavity where the light (both IR and green light) is generated. After the insulated Utah door is opened, there is an additional heavy door blocking off the laser cavity; that door is not fastened in place but just held by gravity; it can be opened just by grabbing the handles and pulling. Before opening the laser cavity door, make sure that the laser is not flashing OR that all people in the dome have been provided with laser safety glasses. When the laser is flashing and the cavity is open, there are eye safety issues but also there can be danger to skin if anyone reaches into the cavity. Only have the cavity open with the laser flashing if you know what you are doing.

As long as the laser is not flashing, there is no danger to humans in opening the cavity, but try to minimize the amount of time it is open in colder weather.

Oscillator

Laser shutter

AOML

Rear mirror

Amplifier

Bottom mirrors

Frequency doubler

Bolometer

Interlock shutter

1.1.3 Right side / receiver

FPD

T/R mirror

"Snout"

STV

Diffuser

ACS

APD

1.1.4 Top section

CAMAC crate

Raspberry Pi

Flip mirror switch

1.2 The obs-level cabinet aka "phone booth" aka "TARDIS"

The APOLLO cabinet on the observing level holds supporting electronics and low-to-medium voltage power supplies for the laser and detector, some control interfaces (most of which can also be controlled remotely), and the computer named houston along with accompanying monitor, keyboard, and mouse.

The cabinet is insulated so that it doesn't cool down too much, and kept warm mostly by passive heating. If the cabinet gets too warm active ventilation to the ILE will start up; the temperature sensor in the cabinet has a history of incorrect high-temperature readings, but as of April 2023 the sensor was recalibrated and should behave better.

1.2.1 Opening and closing the cabinet

The cabinet has metal panels on the outside and insulated panels on the inside. The aluminum panels are held on by velcro, so just grab the knobs and yank. The middle aluminum panel must come off first, then your choice of top or bottom. Set the removed panel(s) to the side.

The insulated panels are each held by four small black latches which fasten into vertical rails. The top panel must come off first; turn the latches vertical to release, then gently wiggle the insulated panel free. Optionally, if you only want to access the middle, you can just push the top panel up as high as it will go and undo the latches for the middle panel. Set the removed panels to the side.

To reclose the cabinet, set the bottom insulated panel in first and tighten the latches to hold it to the rails. Then the middle panel insulated panel should go directly on top of the bottom one, and the top panel directly on top of the middle. There will be a gap between the top panel and the "roof" of the cabinet. Once the insulated panels are in place, press the bottom and top aluminum panels against the velcro with the top one flush against the top of the cabinet; then the middle aluminum panel should fit neatly between the other two with overlapping lips.

1.2.2 Top section

Bolometer control paddle

STV control box

Flow meter

Osc & Amp tuners

1.2.3 Middle section

Continuum control interface

houston KVM

1.2.4 Bottom section

houston tower

power supplies

1.3 The ILE / "clubhouse"

The Intermediate Level Enclosure (ILE), also sometimes called "the clubhouse", is located on the ceiling of the intermediate level but is accessed from the observing level from the pit behind the telescope.

The ILE contains cooling systems and high-voltage power supplies for the main APOLLO laser, plus the main driver of the interlock system that controls the safety shutter determining whether laser light is allowed out to the telescope.

The temperature inside the ILE should be maintained between about 5 C and 35 C, and must be kept above freezing because cooling water is held in reservoirs in the ILE. A small heater with its own thermostat will come on when the temperatures get below 5 C. In warmer weather or when the laser is in use, ventilation systems controlled through the housctl program can run fans and open one of the ILE louvers; the other louver is kept open in summer and closed in winter. As of 2022 the main temperature sensor for the air in the ILE was malfunctioning and reading high, causing ventilation even when the dome air was very cold. In January 2023 the ventilation decisions were changed to be based on another temperature sensor on the wall of the ILE, instead of the one hanging in the air in the middle of the ILE.

1.3.1 Opening and closing the ILE

You can reach the ILE by going underneath the telescope if it is at high altitude, going through the NA2 pit, or crawling down from the observing level floor in the Crush zone behind the telescope and below Utah, if the telescope is at low-to-medium altitude. The railings around the crush zone can be retraacted, and there are two metal steps (one small and one large) that fold out of the wall just behind the altitude drive disk to facilitate climbing up and down.

The two doors each have two black latches which are unlocked with a key, which should be kept on top of the gray electronics box just to the right, at the edge of the NA2 pit. If there is no key there, a spare should be in the bottom left corner of Utah, outside of the laser cavity. Another spare may be somewhere in the APOLLO supply cabinet next to Ben's cubicle in the main lab of the Ops Building. Use the key to loosen the latches, then slide the left door to the left or the right door to the right. Pull the door free and set it out of the way; avoid tangling cables or wedging under the telescope skirt.

One or both doors may be opened at a time. In sub-freezing weather, try to limit the amount of time the doors are opened since the ILE must be kept above freezing and it only has a small heater. There are two fluorescent lights at the back of the ILE with simple rocker switches: one on the left (crawl straight back on the black pads and then look up) and one on the right (crawl behind the cooling and electronics units to reach this, or possibly a long-armed person can reach it over the main electronics rack). Make sure the lights are turned off before closing up the ILE.

To close the ILE, slide the non-latched side of the door under the metal holding lip, push the latched side flush, and tighten the latches. Once both doors are latched, return the key to the top of the gray electronics box on the NA2 side.

1.3.2 Right side

Laser rack

Power Relay

Main Breaker

PU 610

PU 620

MV 70

Water reservoir

Main pump

Aux pump

Heat exchanger

M75 chiller (water only)

1.3.3 Left side

Light / Louvers

Power

M33 chiller (water/PG mixture)

Allen Bradley unit

1.4 The ACS cabinet

The cabinet for the Absolute Calibration System (ACS) is located on a platform in the rafters of the intermediate level, between the hanging housing holding the NA2 Lakeshore controllers and the stairs to the observing level. It can be accessed from the intermediate level by stepladder - be sure to press a stop button next to where the ladder stands, since it has to be set up in the pink portion of the floor.

Laptop: acs-laser.apo.nmsu.edu

Cs clock control

PPS check

1.5 TBAD

The functional part of the TBAD system is located on the top end of the telescope above the secondary mirror, facing the sky. The power unit for the TBAD system is on the NA2 side electronics box on the front of the primary mirror cell. Additionally, the TBAD system uses telescope telemetry passed through a terminal server on the intermediate level.

See also Monthly: TSIM test under Operations

1.6 Interlock system and keyswitch box

The interlock system is controlled by a very low-level Allen Bradley "computer" which does not have a monitor or keyboard and which nobody is currently capable of reprogramming. The primary Allen Bradley unit is in the back left corner of the ILE on the floor [get photos to document the desired state of this thing!].

A secondary Allen Bradley unit is located in the 3.5m control room on a shelf under the desk in the left-hand corner; this unit must be turned on in order for the keyswitch box on top of the desk to work. In this picture, taken with the laser powered off, there are two red LEDs flashing, and a green "COMM" message illuminated on the readout.

The keyswitch box in the 3.5m control room is the primary interface for a human to control the interlock shutter which allows laser light out of Utah into the dome along the telescope's light path. This box does not control the laser power. If Utah is opened, laser light can shine into the dome regardless of the state of the interlock shutter.

When the red button is illuminated, like the image on the left, that means the interlock shutter is closed and laser light cannot travel through the telescope light path. When the green button is illuminated, like the image on the right, that means the interlock shutter is open, and ranging is possible.

In order for the interlock shutter to open, multiple conditions must be met, each matching a status LED on the keyswitch box:

The key must be turned on so the box has lights!

Laser power must be on (top LED "LASER AC PWR" must be lit solid red).

The stairwell door on the intermediate level must be closed (second LED "Dome Access" must be lit solid green).

Telescope motors do not need to be active (third LED "TELESCOPE STOP" may be either blinking or solid)

TBAD must be powered on, which requires dome open AND it must not see a plane within its pointing area (fourth LED "APOLLO CONTROL" lit solid green).

OR a bypass-dome file exists on cocoa (allowing TBAD to be turned on with dome closed), AND TBAD does not think it sees a plane inside the dome (so the TSIM box is not plugged in).

OR the top "BYPASS" toggle switch is up, so TBAD is ignored ("APOLLO CONTROL" LED will be blinking instead of solid). ONLY USE THIS FOR CLOSED-DOME WORK! NEVER BYPASS TBAD IF LASER LIGHT CAN LEAVE THE DOME!!!

The fifth LED for "SKY SENTRY" refers to an IR plane-spotting camera which is defunct, so its status (usually blinking green) does not matter.

The sixth and seventh LEDs for human spotters are no longer used but the interlocks are still active, so the bottom "BYPASS" toggle switch needs to be up. These LEDs should be blinking green.

If some condition caused the shutter to close previously - for example, if someone opened and then closed the stairwell door - the green button has to be pressed again to re-open the shutter after the interlock condition is resolved (after the door is closed).

The normal operating state for the keyswitch box is to have the top toggle down or not-bypassed (TBAD can close the shutter if it sees a plane) and the bottom toggle up or bypassed (human spotters are not present to look for planes).

If you look closely at the two example photos above, you can see the photo with the shutter closed has the toggles in the correct state. The interlock shutter will not open in this case because of two conditions: 1) the laser is not powered on, and 2) TBAD is not operating because the dome is closed. So the first and fourth LEDs are not lit, and the shutter will not open. The second photo, with the interlock shutter open, shows that the laser power has been turned on, so the top LED is lit solid. Also, the top toggle is in the up position so TBAD is bypassed. The fourth LED is blinking green (not obvious in the still photo) to indicate the bypass state.

A second keyswitch box is available in the blue-lidded APOLLO supplies bin on the observing level to allow the interlock shutter to be controlled by someone working in the dome (in which case they will definitely be wearing laser safety glasses, also available in the blue-lidded bin.) This keyswitch box attaches to a cable which emerges by the floor grating just below Agile; look for the tape with arrows to align the connectors with each other:

The upstairs keyswitch box has the same conditions as the downstairs box in order to open the interlock shutter, plus two additional conditions: the downstairs box needs to be turned on in order for the upstairs box to work, and also the upstairs box has to be in a good mood because it's flaky and sometimes it just doesn't want to work. You may need to coordinate with someone downstairs to open the interlock shutter for you.

2. Software

2.1 ATUI

ATUI is the APOLLO control interface inside TUI, equivalent to an instrument control window and accessible from the Instrument menu for APOLLO-enabled TUI installations. But in fact ATUI is a complex of multiple control windows, each accessed by different tabs along the top of the window.

2.1.1 Tabs along the top

Active/Main Tab

The first tab is listed as "Active" in some versions of ATUI and "Main" in other versions. In this document I mostly call it the Main/Active tab. Some ATUI versions have a bug where the name of the tab disappears anyway, so the name is not important, but the tab is. This is where most of the ranging work takes place.

Take Control

The Take Control button in the top right of the Main/Active tab is like giving yourself permission to control things through ATUI. When pressed, the label on the button changes to say Houston: Active Control. You can press the button again to rescind control. The primary purpose of this button is just to avoid accidental button-presses. More than one person can have control at a time, in which case they should be coordinating with each other to avoid command collision.

StartNubs / Listen / ConnDev / Get Status

These four buttons in the middle of the Main/Active tab are pressed, in order from top to buttom, to connect ATUI to the housctl and ICC programs and make sure those programs are talking to each other and the instrument. When planning to run anything from ATUI, you will usually start with these four buttons. If you or someone else did all four buttons recently and then disconnected, you may only need the Listen and Get Status buttons to re-connect your own ATUI without going through the rest of the initialization. If you were disconnected for longer than a couple of hours, it's safest to just use all four buttons.

Bottom section

The bottom half of the Main/Active tab has buttons, number entry boxes, and sliders which are primarily used while ranging. Some of these settings will be changed frequently, and some of them should never be changed; see the image below for reference and be careful!

Log output and houston command line

In the middle of the Main/Active tab is a command line which highlights with a cursor when you click on it, and just above that is a window showing 7 lines of recent output, where you can scroll up to see previous output. This is effectively a TUI Log window where the output is filtered to show only housctl messages, and the command line sends commands directly to the actor "houston" which is the housctl program. You can change the severity filtering and search for text with the "Replies" and "Find" controls just above the mini-log.

Run Statistics

The Run Statistics section at top left of the Main/Active tab has information about the successfulness of the most recent ranging run. You will keep an eye on some of these numbers during ranging to see if you need to change anything. Then, after each run, you will copy some information from this section into the paper observing log. Be sure to grab whatever information you need before starting the next run!

The only control button in the Run Statistics section is a button saying "Reset Everything for New Run." This used to be needed in an older version of ATUI to refresh the plots and statistics, but it's mostly defunct now and should not be needed. You can experiment with it sometime in between runs if you want to try.

Offsets

The Offsets section in the top right of the Main/Active tab includes controls for centering up targets, keeping them centered, searching for signal, and keeping the signal. Some of the controls are non-intuitive for observing specialists.

Four offset types are possible, chosen with mutually-exclusive checkboxes. It's important to make sure you are moving the right thing!

Boresight offsets are used for initial telescope centering before ranging begins

Raster Scope moves the telescope with Guide offsets, used during ranging to see if tracking problems accumulate over time.
Raster Optics adjusts the position of the receiver (aka "rx") optics; this is the "shoot ahead, look behind" adjustment that compensates for the moon's motion while the light is traveling to and back from the moon. Moving these optics will cause the moon to shift position on the guider so it's like offseting the telescope, but it also has an effect on signal strength.

Raster Beam adjusts the receiver optics and also offsets the telescope in the opposite direction. The ideal is that you get signal centered up, and then you can use Raster Beam to optimize signal strength and not worry about having to follow the tracking at the same time. In practice, it isn't quite simple. Raster Beam might be useful for certain tests, but Russet generally prefers to Raster Optics by themselves and then adjust the telescope position separately.

Once you have decided what kind of offset to use, the size of the offset is set in the Offset Increment box. Enter the desired value and then press return to make the pink go away. The maximum offset permitted through this interface is 10 arcsec, generally used for Boresight offsets during the initial centering. While ranging, the Offset Increment will usually be less than 1 arcsec. Moves larger than 1 arcsec will be Computed offsets with sounds and a countdown in TUI. Smaller offsets will be uncomputed and basically instantaneous (but remember the round-trip time to the Moon is about 3 seconds, so even an instant offset takes a while to change the returns!).

The three checkboxes on the right determine what coordinate system or "frame" the arrows will move in.

"Native" is the natural alt/az coordinate system of the Guide or Boresight offsets, where the > button increases az or X by Offset Increment, and the ^ button increases alt or Y.

"APD" is the frame of the APD science detector, displayed in the hitgrid at the top of the Plots window; the > button pulls the signal to the right on the hitgrid, and the ^ button pulls the signal up.

"CCD" is the frame of the STV guide camera, displayed in a browser window; the > button pulls the star or crater to the right, and the ^ moves the star or crater up.

Fun fact: Offseting in the wrong frame is the most common mistake to make while ranging!

If you get lost, you can use the Home button in the center of the arrows to return the Guide offset to 0, 0 (if Raster Scope is checked) or the Boresight offset to 0, 0 (if Boresight Offset is checked). If you accidentally moved the rx optics, the Go To Target button will bring them back to the desired shoot-ahead-look-behind position. Please use care not to press the "Commit Rxx, Rxy" button unless you're really sure what you're doing (described in the Operations section.)

Pointer Tab

Show / Hide craters

Numbers table

Graphic Interface

Month / Day / Hour / Minute forward and back

Slew Telescope (always for Now)

Polynomials and Vtarget

Plot Control Tab

The Plot Control tab is somewhat useful, but has bugs. Most notably, when you enter a number in a box and press return, the number will stay pink or perhaps change briefly to white and later back to pink. Some of the numbers seem to have no effect when changed, but others do take effect even though they stay pink. Some of the labels are incorrect, for example the APD Hitgrids Time Window says it shows a certain number of hits which implies incoming photons, but in fact it's showing the hits that came in during the displayed number of outgoing shots.

The default settings are reasonable for strong return signals and it's okay to leave this tab alone completely. In the case of weak signal it may be helpful to increase APD Hitgrids Time Window from 200 to 400 and the Return Time Histograms window from 200 to 800. Those are Russet's preferred settings.

Alarms Tab

The main body of the Alarms tab is like a TUI log window which shows APOLLO-specific warning messages of a certain severity level ("Error" by default). The severity level appears to be adjustable but may not actually change in some versions of ATUI. Use this window to check on error messages which went by too quickly to catch. Once you have decided if a message is or is not a problem, you can press the Acknowledge Alarms button which should cause the name on the Alarms tab to change from red (or invisible) to black, until another error message comes through.

Below the main section is another section dealing with blocks from space command. If there is no block, it will say "Time remaining in release: ### s" in green (or in yellow if there's only a short time remaining). If there is a block, it will say "Time remaining in blockage: ### s" in red. If no blockfile exists, it will say there are many nines of seconds remaining in the blockage. Whether there is currently a blockage or a release will also affect the color (or perhaps visibility) of the tab for Main/Active.

If there is a block currently in place, you will not be able to open the interlock shutter (green button on keyswitch box). Put a blockfile with valid times in houston:/home/apollo/daily/housctl.blk and press the readblock button on the left side of ATUI, or use the "unblock #" command to permit opening for a certain number of minutes.

Raster Tab

The raster tab is an attempt at automating a search raster which was never fully implemented and probably doesn't work. Best leave it alone.

STV Tab

The STV tab is mostly a visual display of the physical control box located in the observing level cabinet. This interface is used for controlling the APOLLO guide camera. [Add a link to an SBIG manual maybe?]. There are some additional buttons at the bottom of the page which haven't been tested and maybe don't do what you think they do. Or maybe they do.

Channels Tab

The Channels tab is very simple: A 4x4 checkbox grid, and a 4x4 grid of numbers. This is a reference and control for the hitgrid plot in the Plots window. If it seems like a particular pixel of the detector is always hot, you can come to this tab to identify which number channel that pixel is; for example, the bottom-left corner pixel is known as "channel 12." Then you can uncheck the misbehaving pixel in the grid of checkboxes so that its output will no longer be displayed in the hitgrid and won't make it impossible to see what the other pixels are seeing. Note that unchecking only affects the display of the hitgrid, it does not actually turn off one of the pixels of the detector. So if you do see misbehavior, you should also report which channel is misbehaving to someone who can do something about it.

Power Tab

The Power tab is just a bunch of checkboxes which allow you to turn on and off individual components of the APOLLO systems. This is generally only used for testing or troubleshooting. During normal operations, groups of components will be turned on and off at the same time as the system state changes from Idle to Warmup to Run to Standby to Cooldown, and so forth. There are also automated environmental checks in the housctl program which may turn heaters or fans or chillers on and off as needed. It's usually best not to mess with them unless you're trying to unstick something that's stuck, such as the GPS clock (checkbox #24 on the list).

Laser Tuning Tab

The Laser Tuning tab is used to control the main APOLLO Laser, also known as the Leopard laser.

ROOT Tab

The ROOT tab is used for connecting to and disconnecting from the separate python program which runs the Plots window. You will need to press the Connect to ROOT button before opening the Plots window, and press the Disconnect button before you try to close the Plots window. If you forget to disconnect, you might have to force-quit the python program.

ACS Tab

The ACS tab is used to control the Absolute Calibration System.

2.1.2 Control/Status buttons down the left side

Idle / Warmup / Standby

Run

Fidlun

Caltdc

Stare / Dark / Flat

Lpower

Lasercal

Cooldown

TR sync / clear / dark

Readblock

Spotter 1 / Spotter 2

2.1.3 The Plots window

[This is the latest I have: After clicking the CONNECT button in the Apollo/ROOT tab, in Terminal window: python3 Apollo_Plots_ROOT.py twm2.json. Need to know if this is going to be stable - what about directories?]
The plots that are most useful for ranging are on the left side of the window, to make it easier to observe with overlapping windows. Some of the other plots are useful during checkout and troubleshooting.

2.2 housctl

housctl is half of the APOLLO control program, which runs on the machine called houston. housctl takes input commands from the hub and sends them to various APOLLO subsystems, and housctl also controls many of the subsystems including environmental controls and the gps clock. If the housctl program is not running, ranging will not be possible and also the big Leopard laser could be in an unsafe state with either cooling or heating running without feedback control.

To check that housctl is running properly, login to observer@houston and run
$ ps aux | grep housctl

There should be four or five housctl processes running, depending on who else is connected. If more than six processes are running they may get hung up. If one or no processes are running, housctl is not working properly. In either event, housctl will normally be automatically restarted within a few minutes by a cronjob on houston, which also sends an alert email saying that it needed to do a restart.

However, sometimes a human needs to restart housctl, maybe for a software update or if housctl is running but not communicating correctly with the ICC program on cocoa. In this case, try to kill the old housctl process and restart the new process within a few minutes so that you do not collide with the cronjob which might try to restart housctl at the same time.

To kill the old housctl, login to root@houston and run:

$ ps aux | grep housctl

Identify the PID for the oldest process (lowest numbered, first on the list).

$ kill [PID]

This should kill all of the housctl processes running. Check to be sure:

$ ps aux | grep housctl

To restart housctl, working as root@houston:

$ cd /home/apollo/

$ nohup ./housctl &

$ ps aux | grep housctl

Get the PID for the oldest, lowest-numbered process (also make sure there are only five running, or you might have to kill and start over!)

$ renice -20 [PID]

2.3 ICC

The ICC is the other half of the APOLLO control program, which runs on the machine called cocoa. It handles the lunar ephemeris, science data, and interlocks. If the ICC is not running or if it is not communicating properly with housctl, it will not be possible to do ranging.

The most common symptom that the ICC is either stopped or not communicating with housctl is that when you initialize your TUI session with StartNubs / Listen / ConnDev / Get Status, you will see multiple error messages saying "NoneType object has no attribute 'call'." Sometimes the error will be "the target named apollo is not connected." If there is a single example of the error, you might be okay, or you might be safer restarting. If there are multiple error messages, you definitely need to restart. Most often it will be best to restart the ICC followed by restarting housctl, to make sure that both programs are talking to each other.

First check whether or not the ICC is running. Login to observer@cocoa and run
$ ps aux | grep -i icc
      If the ICC is already running, look for more signs that it is not running properly. If you are confident that it is not, use the PID to kill the process. Do not restart the ICC program if a copy is already running.

To restart the ICC, login to root@cocoa or, if already logged in, su root.

$ export DISPLAY=0:1

[NOTE: As of 2022, this command is for the physical cocoa machine. If the virtual cocoa is running, use 'export DISPLAY=:0' instead.]

$ cd /home/tmurphy

$ ./icc &

$ ps aux | grep -i icc

Sometimes it will be sufficient to restart only the ICC, but most often you may find it necessary to restart housctl right afterward to make sure the connection between the two programs is correct.

Got a traceback? Some solutions are offered in the Troubleshooting section.

2.4 Miscellaneous scripts

2.4.1 TBAD monitor

The TBAD monitor script on cocoa logs the state of TBAD. If it is not running, the TBAD system can still operate and close the interlock shutter when a plane moves close to the laser beam, but the output of TBAD will not be logged and checkable. Once a day at 17 UT, a cronjob checks the status of the TBAD system, and if the monitor script is not running there will be an email alert.

After a reboot of cocoa, the monitor script may need to be restarted. To restart the TBAD monitor script, login to root@cocoa and
   $ cd /home/apollo/xponder
   $ ./xp_monitor.py &

After this, logging should resume. You can run a TSIM test to confirm.

2.2.2 Webpage server

cocoa hosts some webpages with important information about the status of the APOLLO systems, including two-day plots of temperature / coolant flow / GPS clock behavior and also tables showing all the temperature sensors, what systems are on, and error messages from housctl. The tables update every 10 seconds.

Direct access to these webpages is at Tables and Plots. However, these links are not open to the world and may only be accessible from onsite, in order to avoid burdening cocoa with web-server duties when its primary job is handling APOLLO data. Russet McMillan and Tom Murphy have been able to reach these pages with a tunnel using ssh forwarding, but other people from UCSD and NASA have been unable to make the tunnel work.

Therefore, the pages are echoed on the regular APO website here: Tables and Plots. This copy of the pages updates once per minute, and might need to be refreshed by hand in some browsers.

After a reboot of cocoa, the web hosting may need to be restarted. To restart the webpages, login to root@cocoa and
   $ cd /home/environment
   $ python htmlmonitor.py &

2.4.3 Generating quarterly schedule request

/home/apollo/ephem/pytools/longplan.py 04 01 2023

2.4.4 Generating notification files and emails

2.4.5 Generating crater slew commands

3. Operations
3.1 Periodic Communications and Checks
3.1.1 Summer of odd years: get new Letter of Determination from Laser Clearinghouse

Every few years (next up in July of 2026), renew the Letter of Determination with FAA. This is really a job for Nancy Chanover and/or Stephen Merkowitz, but the operator should be aware of it. The date of the LOD listed in the quarterly email to FAA should be updated when a new letter comes into force. Do not operate if the LOD is expired.

3.1.2 Quarterly: communicate with FAA and Space Command

Before the beginning of a new quarter, run the notify.py script on the APOLLO instrument schedule for the upcoming quarter.

Input file is created by going to the quarter schedule page and scrolling down to Instruments, APOLLO, click on Schedule, copy the table into a text file with a name like 2021_Q1.sched. Example file:

./notify.py 2021_Q1.sched

Example file to go to FAA:

Output files include 2021_Q1.faa (this has all the ranging sessions scheduled for the quarter, but FAA actually wants to be informed about one month at a time), 2021_Q1.csv (to be uploaded to space-track.org website), and 2021_Q1.ssc (defunct).

Before the beginning of the new quarter, log in to https://www.space-track.org/, go to Files - Upload, select destination /zLCH/Apache Point/PRM , then use Choose Files to find the .csv file on the local machine to be uploaded.

Before the beginning of a new quarter, copy the addresses from the .faa file into an email, subject line like 'Laser ranging at APO, March 2021', copy the email content from the .faa file but then remove the lines that are not for the correct month, and send email.

3.1.3 Monthly: TSIM test
Approximately once a month, test the TBAD plane transponder system using the TSIM aka "fake plane" in the dome.

With dome closed, point telescope at 20 deg alt. Lights on/off do not matter. Covers open/closed do not matter.

Take an extension cord into the pit in front of the telescope. There is usually an extension cord on the bottom shelf of the NICFPS cart or the bottom of the NA2 cabinet. Plug in the extension cord beneath the dome controls or beneath the phone, then run it over to the seam between the dome doors. At the bottom of the left dome door is a gray box with an amber warning light and a very short power cord. Plug this into the extension cord. The amber light should come on. This will not work with the dome open.

Open up the gray box (be careful, the latch likes to pinch fingers!) and look inside. On the top right side there is a toggle switch, which should be On, and a green LED that says "transmit" which should be flashing. It may flash quickly or slowly; either is normal. Confirm the switch and the flashing LED, then close the box again.

On a computer in the APO network:

$ ssh observer@cocoa

$ cd /home/apollo/xponder

$ touch bypass-dome
If this file exists in the directory, the TBAD system will turn on with the dome closed. Normally TBAD turns off when the dome is closed.

Wait at least 90s with the box plugged in AND the bypass-dome file existing, for a full test cycle.

$ rm bypass-dome

Unplug the gray box and return the extension cord to its proper place.

The TBAD output is stored in /home/apollo/xponder/log/ under a file named by the UT date. Optionally, you can check this file to see if the system is behaving normally. It should look something like this:
2024-04-16 03:29:57.412 s4243..BHFX.F2  ----- TSIM     104.00  20.00 B
2024-04-16 03:30:05.412 s4143..BHFX.F1  ----- TSIM     104.00  20.00 B
2024-04-16 03:30:07.295 o8888...H...AB  ----- OPEN     104.00  20.00 B
2024-04-16 03:30:08.007 o4141..BHFX.EB  ----- TSIM     104.00  20.00 B
2024-04-16 03:30:08.162 o4143..BHFX.ED  ----- TSIM     104.00  20.00 B
2024-04-16 03:30:08.318 o4145..BHFX.EF  ----- TSIM     104.00  20.00 B
2024-04-16 03:30:08.474 o4147..BHFX.F1  ----- TSIM     104.00  20.00 B
2024-04-16 03:30:08.630 o4151..BHFX.EC  ----- TSIM     104.00  20.00 B
2024-04-16 03:30:08.786 o4153..BHFX.EE  ----- TSIM     104.00  20.00 B
2024-04-16 03:30:08.942 s4155..BHFX.F4  ----- TSIM     104.00  20.00 B
2024-04-16 03:30:09.098 s4157..BHFX.F6  ----- TSIM     104.00  20.00 B
2024-04-16 03:30:09.254 s4161..BHFX.F1  ----- TSIM     104.00  20.00 B
2024-04-16 03:30:09.410 s4163.DBHFX.09  ----- TSIM     104.00  20.00 B
  
The first few columns after the timestamp indicate the "fake plane" changing the strength of its signal as it goes through the test cycle. You want to make sure that almost every line says TSIM, as shown here, and that there are no clusters with many OPEN lines in a row. A few intermittent OPEN lines are a normal part of the test. If you want, you can look at this log file sometime when the dome is open (APOLLO does not need to be in use), and you should see some real plane detections with transponder and location information. Remember that TBAD will not detect all planes, but only the ones near where the telescope is pointing.
2024-04-16 05:02:38.403 o3670...HF.FCB  -----          35.90  50.43 O
2024-04-16 05:02:38.414 o8DADB35C58AF81E3AA99E96729CD...HF..77 DF-17 35.90  50.43 O
   DF-17: PPass, ID ADB35C, Alt 34000, Lat=32.83397, Lon=-105.83564
2024-04-16 05:02:38.414 o1524...HF.FC7  34000          35.90  50.43 O
2024-04-16 05:02:38.436 o3670...HF.FCB  -----          35.90  50.43 O
2024-04-16 05:02:38.442 o8DADB35CEA428864015C080AF03F...HF..38 DF-17 35.90  50.43 O
   DF-17: PPass, ID ADB35C, MCP alt: 34016; Hdg: 0; 213.6 mbar; TCAS
2024-04-16 05:02:38.470 o3670...HF.FCB  -----          35.90  50.43 O
  
    
3.2 Preparation and Warmup
3.2.1 Few days before ranging: Request NOTAM
A NOTAM is a "NOTice to AirMen" which warns pilots to stay away from a particular area of sky. Our Letter of Determination requires us to issue a NOTAM for each ranging session. FAA has approved APO as a "station" to issue NOTAMs, but it can only be done from a browser running Microsoft Edge in Internet Explorer 7 mode, which means it has to be on a machine running a particular version of Windows. We have a virtual machine set up to do this, but communicating with the machine requires either being onsite or connected to APO by VPN.

A few hours up to a week before a ranging session (if the weather is anything better than hopeless), request a NOTAM by this method:

Use Safari or a VNC tool to connect to vnc://10.75.1.216 (first-guess password for screen sharing)

If you need to login to the machine, the password is on the list.

There should be a Microsoft Edge browser already open on the desktop; if not, open it.

The browser should already be connected to the NOTAM interface; if not,

In the top-right of the browser window, click on the "..." menu icon

Near the bottom of the menu, choose Internet Explorer Mode

Send the browser to https://nes.notams.faa.gov/nesv3/index.html

Tell the popup window to accept the certificate.

For login, the station name is APO-LASER and the user is RM. Password should be automatically remembered, also on the typed list, but it may change quarterly

Pro tip: if you login, and then as soon as you try to do something it asks you to login again, it probably means you are not in Internet Explorer mode!

From the menu on the left of the browser window, select Draft Notam. You should get a big text entry box.

Click on the Apply Template button above right of the entry box, which fills in almost all the info, then edit the date and time:

Go to the bottom of the text where it says YYMMDDHHMM-YYMMDDHHMM

Convert the schedule for the upcoming APOLLO run to UT date and time, with some padding at the start and end.

Enter the padded start/end datestamp in place of YYMMDDHHMM-YYMMDDHHMM

Below the text entry box, click the Preview button

Make sure there are no red error messages that need to be addressed. It is normal for ^^ to appear under the date.

Press the Return to Draft button.

Now press the Submit to USNOF button and say yes, you really want to.

A list will appear showing the work number; note the work number.

A few minutes to an hour later, click on NOTAM List on the left-side menu to make sure it was approved.

Example NOTAM request:
ZAB NM..AIRSPACE SUNSPOT, NM..LASER RESEARCH WI AN AREA 
DEFINED AS 5NM RADIUS OF 324649N1054913W (BWS098010) SFC-FL600. 
APACHE POINT OBSERVATORY. AT A TYPICAL ANGLE OF 45DEG, 
FM THE SFC, PROJECTING UP TO FL600 AVOID AIRBORNE HAZARD BY 5NM. 
THIS BEAM IS INJURIOUS TO PILOT'S/AIRCREW'S AND PAX EYES. 
ALBUQUERQUE /ZAB/ ARTCC TEL 505-856-4500 IS THE FAA CDN FACILITY.
2303250110- 2303250230
3.2.2 Hour before ranging: Set up observing log

Every APOLLO ranging session since 2005 has been recorded on paper logs. Maybe someday this will be modernized, but that day is not today.

Get a blank log form or print one out from here.[It's a PDF; link works in Chrome and Safari but not in Firefox?]

Go to the Pointer tab of ATUI (you do not need to initialize anything else inside ATUI, or even connect to the hub).

Press the Now button below the Moon graphic to make sure the ephemeris information is current. Then use the Hour++ and Min++ buttons to advance to approximately the time when you will start ranging.

Record the Phase, Libration (X,Y and total) and reflectors lit (1) or unlit (0) on the top line of the page.

Record the UT date (use the after-midnight date for winter sunset runs that start before 0 UT), APOLLO observer, Telescope obs-spec, and sky conditions on the second line.

Dates and notes for various warmup procedures will be written on lines of the log without regards to the column labels; when you start ranging, then notes for each individual run will be written according to the column labels. See also the section on Logging results.

Here is an example observing log from a successful run as a PDF, or you can check out this image of the log next to how the pointer window looked before the run started:

3.2.3 Hour before ranging: Prepare ATUI and other windows

Many windows are needed to operate APOLLO:

A terminal window for opening ATUI and plots

Another terminal window for ssh to houston and cocoa

A browser window for viewing the instrument status and temperatures

Another browser window for viewing the guider video, which needs to be in a fixed position once the crosshairs are set.

A VNC window for controlling the acs-laser laptop

Regular TUI with, at minimum, Status and Chat and Slew and Sky windows

ATUI which is a whole bunch of windows combined into one

The ROOT Plots window, which may refresh itself or may spawn a new window with every run.

So, you'll want to get started early to set these windows up!

python3! Here are instructions for using the "native" install, which should be included in TUI for all obs-spec accounts on obsA as of summer 2024.

Start TUI normally.

Open ATUI by selecting "Apollo" from the Inst menu. If Apollo is not listed in the menu, you can try the docker version or use the Arc account on obsA.

In the ROOT tab, press the press the Connect to ROOT button.

In a Terminal or Xterm window, enter:

cd /Library/Application\ Support/TUIAdditions/Apollo/
Pro tip: on personal MacOS machines, you may need to use the ~/Library directory tree, or it may be more individualized!

python3 Apollo_Plots_ROOT.py twm2.json

Here are older instructions for starting the python3 version of ATUI through a docker container, as set up in April 2023.

Make sure you already have ATUI-via-docker installed in your account on obsA or whatever computer you are using!

Open XQuartz (hopefully on dock, else look in Applications/Utilities)

Open Docker (hopefully on dock, else look in Applications). Docker desktop will launch.

Open a terminal window. In that window, enter:

xhost +

docker start atui3 [This will launch TUI with the ATUI window. Possibly you can also do this with the play-triangle button in the Docker desktop window.]

docker exec -it atui3 /bin/bash [This gives you a new command prompt.]

cd /workdir/python/TUIAdditions/Apollo

Now you can [optionally do some other things and then] go the ATUI ROOT tab and press the Connect to ROOT button. After pressing that button, you can't do anything else with TUI or ATUI until you open the plots window:

Back in your terminal window: python3 Apollo_Plots_ROOT.py twm2.json

Now, get your ATUI connected. Start in the Main/Active tab.

At top right of window, press Take Control. The name of the tab will change from Main to Active at this point.
In middle of window, press StartNubs then Listen then ConnDev then Get Status

Wait 15s for Get Status to finish.

If you see an error (in the small log excerpt, or in the main TUI log window, or in the Alarms tab) saying "NoneType object has no attribute 'call'" that occurs once or a few times after the StartNubs, but stops recurring by the time you get to Get Status, then you're okay. If that error message continues to show up or you see another message "the target named apollo is not connected," then you may need to restart the ICC and/or housctl. Go do that, then come back to ATUI and try the four middle buttons again.

Note: if someone else already did the initialization using the four middle buttons, or if you disconnect from ATUI and come back a little while later (without restarting anything), you may only need to repeat the Listen and Get Status buttons. If you come back many hours later or if you restarted any software, repeat all four of the initialization buttons.

The status indicators along the left side should indicate that the system is IDLE.

Press the warmup button below the IDLE indicator. Note: In summer, you might want to delay this step until you are ready to do the ACS checkout!

Record the time of warmup in your observing log.

Wait 30s for all the warmup activities to finish, then switch to the Alarms tab to see if there is anything new. If all seems well, you can press the Acknowledge Alarms button to get the tab name back at least for a while.

You are now ready to proceed with other checks and warmup procedures.

3.2.4 Hour before ranging: check GPS clock
The GPS clock provides a reference for the more-precise Cs clock, so we prefer to have both clocks working. The GPS clock gets periodically jammed by exercises at nearby military installations; sometimes it manages to unjam itself, and sometimes not. So it should be checked and if necessary woken up more than half an hour before ranging begins (because it needs some time to settle after unjamming.

The simplest way to check the clock is to go to the webpages accessible through the Plots webpage and scroll down to the "GPS Clock Properties" near the bottom of the page. Make sure the timestamp at the right of the graph is the present time (yes, they are hard to read). If it is not, refresh the page (you may need to hold the shift key and press refresh) and make sure the blue line on the graph is not flatlined. If the graph is flat, you need to wake up the clock.

The clock can be woken up either through ATUI (preferred method), or through direct input to housctl.

To wake up the clock through ATUI, go to the Power tab:

Switch 24 (labeled GPS_clock) should have a checkmark

Uncheck switch 24 (Bug workaround tip! Checkbox not working? You can instead do "power 24 0" in the command line of the Main/Active tab)

Wait one minute, then recheck switch 24 (or "power 24 1" in the command line)

Wait one minute, then refresh the Tables webpage; it should show numbers for gpstime, phase, and offset

Wait one minute, then update the Tables webpage; it should show new numbers for gpstime, phase, and offset

Go do something else for a while, then after half an hour check either Plots or Tables to make sure the clock is continuing to settle.

Here's what the plots for the clock look like when the clock has been stuck for a few hours (left), and then what it looks like a few minutes after being reset (right). Because of the huge jump in scale you can't see any details on the "after" plot, which means you have to look at the numbers on the tables webpage instead to see if small changes occur with each update.

As a backup method, you can use direct access to housctl with a login to observer@houston. This is not the best way to reset the clock, but this access can be useful for other things, so the method is described here:
$ telnet houston 5320
This gives you a direct interface to housctl so you can see log output and send commands; only one person can do this at a time, and be sure to disconnect when you are done!
Inside the telnet interface, watch the log output for lines every 10s that look like
    gps0 gpstime="F03  UTC  04/13/23 08:33:12"; gpsdac="F71 phase= 1.377E-08 s  offset= 3.490E-12  drift= 4.783E-12/DAY  DAC= -5439"
Are the phase and offset changing with at least every other report / every 20s? If so, the clock is fine. If not, you can try resetting it with a warmstart (more gentle) or with a power cycle (more likely to work).
For the warmstart:

Enter 'gps warmstart' directly into the telnet interface

Watch the clock output for the next minute or two to see if the phase and offset change and keep changing. If not, try the power cycle instead.

For the power cycle:

Enter 'power 24 0' directly into the telnet interface

Wait 30s

Enter 'power 24 1'

Watch the clock output for the next minute or two to see if the phase and offset change and keep changing.

Exit the telnet interface with ctrl-]

Note: You can also use the 'gps warmstart' method with the command-line interface in ATUI! But it's awkward because the window with the log murmur is very small and doesn't always show everything, so you might have to look at the webpage for clock phase and offset instead, and the webpage only updates once a minute. But it is an option.

If none of these methods works to unstick the clock, that may mean active jamming is still going on. Check back in half an hour and try again. You can try ranging even if the clock is stuck, but you may find no signal or find the signal at higher predskew than expected, or find that the predskew keeps changing. Any of these symptoms may mean that the ranging results are unreliable.
3.2.5 Hour before ranging: generate polynomial files
The polynomial files report the estimated distance of the Moon using a moderately complicated lunar orbit model. This is important since we only check a short window for the laser light returning from the Moon (about 40 cm of distance), in order to then refine the distance to 1 cm or less. So we have to start out with a pretty-good prediction.

First, get yourself oriented by going to the Pointer tab of ATUI, and use the Hour++ and Min++ buttons to check the Moon's altitude at the start and end times of your scheduled ranging. If the Moon will be below 20 degrees for part of the run, you will need to calculate the polynomials for a lower altitude. The polynomial predictions get less accurate as you go lower from the default minimum of 20 to the absolute minimum of 10, so make a note of the minimum altitude you really need.

The polynomial files make use of earth orientation data and predictions, so the files should be built no more than one day in advance of ranging, using the latest earth orientation measurements. As of April 2022, the scripts are unable to get the earth orientation data automatically, so you will have to obtain that information before you calculate the polynomials:
Login to observer@houston

$ cd /home/apollo/ephem/predict

This method uses no editor, but still needs cut-and-paste:

copy full contents from USNO website

$ cat >gpsrapid.daily

paste text

newline

ctrl-C to exit

OR emacs -nw gpsrapid.daily

delete old contents [ctrl-@ at top, esc-> to go to bottom, then ctrl-w]

paste in full contents from USNO website

make sure file ends with newline

save file; exit editor [if emacs, ctrl-x ctrl-s then ctrl-x ctrl-c]

make a directory with UT date name, such as $ mkdir 2023-03-25

cd to new directory

make polynomials with a start time at least a few hours before ranging.
Command: ../mkpoly YYYY MM DD hh mm s.s {alt}
where the last argument is a optional altitude between 10-20 degrees.
Example commands:

../bin/mkpoly 2023 03 25 00 00 0.0 [if session starts well after 0 UT]

OR ../bin/mkpoly 2023 03 24 20 00 0.0 [if session starts near 0 UT]

OR ../bin/mkpoly 2023 03 24 20 00 0.0 13.0 [if Moon is below 20 deg altitude during the ranging period]
output should end with lines like these:
 
Count for reflector [0, 1, 2, 3, 4, ctr] = [95, 95, 95, 95, 95, 95]
Congratulations, Fits created successfully
numbers in brackets should be in the range 40-120
If lower than 40, rerun with earlier start time or lower altitude limit

If higher than 120, rerun with later start time

Pro tip: If it claims moon is never above altitude limit when you're sure it is, make sure gpsrapid.daily file does not have spurious lines or characters! Delete all files from the date-specific directory, then try again.

copy polynomials to daily directory:

$ cd /home/apollo/daily

$ rm PolysRefl*

$ cp ../ephem/predict/2023-03-25/PolysRefl* .
The name of the polynomial file in use will be shown in the Active and Pointer tabs of ATUI, but it may be colored red or say 'None' until you slew to the position of one of the reflectors on the Moon. After you slew to a reflector, if it still says 'None' or is red, that probably means the correct files are not in the correct place.
3.2.6 Hour before ranging: check space block files
Space Command, also known as "Laser Clearinghouse," is supposed to tell us when we can range without hitting a satellite. APOLLO is not capable of causing harm to satellite structure, but some satellites might have sensitive cameras looking down at the Earth that don't want to see a bright laser.

The normal procedure is that we send Space Command a schedule of when we plan to range; we will basically always be pointing at the Moon when we send the laser into the sky, so that's what is listed in the files we send them. These files are called "Predictive Request Messages," or PRMs. In response, Space Command creates "Predictive Avoidance Messages" or PAMs, which indicate blocks of time where the laser is and is not allowed to shine the laser at the Moon. Usually the block periods will be short (seconds to a couple of minutes), except sometimes when the Moon is near -5 degrees declination, there might be a slow-moving geosynchronous satellite passing near the Moon, so there might be multiple blocks of many minutes.

Since approximately 2019, Space Command has not been generating PAMs in response to the PRMs that we send them. Nevertheless, we should check each time to see if there are any PAMs available. To do this, first login to Space-Track.org [Note: we need to create a group account for this...]. From the top menu bar, choose Files - Download. It will show you a directory tree ending in the directory zLCH; click on the > to open that directory and wait a few seconds. Click on the > for Apache Point and wait a few seconds. Click on the > for PAM and wait a few seconds.

Does it say "-- Folder Empty"? Then go to ATUI Main/Active tab and in the command line at the center of the window, input 'unblock 90' to allow ranging for the next 90 minutes, or else adjust the time to the approximate end of the scheduled APOLLO session.

Does it have a file or files in it? Look for one with the current UT date and the word "moon" in the filename; click on that. Copy the portion of the file which looks something like this:
YYYY MMM dd (DDD) HHMM SS    YYYY MMM dd (DDD) HHMM SS      MM:SS
-------------------------    -------------------------    -------
2023 Mar 07 (067) 0651 00    2023 Mar 07 (067) 0658 13    0007:13
2023 Mar 07 (067) 0658 24    2023 Mar 07 (067) 0659 09    0000:45
2023 Mar 07 (067) 0659 13    2023 Mar 07 (067) 0710 16    0011:03
2023 Mar 07 (067) 0710 20    2023 Mar 07 (067) 0727 28    0017:08
2023 Mar 07 (067) 0727 31    2023 Mar 07 (067) 0829 00    0061:29
  
Login to houston and update the block file:

$ ssh observer@houston

$ cd /home/apollo/daily

Easy method with no editor:
$ cat >housctl.blk
paste in what you copied from the webpage
press return to end with a newline
ctrl-C to exit

OR use an editor:
$ emacs -nw housctl.blk

Delete the old info from the file

paste in what you copied from the webpage.

Save and exit the file [ctrl-x ctrl-s then ctrl-x ctrl-c]

In ATUI, press the "readblock" button on the left side to read in the file you just updated

Now if you look at the Alarms tab, it should either say "Time remaining in release" in green or it should have a fairly short "Time remaining in blockage" indicating that you will be allowed to range at the scheduled time. Additionally, depending on ATUI version, the color or visibility of the tab for Main/Active may depend on whether or not there is a space block in place.
3.2.7 Half hour before ranging: check out ACS

The ACS or "Absolute Calibration sytem" is a second laser with lower power and a much faster pulse frequency (80 MHz) than the main laser. When the ACS laser is properly in phase with the Cesium clock, its fast pulses can be fed into the data stream for the lunar returns to provide a "comb" for calibrating the timing to very high precision. A lot of the checkout steps have to do with tuning the ACS pulses to match the clock, for instance by changing the temperature of a fiber-optic coil in order to change the path length. But for a lot of these processes Russet doesn't really understand what's going on, so some of these instructions are along the lines of "turn this knob until A lines up with B, and then you're good!"

Note: If it's summertime and you chose to delay pressing the WARMUP button, do that now before you start working with ACS! You can check the description in section 3.2.3 for how WARMUP should proceed.

Start in a terminal window:

$ ssh observer@cocoa

$ cd /home/apollo/ACS

$ ./ip-pwr.py
This shows the status of 5 switches; all should be on except #2

$ ./ip-pwr.py pset 2 1
Next, go to the ACS tab of ATUI and check the box for PicoFYb DC Power at top right.
Open a VNC viewer (Safari browser is recommended, VNC Viewer may not work). Please note the machine you are connecting to will only accept one screen sharing connection, and may get into a bad state if two people try to connect at once.

Connect to vnc://acs-laser.apo.nmsu.edu:5959

It will ask for a password.

Once connected, cancel and/or close any windows begging for updates. Do not press Okay, that will start an unwanted update!

Double-click the big red button icon in the middle of the desktop to open TOPAS FemtoFiber smart

Press Menu at top left, choose Connect

Press the "Hardware Disable" button at left

In the Laser Current box at right, change "I set" to 356 mA and press Return; the small fake LED just above the entry box should turn on green.

Press the "Emission" button at left

The "Laser on" fake LED at the top should turn yellow, and the fake warning sign illuminates

Don't close the TOPAS window, but you can slide it down a little out of the way to see the yellow padlock icon on the desktop.

Double-click the yellow padlock icon on the desktop to open PicoFYb-Phaselock. You may need to wait 10-15s for the program to connect fully. At top left, you should have red/black/red buttons in the first column, two green buttons in the next column, and then two gauges indicating a blue signal level. If there is more red or orange than expected, review the instructions to see if you forgot to turn anything on (or turned it on in the wrong order). It may be necessary to back out and start over from the power switch on cocoa. If you're certain you did everything in the right order, call for help. If all is good, proceed to adjusting the temperature for the fiber optic loop:

Look at the number in the "difference frequency" box at top right.

If the number is over 1200 and changing rapidly, go away for a few minutes and check back after it settles down.

If the number is a few hundred up to about 1200 and steady or changing slowly, proceed...

Select the "TEC temperature" tab in the top right section. Initial setpoint will be 25.85 C.
Mistake check! Are you getting weird symptoms like TEC temperature only specified to a tenth of a degree, or maybe the piezo voltage has changed from 65 to 50 or vice versa? This happens sometimes and we're not certain of the cause, but make sure that you did WARMUP before you started all the ACS work. If you didn't, reverse steps all the way to the power commands on cocoa, then do WARMUP, then start over again.

Change the setpoint and press return.

In summer, make the setpoint colder. In winter, make it warmer.

For a difference frequency of 600, change temp by 1 C, less for a smaller difference and more for larger.

The TEC may initially go in the wrong direction but should eventually figure out heating vs cooling

Go back to the "phase mismatch" tab and watch the difference frequency head towards zero. When it gets below 200 a line will appear on the plot next to the difference frequency.

If difference freqency increases instead of decreasing, change the TEC in the other direction.

If difference frequency levels off, change TEC again OR use Forward/Backwards buttons to adjust by 0.01 C.

Once difference frequency gets to less than about 3, go to ATUI ACS tab and check PicoFYb lock

Back in the VNC window, the plot at top is now grayed out. Instead, watch the "piezo voltage" plot at middle right.

You want the voltage to be near the green dashed line, between the two gray dashed lines.

Use the Forward and Backwards buttons to bring the voltage in and make it steady

Once it's more or less flat, press the 'loop open' button to close the loop

Press the R next to the red 'overflow flipflop' button near top left of window

Check the temperature in the log murmur at bottom right; record this and the time in your observing log.

The auto loop can only correct slow changes, so check back again every few minutes to make sure the voltage is staying between the lines, or bring it back if needed. You may need to re-close the loop and press the R next to 'overflow flipflop' again after you bring it back manually.

After a while it should settle down and stop wandering and you can check less often, but don't forget it completely.

Leave the VNC window open with the Phaselock window on top so you can check it at a glance.

Go back to the ACS tab of ATUI.

Press "Set delays to nominal values"

They should be 0 52 76 128 ; if not, set them by hand and press return. [Last changed August 2024]

Just below the delays, set the [DAC Val] to 0 and press return

Press the "ADC Read" button. The second value should be not-too-high, less than 400
Pro tip! If all the values for ADC read are zeroes, try the "AD5592 Recover" button, then ADC read again.

After a minute, press the "ADC Read" button again. It should be pretty close to the previous values; if not, wait a few minutes for it to settle before proceeding.

Press the "DAC Sweep" button. Watch the red and green sinusoids grow across the graph in the bottom of the window. Note the time of DAC sweep with the maximum red and green values in your observing log. If the DAC sweep is flat, you have a problem. Did you forget to do WARMUP before power on ACS? If yes, back out all the way to the cocoa power switch and start over (after WARMUP!). If you already did WARMUP at the right time, try the "AD5592 Recover" button or call for help.

Press the "Phase sweep" button. Watch the weird shapes grow across the graph. Try to mentally remove the square bumps, and the underlying shape should be flat-hump-flat-hump-flat-hump (three cycles). The graph should start on the flat portion shortly before the hump; if it doesn't, note the X value of that desirable position.

There can be some variations in how the phase sweep looks. Here are two examples of the Phase sweep plot: on the left with older values for the delays, so that only narrow downward blips are interrupting the underlying shape, and on the right with more recent delay settings (still not the same as current) so there is a broader square wave that almost hides the underlying shape. Both these examples begin close to the desirable position and don't need an offset.

Sometimes the phase sweep will show an upward slope (rarely downward). This may mean that the temperature hasn't quite settled on the fiber optic. Check the PhaseLock program, adjust temperature if needed, then try the phase sweep again after a few minutes and the slope should be shallower or perhaps gone. This is not believed to be a serious problem. You should be able to adjust the phase bump even when a slope is present.

Enter the X value of the desirable position into the Phase bump box, including a + sign if it's a positive value.

Press return after entering the value. Bug alert: this is one of the boxes with a bug where it stays pink. Watch the fake LED at the top of the window to see when the comman is done, or go to the Main/Active tab and look at the last command that went through to make sure it was the offset you wanted.

After the LSB Status LED turns green, press "Phase sweep" again. Hopefully it now starts on the flat portion shortly before the hump. If not, adjust again. Once you are satisfied, Note in your log whether the phase sweep was flat or rising, and the cumulative offset(s) you made.

Now press the "Extremize bias" button and it should make a nice parabola and automatically set the [DAC Val] number to the minimum.

Change the step size from the initial 200 value to 100, and repeat the "Extremize bias" step. The parabola is smaller now and the DAC value for the minimum is more precise. Note this value in your log.
Here is an example of the plot from extremizing the bias (also called "xbias"), with the black circles showing the measurement points and the red asterisk showing the minimum position.

Starting with an initial guess of DAC value far away from the minimum may cause it to find a maximum instead of a minimum, or not find a flat point at all. Here are two examples of bad xbiases:

Finish ACS checkout by setting the attenuation to maximum with 'acs atten 60' in the Main/Active tab command line OR by entering 60 in the Attenuator [dB] box of the ACS tab (it stays pink).

Hopefully all these steps went smoothly. If they didn't, you might need to call for help or else skip using ACS for this session and try to get someone to work on it later. If you're ready to give up on ACS, you can go through the shutdown steps for ACS only in the Finish and cooldown section, but proceed with checkout and warmup of the main laser system. Turning off ACS if you're not going to use it will reduce the heat load inside Utah, which is desirable in warmer weather.

During a ranging run with ACS enabled and attenuation lowered to about 30, you can look over at some of the plots on the right side of the Plots window to see what ACS is doing. The top-right plot in particular, "Lunar Raw TDC," should show the desired reference comb with 11 teeth. This example shows a closed-dome run, so there are fids but no lunar returns.

3.2.8 15 minutes before ranging: Check APD

The Avalanche Photodiode, or APD, is the primary science detector for APOLLO. It is 15 pixels laid out in a 4x4 grid, with the top left pixel in the grid used for timing rather than light detection. There are gaps between the individual photodiodes, but each has a lenslet above it to focus light from a larger area and ensure no photons fall through without being detected. The lenslets are about a quarter arcsecond on-sky, meaning the size of the detector is just over 1 arcsec side-to-side, or about 1.5 arcsec diagonally.

Look in the middle of the ATUI Main/Active tab, just to the left of the StartNubs button. Make sure NStares = 5; Stare rate = 1000; Stare binning = 500; # of Darks = 10000.

On the left side of ATUI, press the Stare button.

Watch the Hitgrid and Stare Rate plots in the Plot window (top left and top middle) for ~5s.

If the x-axis of the Stare Rate plot counts up to 5 but the line is flat at zero and the hitgrid doesn't change, press the caltdc button and wait a few seconds for status to go back to STANDBY. [Note: if you already did ACS checkout, this step may be unneeded.]

Press the Stare button again, and now you should see positive counts on the Stare Rate plot.

Make sure BC1 eyelid is closed, and press the Dark button.

Wait ~30s until the hitgrid updates and the status goes back to STANDBY.

Press the Stare button again, and now you should see the counts on the Stare Rate plot average zero plus or minus some random noise, because an average dark is being subtracted from the rate.

Record the time of Stare/caltdc/stare/dark/stare in your observing log.

Now the APD is ready to go.

3.2.9 15 minutes before ranging: Set STV crosshairs

The APOLLO guider is a Santa Barbara Instrument Group (SBIG) camera known primarily as the STV, also called "CCD" or "guider" in some places. It has a physical control box and image display screen inside the observing-level cabinet, but the camera is nearly always used in remote instead of local mode. A graphical display in the STV tab of ATUI replicates the appearance and functionality of the physical control box.

Start by opening a browser window to video-35m. From the quad view, select video 4 at the bottom right. This is the feed from the APOLLO STV. If the state is WARMUP or STANDBY, the STV should already be on and probably already in Focus mode, which is the mode we usually operate in. You can tell it's in Focus mode from the square box moving across the bottom of the screen from left to right, showing the progression of frames. Set this browser window with the STV video feed somewhere visible off to the side of ATUI.

In the STV tab of ATUI, click on the white box with red text in the upper-right quadrant of the control interface. This is the display screen for control info of the STV. Sometimes the display does not update with the latest information, so clicking on the display will force an update. If ATUI is in the state WARMUP or STANDBY and the video is running in its default startup state, updating the display should cause it to say
FOCUS
Norm 25ms, 1x Full

This means it's in Focus mode with 25ms exposure time and 1x gain.

Optional practice for getting used to the controls: Move your cursor below the display screen to the knobs which have pasted-on boxes saying "20" and a pair of arrows next to each knob. Click the up-arrow, and you should see the 25ms in the display screen change to 50ms. You can click on the display screen if desired to make certain it's up to date. 50ms or .10s (another up-arrow) are reasonable exposure times for a likely focus star.

To bring up crosshairs, click once on the square black button labeled "Display/Crosshairs" and then on the square black button labeled "Value." Now the video display in the browser should become static, showing crosshairs centered on the upper-left part of the image.

Holding down the control key on your keyboard, click on the right-arrow. This will move the crosshairs to the right by 20 steps (or whatever is in the white box pasted over the knob). Now the display screen should say something like DISPLAY (X,Y)=100,51. You can click on the screen to make sure the numbers are up to date. Use ctrl-right-arrow to make large adjustments until the X number is near 160 or so, then release the control key and use the right and left arrows to make single-pixel adjustments until the X position is 159 (remember to update the display, especially after changing directions!). Then use ctrl-up-arrow and single-pixel up/down adjustments to move the Y position to 72 (note the up-arrow makes the crosshairs move down on the image for higher Y numbers.)

Pro tip! Did you press Display/Crosshairs more than once? Did you press the Parameter button instead of the Value button? Is there now a weird menu showing up in the red & white display? Try pressing the black Focus button twice to go back to Focus mode. Once you see the little white square moving across the bottom of the image, try again. If you get into a really weird mode like a zoomed-in image, you can go to the Power Tab of ATUI and uncheck the STV box, wait a few seconds, then re-check the STV box to power cycle the camera. When the image shows a SBIG start screen, then press the black Focus button twice to get back to Focus mode.

Measured in December 2019 and periodically confirmed since then, (159, 72) is the sweet spot on the STV where light passes through to the APD science detector. The STV is looking through a filter close to the laser wavelength but not exactly on it, so the star does not disappear from the STV while it is on the sweet spot. The X,Y position for the sweet spot is stable unless the STV camera or the "snout" that feeds light to it gets disturbed, in which case you should update the sweet spot position using the method described in Troubleshooting and Maintenance.

Put a piece of tape over the crosshairs position in the browser video window, and mark it with a sharpie. Take care not to move that browser window again after marking it!

Put the guider back in Focus mode by pressing the square black button labeled Focus twice. Confirm that the white square is moving across the bottom of your video window. Now the STV is ready to go.

3.2.10 5 minutes before ranging: warm up main laser

The main laser needs to "flash" in order to warm up before ranging. If not warmed up, the laser may have unusual numbers for threshold voltage and output power, which could lead you to run the laser with the wrong voltage. In warm weather you might want to shorten the warmup time to 5 minutes in order to reduce the heat load inside Utah, but from October to April you probably want more like 10 minutes of warmup.

In ATUI, go to the Laser Tuning tab.

Press the Laser Powerup button.

Watch the display on the Continuum interface until it shows a shot count.

Record the shot count in your observing log, in the box near the upper right corner.

Press the Laser Warmup button right beneath the Powerup button.

Watch the Continuum interface until the first two red and green LEDs are both lit and circled in white to indicate they are flashing.

Record the time of "powerup" and "flashing" in a row of your observing log.

After about 5 minutes (summer) or 10 minutes (winter), you will check the laser threshold and power. This should be done at the position of the Moon, so while the laser is flashing it's time to slew!
3.3 Ranging

3.3.1 Pointing and focus check on a star

Load the APOLLO catalog into your TUI Slew window. Check the ATUI Pointer tab (press the Now button to refresh the numbers) for the Moon's azimuth and altitude. The azimuth listed here is aviation azimuth, so you have to convert it to TUI azimuth, but a rough conversion is sufficient. (Hint: alternatively, look at Omea or IRSC for the general position of the Moon!) Choose a catalog star near the position of the Moon. When the observing specialist gives you slew permission, the first target you will slew to is this pointing star.

Standard APOLLO procedure is to use Boresight offsets for coarse initial centering, and then Guide type offsets for smaller tracking adjustments while ranging. This separation makes it possible to watch for tracking imperfections and patterns which might otherwise go unnoticed. However, in practice either Boresight or Guide offsets can be used for either purpose. DO NOT use ObjArc offsets with APOLLO, because those offsets are used to follow the Moon's nonsidereal velocity, so changing the ObjArc offsets may mess up the lunar pointing very badly.

In the STV tab of ATUI, first make sure it's in Focus mode by clicking on the white display window, or check the guide video for the little white square moving across the bottom of the screen. Then use the up/down arrows to adjust the exposure time on the STV so that you can see the star clearly but it isn't saturated. Note that the maximum exposure time in Focus mode is .25s, but if you keep pressing the up-arrow the gain will increase to make the star look brigher, until you get to .25s, 16x which is the maximum sensitivity setting. If you can't see a third-magnitude star at this setting, the clouds are too thick and you will certainly not be able to range to the Moon. Or maybe the pointing is off or something is obscuring the light path.

Adjust telescope focus to make the star look round and not-lumpy on the video image. Then go to the Main/Active tab of TUI and the "Offsets" section at top right of the window. Check the box for Boresight Offset and set the Offset Increment to a number between 1.0 to 10.0. Make sure the CCD frame is checked at the right side. Now clicking on the direction buttons will pull the star in the indicated direction on the STV image by the Offset Increment amount. Use the direction buttons to center the star and also to get a feel for how big is 1 arcsecond, so that you can do a rough visual estimate of the seeing. Since you are looking at green light, the seeing will rarely be better than 1 arcsec. 2 arcsec is bad enough to make ranging difficult, and 3 arcsec might make ranging impossible. Report the seeing value to the observing specialist so they can record it.

Note: There is a script for python3 machines which will do a quantitative measurement of the star size. This script is almost ready to go in spring 2023, but still a little slow and clunky, so it's probably more efficient to estimate the seeing by eye. But watch for more developments here!

Note the name of the pointing star in the observing log, the accumulated Boresight offset, and the focus value. Put your seeing estimate in the Comments column. Now you're ready to slew to the Moon.

3.3.2 Pointing check on a crater

In the Pointer tab of ATUI, press Show Craters to mark the positions of some smallish reference craters on the Moon. If Apollo 15 is in sunlight, you will usually start with "Nearby Knob" which is a small mountain rather than a crater, but fits in the same pointing-reference category. Nearby Knob is a small isolated bright spot with no misleading shadows and very useful for centering on... when it's in sunlight.

If the Moon is a narrow waxing crescent, you will probably have to use Webb way over on the west side, but if the Messier craters are illuminated, then Messier B is your best choice (small with accurate position). Craters Peirce and Luther have less-reliable positions, so only use those if you also cross-check against another crater. For a waning moon, Fra Mauro B and Milichius A have reliable positions; for a narrow waning crescent, try to cross-check Damoiseau E and Lohrmann A against each other if possible.

If the Moon is a narrow waxing crescent, use Webb, pointing just a little below center:

For a wider waxing crescent, use Webb first then center on Messier B:

During mid-lunation while it's illuminated, Nearby Knob is the best choice:

For a wider waning crescent, use either Fra Mauro B or Milichius A:

For a narrow waning crescent, use both Damoiseau E and Lohrmann A, pointing a little low on both:

Press Slew Telescope in the Pointer tab to move to the position of your selected crater (or mountain) on the Moon. Slews made from the Pointer tab will keep both Boresight and Guide offsets. You may find that the crater position disagrees with the pointer star by up to 10 arcsec; this is mostly because the proper motion of very bright stars is not always well measured. You can do a coarse centering at this time, but don't refine it too carefully just yet, because you have some other things to do first.

IMPORTANT!!! After going to the Moon, be sure to press the Go To Target button in the Offsets section of the Main/Active tab. After you press that button, down in the bottom portion of the Main/Active tab the numbers in the "Current rxx" and "Current rxy" boxes should now agree with the numbers right next door under the "Target Values" label. If you try to do ranging with the Current and Target values disagreeing by 0.2 or more, you will lose signal photons. If you try to do ranging with a disagreement of 0.5 or more, you may never find the signal. If you center on a crater with the Current values disagreeing with the Target values, and then later you remember and press Go To Target, it will shift your pointing. So press Go To Target while you are still at the centering-on-a-crater phase in order to avoid confusion and possible future trouble.

Another important note: Please be careful not to press the very-nearby Commit button instead of the Go To Target button. The Commit button will force Current and Target values to agree with each other not by moving the receiver optics to the Target position, but by re-defining the Current position. So if you Commit the wrong position, the optics position becomes untrue. Someone will need to go into the housctl logfiles and find out out which starting position got redefined to which ending position, then move the optics in the opposite direction by the same amount and Commit again. It's a nasty job, so don't do that. But if you do, tell the truth about it right away!

3.3.3 Laser threshold and power check

The next thing to do is check the laser threshold voltage and output power. Hopefully it has now been flashing for about 5-15 minutes (shorter in the summer, longer in the winter), so it's time to find out if the laser is happy.

In the Laser Tuning tab in ATUI:
Enter 2 or 3 in the box beneath the Measure Power button, and press return to change it from pink to white.
This is how many minutes the power measurement will last.

Press the Measure Power button

Wait until numbers and a plot line appear in the graph in the lower part of the window

Are the numbers exactly 0.0? The bolometer may be off. Press Stop button below Measure Power button, then go to the Troubleshooting section.

Are the numbers around 0 but with some noise? Move voltage up with the ^ button to the right of "Oscillator Voltage," but if you don't get measurable power within four button-presses, something may be wrong. Troubleshooting

Are the numbers somewhere between 1 and 2 Watts? Proceed with threshold check:

Go to the buttons to the right of the "Oscillator Voltage" in the top part of the window, and press the v30 button. This moves the voltage down by 30V, but the number in the box to the left changes by a larger amount because the displayed number is ohms, not volts!

Note: After a restart of housctl, there may be a fake number (0 or 9999) shown in the box initially. Go ahead and change the voltage, and after that it will remember the actual value.

Did the power drop to near 0?

If not, use the v button to go down 5V at a time until the power drops.

Once the power drops, use the ^ button to go up 5V to the lowest voltage with non-zero power. This is the threshold.

Record the threshold value (in ohms, in the box to the right of "Oscillator Voltage") in your observing log

Now press the ^30 button to go up 30V from the threshold. This is the operating voltage.

Press the Clear Stripchart button to refresh the graph so you can see small changes.

In the Continuum interface, experiment with the clockwise-arrow and counterclockwise-arrow buttons to get maximum power.

These buttons are adjusting the second harmonic generator which changes the laser light from IR to green.

If the telescope altitude is higher than when it was last adjusted, the SHG probably wants to go clockwise (top button); if lower, it wants to go counterclockwise (second button).

Find the SHG setting that gives highest power and leave it there.

Record the 20-point average power (at top center of window) in your observing log, in Watts.

When the power measurement finishes (you can use the Stop button below Measure Power to end it early), the Continuum should say that it's in PGM2, which is what we want for ranging. Press the Shutter button in the grid of Continuum buttons, and watch the Shutter LED turn green; this opens the laser shutter that allows laser light into Utah.

Now go back to the Main/Active tab and refine the pointing on your reference crater until it's well centered. Ask the observing specialist to press the green button on the keyswitch box; this opens the interlock shutter that allows the laser light out to the telescope. Now you are ready for ranging.

3.3.4 First target: Apollo 15

Always start ranging with Apollo 15 because it's the biggest reflector with the strongest returns, and also (when it's in sunlight) has the easiest position to find. In the Pointer tab of ATUI, in the graphical interface, click on the red + marking the position of Apollo 15, then press Slew Telescope. Allowing for the possibility of different sky rotation and different shadows and whether or not you have an atmosphere in the way, the location of the landing site looks like this:

The second image above is from the APOLLO STV showing moderately good seeing around 1.6 arcsec, which is nearly perfect for ranging. If you have better seeing, like in the third or fourth images, you might want to consider defocusing the telescope 100 steps for your first ranging attempt, in order to make the laser beam larger on the Moon. Once you detect signal you should walk the focus back to best-focus, 50 steps at a time.

Pointing should be pretty good if your reference was Nearby Knob, but if you need to make an adjustment, you can do it from the Offsets section of the Main/Active tab of ATUI. Use Boresight offsets if you have not been ranging yet, or "Raster Scope" (Guide) offsets if you have already started ranging, but either way, if you are adjusting position by referring to the STV image, be sure that you are doing the offsets in the CCD frame.

If Apollo 15 is not in sunlight, you will just have to start from the position you measured on a reference crater, but you are expecting that you will have to start out with a search raster initially. If it's in sunlight with decent seeing conditions, you have a reasonable chance of acquiring signal right away. In preparation for a search raster, go to the Offsets section of the Main/Active tab, check the Raster Scope box, and set Offset Increment to 1.0 (maybe 1.25 in poor seeing conditions, or 0.75 in good seeing conditions).

Checklist before starting run:

Is laser shutter open? (Continuum interface in Laser Tuning tab)

Is interlock shutter open? (Green button on keyswitch box)

Is the system state in STANDBY? (Left side of ATUI)

Is the APD awake and ready to go? (Got near-zero but not flat-zero counts in last stare)

Is "# of shots" set to 5000-8000? (Lower section of Main/Active tab)

Is ACS off for first run? (LUN_EN and FID_EN boxes unchecked in ACS tab)

Is ACS attenuation set high to begin with? ('acs atten 60' in command line of Main/Active tab)

Is ACS phase properly locked? ("piezo voltage" plot in VNC window)

You're ready!
Press the Run button on the left side of ATUI

Record the time, reflector, and # of shots in the first three columns of the observing log.

Your Plots window should refresh or maybe a new Plots window will open.

Wait 10s for TR mirror to spin up (plus an extra 5s if ACS is enabled)

In the Run statistics section of Main/Active tab, "Shots per second" should go up to 20 and "Fiducial Records" should begin counting up.

A spike should show up right away in the "Fiducial, subtracted" plot in the middle of the Plots window.

After 3 more seconds, occasional photons should start appearing in the "Lunar, subtracted" histogram and "Lunar Residual Scatter" stripchart plots on the left side of the Plots window.

If no clear signal shows up after about 10s, start doing a search raster with the Offsets in the Active window.

check Raster Scope

Native frame suggested, especially if no sunlight, but any frame can be used

Offset down; wait 10-20s after end of offset

Offset left and wait

Offset up and wait

Offset up and wait

Offset right and wait

Offset right and wait

Offset down and wait

Offset down and wait; should be near end of run now.

Still no signal? Optionally go back to Home for end of run (this will zero the guide offsets!)

Optionally increase # of shots (and press return!), and continue searching.

You got a signal spike!

First move the PredSkew slider at the bottom center of the Main/Active tab so the dashed lines in the plot bracket your signal

Record the Predskew in the Comments column of the observing log with "PS=###"

Look at the hitgrid in the Plots window; is the signal off center?

Check Current vs Target rx values in the bottom part of Main/Active tab, make sure they are close to each other

Choose Raster Scope and APD frame for your offsets!

Make Offset Increment smaller; 0.25 is usually best, or 0.5 if you are barely starting to get signal

Use arrow buttons to center signal on APD.

Once lunar photons are coming in, take a look at the "Registered Fiducial Photons" in the middle of the Run Statistics table. We would like this to be between about 1-1.5

If the fid rate is consistently higher than 1.5, change the "Target Diffuser Phase" in the bottom part of the Main/Active tab by subtracting 40 or 50, and press return.

If the fid rate is consistently lower than 1.0, change the "Target Diffuser Phase" in the bottom part of the Main/Active tab by adding 40 or 50, and press return.

Don't change the diffuser phase to less than 1100 or more than 1300 without consulting someone who understands it better.

You didn't get a signal, and now the run is over.

Record the sad run statistics before doing other stuff

Check Current vs Target rx values in the bottom part of Main/Active tab, make sure they are close to each other

Check STV (if in sunlight) or cloudcam to see if you are losing signal to clouds

Slew back to reference crater and adjust focus (or defocus if seeing is very good) and recenter. (Zero Guide offsets with Home button, then switch to Boresight offsets to re-center crater, then back to Raster Scope in preparation for next run)

On next run, try a larger Offset Increment, or spend longer on each position, or just hope for better luck.

You got a signal but then lost it when trying to center up!

Look at the "Lunar Return Rate" plot; it should show the Guide offset where the peak returns occurred.

Use Raster Scope and Native checkboxes, then offset with the direction arrows to adjust Guide offsets to be near where you got the peak returns.

Wait for more returns to come in, and then try again to center up, in the APD frame this time!

You got a signal, and didn't lose it, but it gets weaker when you try to move it to center, as if the APD is "hot" on one side.

Check Current vs Target rx values in the bottom portion of the Main/Active tab; are they close? (Difference should be less than 0.1)

Press Go To Target to move rx optics to Target position.

Is it a single hot pixel? It might be malfunctioning. You can uncheck that pixel in the Channels tab of ATUI

If you're sure this is a real, persistent trend, adjust optics carefully!!!

Choose Raster Optics, APD frame

Offset Increment 0.25 at most

Press direction button to "pull" signal onto chip

Change back to Raster Scope, so you don't move the optics accidentally while you watch for an improvement.
Maybe do some more telescope offsets to center signal.

Not sure that it helped? Press Go To Target to move back to previous optics position.

Yes, it definitely helped? When you have it adjusted so you have good signal in the center of the chip, record the change in the RX Offset column of the Observing log and then press Commit Rxx, Rxy.

You got some signal, and now the run is over!

Record Yield in observing log (bottom of Run Statistics table).

Record Peak in observing log (from "Lunar Return Rate" plot)

Record Guide Offset in observing log

Add Comments to observing log if needed.

You're ready for the next run!

Somewhere around this time, maybe before your first run in summer or after your third run in winter, there will be an error message about Utah temperature exceeding 22 C. The error message looks scary, but we aren't really worried yet. Between runs, go to the Environment Tables webpage, be sure to refresh it, and look at the latest measurement of the Utah temp.

If it's 22 up to 24 C, keep ranging but check again between each run. If it exceeds 25 C, start shortening ranging runs to 3000 shots or leaving longer gaps in between runs. Consider ending your observations if you already got a circuit around the reflectors. You could also turn off ACS but keep going with the main laser. If Utah temp exceeds 27 C or if you start seeing laser dropouts where the the "Shots per second" in the Run Statistics table drops below 15 and doesn't come back up, it's time to end your observations. You might want to do a final power measurement before cooling down the laser, and talk to knowledgeable people about what could be done to improve cooling for the next observing session.

3.3.5 Repeat Apollo 15 with ACS

Running ACS will add an increased level of background noise. The minimum non-ACS signal level that will let you add on ACS and still be able to keep your signal centered is about 0.1 photons/shot in the "Lunar Return Rate" plot, or 0.2 if you feel less confident. So the signal level in your non-ACS runs will determine your decision whether to enable ACS. However, even if you are getting weaker signal than that, if you have any returns at all you still want at least ONE run with ACS which can then be used to calibrate other runs without ACS. Usually, you will enable ACS for a second run on Apollo 15 just because that's most convenient. If signal is weak now but you have some reason (the Moon is rising? The clouds are expected to clear?) to think that later runs will be stronger, you could postpone the one ACS run until later in the session. Only skip ACS if you decide to give up ranging without any usable signal at all.

You can start up a run with ACS enabled but attenuation high ('acs atten 60') and the signal will look like a non-ACS run so that you can get centered up. There will still be two differences: an ACS run has a longer delay at the beginning before laser light starts to come out, and an ACS run with high attenuation will not show "Peak" information in the Lunar Return Rate plot. (This is a bug - not a showstopper, but it's pretty annoying to have to keep the peak rate AND the position where you got that peak in your own mind.) Because of those differences, if you are sure you won't be using ACS on a particular reflector, it's best to go through the step of un-checking LUN_EN and FID_EN. If you think you might use ACS but you're not sure, start up a run with high attenuation, try to track the peak rate with your brain, and you can reduce the attenuation later to let the ACS comb come through. But we do need a minimum of 1200 ACS photons in each run for ACS to be useful, so make the decision whether or not to lower attention while you still have at least 3000 shots left in the run.

Checklist before starting first ACS run:

Is laser shutter open? (Continuum interface in Laser Tuning tab)

Is interlock shutter open? (Green button on keyswitch box)

Is the system state in STANDBY? (Left side of ATUI)

Is "# of shots" set to 5000? (Lower section of Main/Active tab)

Is ACS phase properly locked? ("piezo voltage" plot in VNC window)

Go to ACS tab and do Extremize Bias with a stepsize of 100. Record the DAC value in your observing log.

Check the LUN_EN box and wait for the LSB Status LED to change from yellow back to green.

Check the FID_EN box and wait for the LSB Status LED to change from yellow back to green.

Go to the active tab, and on the command line enter 'acs atten 60' to start out with high attenuation. Wait for command to complete.

Ready to start the run!

Press the Run button on the left side of ATUI

Record the time, reflector, # of shots (if you think you won't change them) in the first three columns of the observing log.

Record the DAC (changed by the extremize bias) over at the right side of the "Comments" column in the observing log. This indicates it is an ACS run.

Your Plots window should refresh or maybe a new Plots window will open.

Wait 15s for system to get up to speed

In the Run statistics section of Main/Active tab, "Shots per second" should go up to 20 and "Fiducial Records" should begin counting up.

A spike should show up right away in the "Fiducial, subtracted" plot in the middle of the Plots window.

After 3 more seconds, occasional photons should start appearing in the "Lunar, subtracted" histogram and "Lunar Residual Scatter" stripchart plots on the left side of the Plots window.

You were already centered on Apollo 15, so hopefully a spike will show up in the histogram right away.

Spend a short while getting the signal centered on the hitgrid with Raster Scope offsets in the APD frame.

Once signal is centered but before the run is up to 2000 shots, lower attenuation:
In the command line of the Main/Active tab of ATUI, enter 'acs atten 30' if your returns are strong

Or 'acs atten 32' if your returns are on the weak side, or somewhere in between

Record the new attenuation and the shot count when you changed it in the Comments column of the observing log.

ACS photons will raise the background in the histogram and stripchart.

On the hitgrid, ACS photons are more likely to show up in the center.

Even though background is subtracted, ACS photons will increase the Yield in the Run Statistics table and the Peak in the "Lunar Return Rate" plot.

If you have enough signal, try to stay centered on the hitgrid by paying attention to the OUTER pixels rather than the inner ones, because the inner ones are more affected by ACS.

If you don't have enough signal, just trust the blind tracking.

Keep an eye on the "# of ACS photons" at the bottom of the Run Statistics table. If # of ACS photons is not going to reach 1200 before the end of the run, either lower attenuation more OR increase the # of shots in the run.

When the run is finished, record normal stuff in your observing log

Record Yield from bottom of Run Statistics table).

Record Peak from "Lunar Return Rate" plot

Record Guide Offset if you were able to follow tracking; leave blank if you used blind tracking

Add Comments to observing log if needed.

3.3.6 Other reflectors

The decision of what reflector to observe next will depend a lot on moon phase and conditions. It is desirable to get returns from as many reflectors as possible, and add ACS whenever the return signal is strong enough to support it. Russet usually makes a clockwise circle around the reflectors on the Moon, but it isn't necessary to do them in this order. If you are doing more than one circuit, however, you may want your second circuit to follow the same path as the first in order to get the longest spacing between two runs on each reflector. And in most cases it's desirable for the last run to be back at A15, in order to get the longest time baseline on the reflector with the strongest signal.

Expected returns from Apollo 15 vary mostly depending on conditions in Earth's atmosphere (airmass, clouds, seeing). In clear conditions with good seeing, peak signal rates from A15 may exceed 1 photon per shot, especially for moon phase around 100-140 and 220-260. Signal will be weaker near full moon (phase 150-210) even with good atmospheric conditions, but keep going even if it's a struggle because measurements near full moon are very valuable and only our system can successfully obtain those measurements!

Lunokhod 2 (the rover for mission Luna 21) is the weakest reflector, usually returning 1-5 percent of the Apollo 15 signal level. It rarely gives any returns in sunlight, and never gives anything at moon phases between about 140 to 220; but data from that reflector is very valuable because it's so rare. If L2 is in the dark, try for it unless seeing is bad and signal from A15 is very weak. If L2 is in sunlight, it may be worth trying if conditions are very good, if signal is very strong from A15, or if it's only just in sunlight. If you're going to try for L2, it's a good idea to go there immediately after A15 when pointing is optimized. Expect signal to appear at a predskew about 30-50 lower than the signal from A15; you can even move the predskew slider before beginning the run.

In most conditions, Apollo 11 is the second-strongest reflector after A15 and may be the second one you try if the situation is not promising for L2. The predskew for A11 should agree well with the predskew you found on A15.

Apollo 14 gives comparable returns to Apollo 11, usually a little bit weaker but sometimes stronger at moon phase near 90. The predskew for A14 is sometimes a little bit higher than A15, maybe by about 20. Start out with the A15 predskew and then adjust if you find the signal is a little off to the side.

Lunokhod 1, the rover for mission Luna 19, is the most variable reflector. In the dark it is sometimes almost as strong as A15, and data from this reflector is also very rare and therefore very valuable. In the sunlight L1 may not give measurable returns, but it's usually worth trying at least once (don't bother if the moon phase is near full, about 140-220). L1 also has a very low predskew, 60-80 lower than the predskew for A15, so you can move the predskew slider before starting your run. You may have to do an extra search raster to pick up L1, or you might have success by searching primarily in the azimuth direction (up/down arrows if you are using the APD offset frame, or right/left arrows if you are using the Native frame).

The areas around A11, A14, and L1 have few landmarks. The positions of A15 and L2 are more recognizable, but it can still be tricky to center up because the field rotates depending on where the moon is in the sky, and the shadows will look different at different moon phases. So in most cases you will rely on your previous centering from a crater or another reflector as a starting point. Here are images of each of the reflector sites taken near full phase with the Moon near the meridian in the south (with uncertainty for some of them):

Russet's preferred algorithm for choosing which reflectors to observe:

Signal from A15 was strong! (peak >0.5 photons/shot):

Is L2 in the dark? Then L2 with ACS enabled, but might keep attenuation high for the whole run. Then A11 and 14 with ACS, and L1 probably with ACS depending how it went on the other reflectors, and lastly back to A15 with ACS. For second circuit, adjust ACS depending on how the first circuit went.

Is L2 in the sunlight but not that close to full moon? Then try L2 with ACS not enabled, A11 and A14 with ACS, L1 probably with ACS depending how it went with the other reflectors, and lastly back to A15 with ACS. For second circuit, adjust depending on how the first circuit went.

Close to full moon phase? Then skip L2 and go to A11 and A14 with ACS, try L1 without ACS but maybe give up early if not finding anything, and lastly back to A15 with ACS. Second circuit probably just the three Apollo reflectors, adding ACS depending on how the first circuit went.

Signal from A15 was fine (peak 0.2-0.5 photons/shot):

Is L2 in the dark? Then L2 with no ACS. Then A11 and 14 with ACS enabled but might keep attenuation high, and L1 probably with no ACS, and lastly back to A15 with ACS.

Is L2 in the sunlight? Then try L2 once without ACS but maybe give up if not finding anything, A11 and A14 with ACS enabled but might keep attenuation high, L1 probably with no ACS, and lastly back to A15 with ACS. For second circuit, adjust depending on how the first circuit went.

Close to full moon phase? Then skip L2 and go to A11 and A14 with ACS, try L1 without ACS but maybe give up early if not finding anything, and lastly back to A15 with ACS. Second circuit probably just the three Apollo reflectors, adding ACS depending on how the first circuit went.

Signal from A15 was weak (peak <0.2 photons/shot):

Is L2 in the dark? Maaaybe try for it without ACS, or maybe skip it for the first circuit and then decide whether to try on second circuit. A11 and A14 probably with no ACS (could add ACS on second circuit if signal is better than expected). L1 without ACS and be prepared to stop early if no signal. Then back to A15 probably with no ACS, unless the signal improves.

Is L2 in sunlight? Then skip it and go to A11 and A14 probably with no ACS (could add ACS on second circuit if signal is better than expected). L1 without ACS and be prepared to stop early if no signal. L1 without ACS and be prepared to stop early if no signal. Then back to A15 probably with no ACS, unless the signal improves.

Close to full moon phase? Skip both Lunokhods and circle the three Apollo reflectors, probably without ACS at first but maybe adding ACS later if conditions approve.

3.3.7 Logging results

Filling out the observing log can be attention-consuming! This section of the manual is best suited for careful study in between observing sessions, with followup reference while observing.

3.3.8 Finish and cooldown

When you are nearing the end of the APOLLO observing session, hopefully after at least one run with signal and at least one run with ACS, you can usually continue ranging right up until the end time the observing specialist gives you. If something weird happened during the past hour, you might want to end 2-3 minutes early so you can do a final laser power check (but you can skip the threshold check and SHG adjustment), just to record any change in performance from the initial power check. This final power measurement should be done with the telescope at the altitude of the Moon, before you hand the telescope back to the observing specialist. But if everything is performing well, you can skip the final power check.

If you got any useful data, you do want a Lasercal at the end. This can be done at any telescope altitude and it does not send light into the dome, so you can do it while the observing specialist is starting the instrument change. You want the laser shutter open (the LED in the Continuum display in the Laser Tuning tab) and the interlock shutter closed (ask the observing specialist to press the red button or just turn off the keyswitch), and # of shots set to 5000 even if some of your ranging runs used a different shotcount. Set these things up and press the Lasercal button on the left side of ATUI, and occasionally glance at the progress messages showing up at the very bottom of the ATUI window to make sure the Lasercal is progressing. It will take 2-3 minutes. Make a note in your observing log that you did the Lasercal (usually the last thing in a typical observing log).

If you used ACS during the run, you can power that down while the Lasercal is running.

Enter 'acs atten 60' in the command line of the Main/Active tab, and wait for that to complete.

Go to the ACS tab and uncheck LUN_EN (wait for LED to turn green)

Uncheck FID_EN (wait for LED to turn green).

In your VNC window to the acs-laser laptop, press loop closed to change it to loop open.

In the ACS tab of ATUI, uncheck PicoFYb Lock.

In the VNC window, close the Phaselock window with the red X

Press the Emission button in the Toptica program and close that window

Close Connect on your screen sharing/VNC program

In the ACS tab of ATUI, uncheck PicoFYB DC power

In the ssh window to cocoa if it is still open (or in the cocoa:/home/apollo/ACS/ directory if you need to reconnect), enter './ip-pwr.py pset 2 0' to turn off the main power to the ACS laser.

Once the Lasercal completes, you can press the Cooldown button at the left side of ATUI. This will spend a while cooling the main laser and Utah and the chillers, so the state will say COOLDOWN for about 10 minutes before it changes to IDLE. During this period the "Laser power" LED on the keyswitch box will say that the laser power is still on, but it is safe to quit and disconnect before the whole process finishes, and let housctl take care of everything.

Before you exit TUI, go to the ROOT tab and Disconnect. Then you can close the plots window either with the red X in the corner or by quitting python from the top menu bar. If you forget to Disconnect from ROOT, closing the plots window might need a Force Quit.

Oops, did you forget about powering off ACS until after you pressed the Cooldown button? No big deal; it took care of most of the power-off steps for you, but you might find some messy open windows the next time you VNC to acs-laser. And you do still need to do the final './ip-pwr.py pset 2 0' on cocoa, which is the only thing that Cooldown is not able to turn off for you.

4. Troubleshooting and Maintenance

4.1 Viewing/adjusting pulse train and build-up time

First, gather the equipment you need to take upstairs:

The APOLLO oscilloscope from the bottom storage drawer in the rear lab.

The flat black box holding burn paper and miscellaneous tools, from the cabinet in rear lab.

A BNC cable from the box labeled "coax cables" in the cabinet in the rear lab.

The black electronics box labeled "FAST PHOTODIODE" from the cabinet in the rear lab

Take these tools upstairs along with a laptop that can run ATUI. Set the oscilloscope on a cart or chair or box just outside of the sliding bars behind Utah, and get an extension cord to plug it in.

Get laser safety glasses out of the bin for everyone present. The lead person should use the Laser alignment glasses, which have a dark red frame instead of a black frame. This pair of glasses lets a small percentage of the green laser light through, so the person wearing them will be able to see the adjustments they are making. Our other safety glasses block all of the laser light.

Move the telescope to high altitude (82-83 for a short person, 80 for a tall person) so that you can access Utah, then make sure a stop button is active.

Start up ATUI and initialize with Take Control, StartNubs, Listen, ConnDev, and Get Status. (Warmup and the plots window are not needed for laser optimization.)

In the Laser Tuning tab, press the Laser Powerup button (wait for completion and shot count) and the Laser Warmup button. Let the laser flash for 5-10 minutes to warm it up.

Open the top and middle panels of the phone booth/TARDIS. Find the BNC cable coiled in the right side of the phone booth below the capacitor banks. Plug this cable into the EXT TRIG port on the lower right of the oscilloscope.

Use the BNC cable you brought upstairs to plug the fast photodiode into CH1 of the oscilloscope.

Has the laser been flashing long enough to warm up? In the Laser Tuning tab of ATUI, check the threshold, measure power, and optimize the SHG just as you would before ranging. With the telescope at high altitude, SHG probably wants to go several jumps clockwise (top button).

Get a roll of tape from the cabinet at the top of the stairs. Open the left door of the Utah box, and use the tape to hold the door open by attaching it to the phone booth.

Put on safety glasses before opening the laser cavity. The laser should not be flashing; confirm this is true, and then you can take off the safety glasses if desired, but keep them ready. Move the cavity door out of the way.

At this point, there are several different things you can do to check or optimize the laser health. Here is a description of different tasks approximately in order of how common they are:
Improve power using steering mirrors:

Check beam using burn paper:

Check/optimize pulse train and build-up time:
Use tape to secure the Fast Photodiode sensor-side up (keep tape off the sensor!) on top of the oscillator box, right next to the shutter but not blocking the shutter. Use more tape to hold up the BNC cable so that it will not pull down on the Fast Photodiode.

4.2 Troubleshooting common problems

Problem: Measured laser power is exactly 0.0 and flat

This means the bolometer has been turned off, which sometimes happens after power disruptions to the obs-level cabinet. Open the top portion of the cabinet and check the bolometer control paddle hanging near the center of the top portion. If there is nothing on the display, turn the control paddle on. Once the paddle is on you should at least see noise during a power measurement.

Problem: Measured laser power is around 0 with some noise

This means the bolometer is working, but the laser light is not reaching the bolometer. You already tried going to a higher oscillator voltage, and when that failed you moved the voltage back down to near the starting point. Now try moving the flip mirror in and out with the buttons that say "Put Bolometer IN" and "Take Bolometer OUT" in the Laser Tuning tab. If this does not improve the power measurement, stop the measurement.

Make sure the laser is not flashing (red and green LEDs on the Continuum display are dark) and the laser shutter is closed (third LED on the Continuum display is dark), then move the telescope to high altitude, take a laptop with ATUI upstairs, and open the left door of the Utah box. Take the door off the laser cavity. Safety glasses are not needed at this point, as long as you made sure the laser is not flashing.

Locate the bolometer on the right side of the laser cavity, and the flip mirror that feeds light to the bolometer. Command the flip mirror to move in and out using the buttons in the Laser Tuning tab. If the mirror does not move under computer command, try reaching up above the laser cavity to move the rocker switch by hand. If the mirror still does not move, it probably has a dead battery. Now that you have confirmed the problem is the flip mirror and not the bolometer (and hopefully also not the laser power), you might want to close Utah and go back to your ranging run with no power measurement, then fix the flip mirror later.

When you have more time, take a 9V battery upstairs. With the laser NOT flashing, open the left and right doors of Utah and then the top section of Utah. Pull the unit with the rocker switch out from under the metal bracket, open it up, and replace the 9V battery. Test the flip mirror using the rocker switch and then the computer commands, and if the mirror works you can close up Utah and measure the laser power normally.

Error message: Traceback ending in "couldn't connect to display" when restarting ICC

This probably means the xhost is not running. These instructions assume you are already logged into root@cocoa in the tmurphy diectory, and "$" represents the cocoa command prompt:

$ ps aux | grep icc
if icc program is running, all is well. If no icc program, proceed...
$ xrandr

If output lists many resolutions, xhost IS running, so just try the ICC restart again
If output ends with a line saying "can't open display", xhost is NOT running, proceed...

$ /usr/bin/X :0 &
$ export DISPLAY=:0 [note this is for NewCocoa virtual machine]
$ .icc &
Repeat checks to make sure icc and xhost are running, then try connecting through TUI.
Did it crash again as soon as TUI connected? Shane had luck one time using DISPLAY=:3 instead, but the reason it worked is unclear.

Problem: During a run, a big streak of stray light appears on the STV images

This means the onband filter is in place for the STV so it is getting scattered light from the outgoing laser pulses. During normal operations, you want the offband filter to be in place.

In the STV tab of ATUI, you should already be in Focus Mode:
Press the Setup button (display probably says "Date/Time")
Press the Parameter button (display probably says "CCD Temp")
Press the up-arrow button four times, cycling through "Grid," "Night Vision," "Filter Wheel - Lunar," until you get to "Filter - Yes." This is the on-band.
Press the right-arrow to change the filter to "No," the off-band filter which is near but not at the laser wavelength.
Press the Focus button twice to return to Focus Mode, and hopefully the stray light is now gone!

Oops! Pushed a wrong button and put STV in a weird state (e.g. zoomed in)

The safest thing to do is start over, but you can restart just the STV and not everything else. Go to the Power tab of ATUI and turn off the STV (number 6), wait a few seconds, then turn it back on. Alternatively, in the command line of the Main/Active tab, enter "power 6 0," wait a few seconds, then "power 6 1". When you see the SBIG start screen on the guider, go to the STV tab and press the square black Focus button twice to go back to Focus Mode. Use the up/down arrows to adjust the exposure time as needed.

Error message: "Shot time out of range: 103.488018 not in [100.322917, 100.475694]"

This means your polynomial file was calculated for the wrong date or range of time, or else the moon is too low for ranging right now. Go back and check that you generated the polynomials correctly AND copied them to the /home/apollo/daily/ directory correctly. It may be necessary to delete all the old files from the date-specific directory and start over again.

Error message: "Lunar frc mismatch"

If you get a single instance of this error message, keep going. It happens sometimes. If you get them with every laser pulse during a run, stop the run by pressing the Standby button. There is something wrong with the software. Was there a recent software update to housctl? Can you revert to a previous working version? Consider restarting the ICC and housctl. Or, if you noticed any other error messages earlier on, investigate that problem. You will not be able to range until this problem is resolved.

Error message: "Failed to set initial ACS delay values during warmup"
This means one or both raspberry pi computers ("pirx" located in the upper right section of Utah, or "pico" located in the ACS cabinet on the mezzanine above the intermediate level) is turned off or misbehaving. This will probably require help from daystaff to fix it. Before quitting ATUI, go to the Main/Active tab and look in the lower section for the "Current rxx" and "Current rxy" values. Include these values in your writeup when you report the problem. In the event that the Rx mirror is lost when it wakes up again, knowing the previous position may be very important.

Error message: "Can't open CAMAC"
This means something has come disconnected on the CAMAC crate in the top left portion of Utah. Open the top section, pull the CAMAC crate out carefully, and wiggle wires and look for anything not plugged in.

4.3 Things to check regularly

Maintenance: Checking/updating APD-on-STV position

The STV guider receives light from a half-silvered mirror, so as a consequence the lightpath that leads to the APD science detector is not visible on the guider images. There is no apparent slit or hole to indicate where a target should be centered on the STV in order to send its light to the APD. This is the location where the STV crosshairs should be placed, which was a consistent 159, 72 position from 2017 to at least 2024. The APD-on-STV position does not wander by itself, seasonally or yearly, but it may shift any time work is done on the right side of Utah, especially any work that affets the APD, the STV camera, or the "snout" that feeds light to the STV.

Checking the current APD-on-STV position must be done on sky with instrument set to APOLLO using a bright star in conditions that are at worst mostly clear with moderate stable seeing. You will need to be able to see the ROOT window and also the STV guider image. At minimum you should do WARMUP to turn on both the APD and the STV, plus CALTDC to make sure the APD is awake. The main laser and keyswitch do not need to be on, but this procedure can be done during a normal run. Set crosshairs and mark the last-known correct position, then return the STV to focus mode.

Slew to a first or second magnitude star at altitude 45 or higher. In the bottom section of the Main/Active tab, set Nstares to 200 if you are just confirming the position, or 300-400 if you think the position may have changed. Center the star using Boresight offsets in the CCD offset frame. Once the star is close to the expected or previous position use the button on the left side of the ATUI window to enter the STARE state. Pro tip! No signal on the APD? Try a caltdc to make sure the detector is awake.

Now make your offset size 0.25 arcsec and use the APD frame. Look at the hitgrid in the top left of the ROOT window, and try to bring the star to the center of the hitgrid. Once you are satisfied with the centering, press the Display/Crosshairs button in the STV tab. You should see a static image of the star with crosshairs superimposed. If seeing caused the star to jump right at the moment you froze the image, you can press Focus twice to go back to focus mode, then Display/Crosshairs to freeze the image again. Press the Value button and then you can use the arrows in the STV controls to move the crosshairs to the position of the star. Click on the white+red display to make sure the numbers are up to date, then record the position you found. If the position has only changed by one pixel or so, it may not be a real change. But if it changed by several pixels, then make sure all the observing specialists know about the new position and also, ideally, edit this document!

Maintenance: Checking beam position

The Rx mirror (short for "Receiver" mirror) is reponsible for setting the beam offset to "shoot ahead, look behind" because we are ranging to moving targets. The motor is controlled and the position is remembered by the raspberry pi known as Pico, located in the top right portion of the Utah box. The beam offset is also called the vtarget position. The Pointer tab and the Laser Tuning tab show slightly different sketches of the rectangular STV field with the position of the square APD in the middle, and also a little circle which is usually offset to one side. That circle represents the beam offset for the position of the Moon (if you are looking at the Pointer tab) or the vtarget position (if you are looking at the Laser Tuning tab). The vtarget position is also shown in the "target" Rxx/Rxy shown in the Laser Tuning tab and the Main/Active tab, not the "current" Rxx/Rxy.

Pro tip: Remember that the "Go To Target" button tells the mirror to move to the target position, and this should be used several times during a typical ranging session. The "Commit Rxx,Rxy" button right below that does not move the mirror, but instead redefines the current position to be the same as the target position. This will be used more rarely, as described here. Pressing the Commit button at the wrong time is one way the mirror might become lost.

Coarse method:
Adjusting beam or "fid flower" off-sky

This method uses laser light returning from the fiducial corner cube attached to the secondary mirror to make the return beam visible as a six-lobed pattern (the "fid flower") on the STV guider, and then to make certain the beam is where it should be. This should be done with dome closed to avoid unnecessary laser light into the sky. This also means it can be done during the daytime or during poor weather at night, but it does require the primary mirror covers to be open, so this should not be done during a precipitation event.

This method also has a history of imperfect results, as if there may be some systemic offset between the beam measured by this method versus the fine-tuning method which can only be done on sky. Tom Murphy and Russet McMillan were never quite able to figure out the cause or exact characteristics of the apparent difference between the two methods. Therefore, this coarse adjustment method should be used only when there is a strong suspicion that the beam may already be off by a significant amount, and it should be followed up by the fine-tuning method at the first on-sky opportunity with decent conditions.

Pro tip: Only do this if the APD-on-STV position has been checked somewhat recently!
First, get the physical parts set up:

Dome closed
Tertiary facing BC1
Instrument set to BC1, sec and tert mirrors allowed to adjust
Mirror covers and BC1 eyelid open
Keyswitch on
Stairwell door closed and nobody expected to go up to the dome during the testing.

If you are interrupting a ranging run to do this test, you may have ATUI ready. If you are not set for ranging, here are the minimum steps needed to prepare the needed windows:

Open ATUI
Open ROOT plots window
Open browser window to video-35m and choose video 4, the STV guider (the video will be blank until the Warmup step below).

Once all that is ready, start the test:
In Main/Active tab: Take Control, StartNubs, Listen, ConnDev, and Get Status.
At left side of ATUI window, Warmup (STV video should now be in Focus Mode with the white square moving across the bottom of the image).
In Laser Tuning tab: Laser Powerup and Laser Warmup (note time of start of flashing).
In STV tab:
Display/Crosshairs, then move the crosshairs to the normal position (159, 72 as of summer 2024) and mark with tape.
Press Focus button twice to return to Focus Mode
Press the Setup button (display probably says "Date/Time")
Press the Parameter button (display probably says "CCD Temp")
Press the up-arrow button four times, cycling through "Grid," "Night Vision," "Filter Wheel - Lunar," until you get to "Filter - No." This is the off-band filter which is near but not at the laser wavelength.
Press the right-arrow to change the filter to "Yes," the on-band filter which allows the STV to see the laser light.
Press the Focus button twice to return to Focus Mode, then up-arrow to get the exposure time to the maximum "0.25s, 16x"

Has the laser been flashing 5-10 minutes? Check threshold and power and adjust the SHG. When this is done, the Continuum display should show that it is in Pgm2.
On the Continuum interface, open the laser shutter (you may have already done it after the power measurement).
On the left side of the ATUI window, press the button that says "tr dark" (near the bottom).
Confirm no one is in the dome and stairwell door closed.
On the keyswitch box, flip up the top toggle (now both toggles should be up) and press the green button.
On the Continuum interface, press the Start button (near bottom right corner).
On the STV video, you should now see a bright streak of scattered light, but more faintly behind that there should be a pattern like a six-petaled flower.
In the STV tab, press the Display/Crosshairs button, and now the crosshairs should appear on top of the frozen image of the flower pattern:

Now, the hard part is interpreting the position of the flower. In the above image, you can see that the center of the flower is offset below and to the right of the crosshairs. That is clear enough, but it can be hard to tell where the exact center of the pattern is. The central lobe of the pattern has a bright spot which is perhaps not at the center of the lobe. If you imagine an asterisk connecting each "petal" of the flower to its opposite, that might give a different center. Which one is correct? No one knows. This is why the fid flower method is only used for coarse adjustment.

The image above shows a vtarget position of Rxx=1 and Rxy=1, which is fairly typical for lunar ranging. In the Laser Tuning tab, the sketch of how this should look is represented like this:

Remember, on the STV, positive Y is down, which is why a positive Rxy offset moves the pattern lower - so that part is okay. But it looks like the offset seen on the sketch is larger than the one seen on the STV. Possibly a scale miscalculation affects the sketch? In order to avoid this problem, we will tell the beam to center on the APD position, and then we will not have to worry about how big is 1 arcsec, we only have to worry about centering of the flower pattern.

To adjust the vtarget position and refresh the STV display:

In the command line of the Main/Active tab, enter vtarget 0.0 0.0
In the Offsets section of the Main/Active tab, press Go To Target.
In the STV tab, press the Focus button twice to go back to Focus Mode.
Verify the image on the STV still looks okay by comparison with your tape mark.
Optionally, you can cycle between Focus and Display/Crosshairs to freeze the image with crosshairs on top.

Now the fid flower and the sketch in the Laser Tuning tab look like this:

The center of the flower pattern is now closer to the crosshairs but still not perfect. Use the vtarget ## ## command to make adjustments to the position by increments of 0.25 arcsec until it looks as good as you can get it compared to your tape mark. When it looks good, check the crosshairs at least one more time and get a screengrab for future reference. Then press the "Commit Rxx, Rxy" button in the Main/Active tab to redefine this position as the new 0, 0.

All done except for cleanup!

In the Continuum interface of the Laser Tuning tab, press the Stop button (bottom right of the 4x4 square).
In the STV tab, press Setup, then Parameter, then up/down arrows if needed to get back to "Filter - yes" and do a left-arrow to change to "Filter - no". If the onband filter is in place during ranging, the STV will show scattered light from the outgoing laser pulses. Pro tip! This step is really only needed if you are going to move on/back to ranging - if you leave the filter at Yes, it will move back to No at the start of the next session.
On the keyswitch box, flip the top toggle down again. The red button should light up. You can now turn off the key.
On the left side of the ATUI window, press the "tr sync" button to move the TR mirror back to normal position.
On the left side of the ATUI window, press "Cooldown" (unless you want to return to ranging!)

Fine method:
Adjust beam position on sky while ranging

The most accurate way to ensure the beam is really where it should be is to optimize signal on A15 while ranging in clear conditions with stable seeing. Because of the requirement for good conditions, the need to spend ranging time on this optimization, and the fact that the method works best when the returns are at least moderately strong to begin with, this method is used for fine-tuning rather than trying to get the beam back if the mirror is really lost; in that situation, start with the coarse adjustment method first, and then use this method to improve upon it. For regular maintenance purposes, this task should be done a few times a year even if there is no indication of anything wrong, whenever there is an opportunity with good return signal strength.

Acquire A15 and get it centered on the hitgrid; signal must be at least moderately strong, with a rate at least 0.25 photons/shot and hopefully stable.
Press the "Go To Target" button to make certain the current Rxx/Rxy is at the target position.
Start a fresh run and make sure the signal is still centered on the hitgrid.
Set the offset type to Raster Beam, the offset frame to APD, and the offset size to 0.25 (optionally 0.5 if you suspect the current position is off).
Note: This offset type adjusts the Rx mirror and also moves the telescope in the other direction, to keep the signal centered.
Make an offset to the right and wait at least 10s to see if the signal changes.
If signal got stronger, keep moving in the same direction until the signal starts to drop, then move back to where the signal is best.
If signal got weaker, move back to the starting position (where current Rxx/Rxy matches target), then try moving left instead.
Once you find a good position by moving left/right, try adjusting up/down in the same way.
Iterate between right/left offsets and up/down offsets to get the best signal you can, until the end of the run.

Are you confident that your adjustments made the signal better, and is the accumulated adjustment more than 0.25 arcsec? If yes:

Reslew to A15 to update the target Rxx/Rxy.
Write on your paper log the offset: current x minus target x, current y minus target y.
Press the "Commit Rxx,Rxy" button to redefine the current mirror position as matching the target, and note on your paper log that the change was committed.

If you are not so confident the adjustments helped, or if the offset is quite small:

Write on your paper log the offset: current x minus target x, current y minus target y.
Do not commit the change, note in the log that it was not committed and the change may or may not have helped.
You might decide to commit the change later if more data looks like it was helpful, but remember the target position will update with every slew, so consider whether it makes sense to press "Go To Target" and get rid of your earlier changes in order to start over. Fortunately you logged the maybe-good offset from your first try so you can repeat that if needed!
Pro tip: If you are trying to get back to a previous beam position, offset type Raster Beam with frame Native will increase Rxx for a right-arrow and Rxy for an up-arrow.

Regardless of whether you did or did not commit, you can continue ranging to other reflectors, and make iterations to the beam position any time you are getting reasonably strong returns.

The above method is the preferred approach by Tom Murphy for peaking up a good signal and keeping it centered on the APD at the same time.

An alternative method is to note whether one edge or corner of the APD seems to be more "hot" or more sensitive than the rest. For example, if you have a strong signal on the left edge of the detector which suddenly gets weaker when you use Raster Scope offsets to center the signal on the hitgrid, that might mean that the beam is out of position to the left. Use Raster Scope offsets in the APD frame to put the signal on the left side where it looks strong, then Raster Optics offsets in APD frame to pull the signal to the right, and hopefully you can achieve a signal which is both strong and centered. An advantage to this approach is that the telescope and Rx mirror offsets are done separately, which might make it easier for a human brain to keep track of the effects. The disadvantage is that you could end up on a wild goose chase if there is a non-beam reason for one side of the APD to be more sensitive (e.g., a misalignment of the lenslets in front of the APD). For this reason, do not use the Raster Optics method to make large changes, but it may be suitable for small quick tests during ranging.

Poor telescope pointing

How to do PPS sync

No laser power

No laser in dome / no fid returns

Yes fids but no lunar returns

Yes fids, but shot rate is dropping below 18

To access the secret control panel in the Phaselock program, use Ctrl+Shift+F9. This screencap of the control panel from 2022 may have wrong values?

In June 2024, the piezo_target value mysteriously changed from 65 to 50. After that, minimizing the "difference frequency" left us at a piezo voltage of 50 and also kept the rms jitter low. After a few experiments with forcing the target voltage back to 65, causing more jitter and larger phase mismatch, we concluded that 50 was the best value even though the cause of the change was unknown. This screencap of the control panel is from September 2024 showing the updated values:

APOLLO Operator's Guide

Table of Contents

1. Hardware

3. Operations

4. Troubleshooting and Maintenance

Coarse method:

Fine method: