APOLLO Operator's Guide

Last updated: June 8, 2024 - RM

1. Hardware
  1.1 The Utah box
    1.1.1 Opening and closing Utah
    1.1.2 Left side / laser cavity
    1.1.3 Right side / receiver
    1.1.4 Top section
  1.2 The obs-level cabinet / "phone booth"
    1.2.1 Opening and closing the cabinet
    1.2.2 Top section
    1.2.3 Middle section
    1.2.4 Bottom section
  1.3 The ILE
    1.3.1 Opening and closing the ILE
    1.3.2 Right side
    1.3.3 Left side
  1.4 The ACS cabinet
  1.5 TBAD
  1.6 Interlock system and keyswitch
2. Software
  2.1 ATUI
    2.1.1 Tabs along the top
    2.1.2 Control/status buttons on left
    2.1.3 The Plots window
  2.2 housctl
  2.3 ICC
  2.4 Miscellaneous scripts
    2.4.1 TBAD monitoring
    2.4.2 Web page server
    2.4.3 Generating quarterly schedule
    2.4.4 Generating notification files
    2.4.5 Generating slew commands

3. Operations
  3.1 Periodic communications and checks
    3.1.1 Summer of odd years: LOD
    3.1.2 Quarterly: FAA and Space Command
    3.1.3 Monthly: TSIM test
  3.2 Preparation and warmup
    3.2.1 Day before: Request NOTAM
    3.2.2 Hour before: Set up observing log
    3.2.3 Hour before: Prepare ATUI and other windows
    3.2.4 Hour before: Check GPS clock
    3.2.5 Hour before: Make polynomial files
    3.2.6 Hour before: Check space block files
    3.2.7 Half hour before: Check out ACS
    3.2.8 15m before: check APD
    3.2.9 15m before: STV crosshairs
    3.2.10 5m before: warm up main laser
  3.3 Ranging
    3.3.1 Pointing and focus check on a star
    3.3.2 Pointing check on a crater
    3.3.3 Laser threshold and power check
    3.3.4 First target: Apollo 15
    3.3.5 Repeat Apollo 15 with ACS
    3.3.6 Other reflectors
    3.3.7 Finish and cooldown
4. Troubleshooting and Maintenance
  4.1 Pulse train and BUT

1. Hardware

1.1 The Utah box

The Utah box is a box shaped like the eponymous state, attached to the rear of the telescope's mirror cell adjacent to port BC1. The box contains an optical bench which holds the main APOLLO laser (the "Leopard" laser), the optics which send the outgoing light to the telescope and receive the incoming light from the Moon, the APOLLO detector (the APD), and various electronics to support the laser and detector.

Heating and cooling systems attempt to maintain the temperature inside Utah at 20+-1 C. The temperature may get higher while the box is closed and the laser is in use, or lower if the box is open in cool weather. Laser performance will be affected if Utah temperatures go above 25 C or below 15 C.

1.1.1 Opening and closing Utah

In order to access Utah, you will need the telescope at high altitude: 80 degrees for a tall person to reach, 83-84 for a short person. The Utah box has three sections: the left side door is the most frequently accessed and also has to be opened first, the right side door is the second most common, and the top part rarely needs to be opened, which is good since it is not on a hinge and therefore is the most difficult to deal with. Each section has a number of small fiddly latches which are prone to getting out of alignment or sometimes getting trapped in between sections so that they can't be properly latched.

To open the left door, undo three latches at the top of the door, three down the middle, and three underneath the left door (one of those may be unlatched already because it is permanently out of alignment). Gently pry the door open at the middle. If you are going to be working inside Utah for more than a minute, you may want to tape the left-side door to the standing cabinet to hold it out of the way, otherwise it will fall closed again.

To open the right door, with the left door already open, undo an additional two latches at the top of the right door and two latches underneath the bottom of the right door. Be careful! There are some extra latches further underneath the right side which are for accessing the APD which you do not want to unlatch in the normal course of things. So make sure that the latches you do open are near the edge of the underside. Once the latches are undone, you can fold the right-side door all the way open and out of your way.

To open the top of Utah, first open the left and right doors, then place a small stepladder directly in front of Utah, and climb at least partway up the ladder. Undo three additional latches on the top of the right side, two latches on the center vertical portion, three latches on the top of the left portion, and three on the leftmost vertical. Pull the ell-shaped cover gently clear and lower it down to the floor out of your way.

Reclosing the top section is tricky because these latches are the most likely to get trapped in between the box and the cover. Stand on the ladder and pick up the cover, holding it with the outward side down so that gravity will open the latches. Then bring it upright and into position carefully and fasten one latch (the rightmost latch on the top section is usually a good choice). With that latch closed you can support the cover with one hand while checking that all the other latches are free, and if necessary open a small gap to free any that are trapped. Once they are all free, you can fasten the remaining latches.

Reclosing the right door is the simplest because it has the fewest latches and they are mostly well-behaved. When you close the left door, sometimes the latches along the center seam get trapped, so check those are free before you start fastening any of them. You may find that you need to press the left door closed with one hand until you have two or three latches fastened (top and center are recommended), and after that you can let go.

1.1.2 Left side / laser cavity

The left side of Utah houses the laser cavity where the light (both IR and green light) is generated. After the insulated Utah door is opened, there is an additional heavy door blocking off the laser cavity; that door is not fastened in place but just held by gravity. Before opening the laser cavity door, make sure that the laser is not flashing OR that all people in the dome have been provided with laser safety glasses. When the laser is flashing and the cavity is open, there are eye safety issues but also there can be danger to skin if anyone reaches into the cavity. Only have the cavity open with the laser flashing if you know what you are doing.

As long as the laser is not flashing, there is no danger to humans in opening the cavity, but try to minimize the amount of time it is open in colder weather.

Oscillator

Laser shutter

AOML

Rear mirror

Amplifier

Bottom mirrors

Frequency doubler

Bolometer

Interlock shutter

1.1.3 Right side / receiver

FPD

T/R mirror

"Snout"

STV

Diffuser

ACS

APD

1.1.4 Top section

CAMAC crate

Raspberry Pi

Flip mirror switch

1.2 The obs-level cabinet aka "phone booth" aka "TARDIS"

The APOLLO cabinet on the observing level holds supporting electronics and low-to-medium voltage power supplies for the laser and detector, some control interfaces (most of which can also be controlled remotely), and the computer named houston along with accompanying monitor, keyboard, and mouse.

The cabinet is insulated so that it doesn't cool down too much, and kept warm mostly by passive heating. If the cabinet gets too warm active ventilation to the ILE will start up; the temperature sensor in the cabinet has a history of incorrect high-temperature readings, but as of April 2023 the sensor was recalibrated and should behave better.

1.2.1 Opening and closing the cabinet

The cabinet has metal panels on the outside and insulated panels on the inside. The aluminum panels are held on by velcro, so just grab the knobs and yank. The middle aluminum panel must come off first, then your choice of top or bottom. Set the removed panel(s) to the side.

The insulated panels are each held by four small black latches which fasten into vertical rails. The top panel must come off first; turn the latches vertical to release, then gently wiggle the insulated panel free. Optionally, if you only want to access the middle, you can just push the top panel up as high as it will go and undo the latches for the middle panel. Set the removed panels to the side.

To reclose the cabinet, set the bottom insulated panel in first and tighten the latches to hold it to the rails. Then the middle panel insulated panel should go directly on top of the bottom one, and the top panel directly on top of the middle. There will be a gap between the top panel and the "roof" of the cabinet. Once the insulated panels are in place, press the bottom and top aluminum panels against the velcro with the top one flush against the top of the cabinet; then the middle aluminum panel should fit neatly between the other two with overlapping lips.

1.2.2 Top section

Bolometer control paddle

STV control box

Flow meter

Osc & Amp tuners

1.2.3 Middle section

Continuum control interface

houston KVM

1.2.4 Bottom section

houston tower

power supplies

1.3 The ILE / "clubhouse"

The Intermediate Level Enclosure (ILE), also sometimes called "the clubhouse", is located on the ceiling of the intermediate level but is accessed from the observing level from the pit behind the telescope.

The ILE contains cooling systems and high-voltage power supplies for the main APOLLO laser, plus the main driver of the interlock system that controls the safety shutter determining whether laser light is allowed out to the telescope.

The temperature inside the ILE should be maintained between about 5 C and 35 C, and must be kept above freezing because cooling water is held in reservoirs in the ILE. A small heater with its own thermostat will come on when the temperatures get below 5 C. In warmer weather or when the laser is in use, ventilation systems controlled through the housctl program can run fans and open one of the ILE louvers; the other louver is kept open in summer and closed in winter. As of 2022 the main temperature sensor for the air in the ILE was malfunctioning and reading high, causing ventilation even when the dome air was very cold. In January 2023 the ventilation decisions were changed to be based on another temperature sensor on the wall of the ILE, instead of the one hanging in the air in the middle of the ILE.

1.3.1 Opening and closing the ILE

You can reach the ILE by going underneath the telescope if it is at high altitude, going through the NA2 pit, or crawling down from the observing level floor in the Crush zone behind the telescope and below Utah, if the telescope is at low-to-medium altitude. The railings around the crush zone can be retraacted, and there are two metal steps (one small and one large) that fold out of the wall just behind the altitude drive disk to facilitate climbing up and down.

The two doors each have two black latches which are unlocked with a key, which should be kept on top of the gray electronics box just to the right, at the edge of the NA2 pit. If there is no key there, a spare should be in the bottom left corner of Utah, outside of the laser cavity. Another spare may be somewhere in the APOLLO supply cabinet next to Ben's cubicle in the main lab of the Ops Building. Use the key to loosen the latches, then slide the left door to the left or the right door to the right. Pull the door free and set it out of the way; avoid tangling cables or wedging under the telescope skirt.

One or both doors may be opened at a time. In sub-freezing weather, try to limit the amount of time the doors are opened since the ILE must be kept above freezing and it only has a small heater. There are two fluorescent lights at the back of the ILE with simple rocker switches: one on the left (crawl straight back on the black pads and then look up) and one on the right (crawl behind the cooling and electronics units to reach this, or possibly a long-armed person can reach it over the main electronics rack). Make sure the lights are turned off before closing up the ILE.

To close the ILE, slide the non-latched side of the door under the metal holding lip, push the latched side flush, and tighten the latches. Once both doors are latched, return the key to the top of the gray electronics box on the NA2 side.

1.3.2 Right side

Laser rack

Power Relay

Main Breaker

PU 610

PU 620

MV 70

Water reservoir

Main pump

Aux pump

Heat exchanger

M75 chiller (water only)

1.3.3 Left side

Light / Louvers

Power

M33 chiller (water/PG mixture)

Allen Bradley unit

1.4 The ACS cabinet

The cabinet for the Absolute Calibration System (ACS) is located on a platform in the rafters of the intermediate level, between the hanging housing holding the NA2 Lakeshore controllers and the stairs to the observing level. It can be accessed from the intermediate level by stepladder - be sure to press a stop button next to where the ladder stands, since it has to be set up in the pink portion of the floor.

Laptop: acs-laser.apo.nmsu.edu

Cs clock control

PPS check

1.5 TBAD

The functional part of the TBAD system is located on the top end of the telescope above the secondary mirror, facing the sky. The power unit for the TBAD system is on the NA2 side electronics box on the front of the primary mirror cell. Additionally, the TBAD system uses telescope telemetry passed through a terminal server on the intermediate level.

See also Monthly: TSIM test under Operations

1.6 Interlock system and keyswitch box

The interlock system is controlled by a very low-level Allen Bradley "computer" which does not have a monitor or keyboard and which nobody is currently capable of reprogramming. The primary Allen Bradley unit is in the back left corner of the ILE on the floor [get photos to document the desired state of this thing!].

A secondary Allen Bradley unit is located in the 3.5m control room on a shelf under the desk in the left-hand corner; this unit must be turned on in order for the keyswitch box on top of the desk to work. In this picture, taken with the laser powered off, there are two red LEDs flashing, and a green "COMM" message illuminated on the readout.

The keyswitch box in the 3.5m control room is the primary interface for a human to control the interlock shutter which allows laser light out of Utah into the dome along the telescope's light path. This box does not control the laser power. If Utah is opened, laser light can shine into the dome regardless of the state of the interlock shutter.

When the red button is illuminated, like the image on the left, that means the interlock shutter is closed and laser light cannot travel through the telescope light path. When the green button is illuminated, like the image on the right, that means the interlock shutter is open, and ranging is possible.

In order for the interlock shutter to open, multiple conditions must be met, each matching a status LED on the keyswitch box:

The key must be turned on so the box has lights!

Laser power must be on (top LED "LASER AC PWR" must be lit solid red).

The stairwell door on the intermediate level must be closed (second LED "Dome Access" must be lit solid green).

Telescope motors do not need to be active (third LED "TELESCOPE STOP" may be either blinking or solid)

TBAD must be powered on, which requires dome open AND it must not see a plane within its pointing area (fourth LED "APOLLO CONTROL" lit solid green).

OR a bypass-dome file exists on cocoa (allowing TBAD to be turned on with dome closed), AND TBAD does not think it sees a plane inside the dome (so the TSIM box is not plugged in).

OR the top "BYPASS" toggle switch is up, so TBAD is ignored ("APOLLO CONTROL" LED will be blinking instead of solid). ONLY USE THIS FOR CLOSED-DOME WORK! NEVER BYPASS TBAD IF LASER LIGHT CAN LEAVE THE DOME!!!

The fifth LED for "SKY SENTRY" refers to an IR plane-spotting camera which is defunct, so its status (usually blinking green) does not matter.

The sixth and seventh LEDs for human spotters are no longer used but the interlocks are still active, so the bottom "BYPASS" toggle switch needs to be up. These LEDs should be blinking green.

If some condition caused the shutter to close previously - for example, if someone opened and then closed the stairwell door - the green button has to be pressed again to re-open the shutter after the interlock condition is resolved (after the door is closed).

The normal operating state for the keyswitch box is to have the top toggle down or not-bypassed (TBAD can close the shutter if it sees a plane) and the bottom toggle up or bypassed (human spotters are not present to look for planes).

If you look closely at the two example photos above, you can see the photo with the shutter closed has the toggles in the correct state. The interlock shutter will not open in this case because of two conditions: 1) the laser is not powered on, and 2) TBAD is not operating because the dome is closed. So the first and fourth LEDs are not lit, and the shutter will not open. The second photo, with the interlock shutter open, shows that the laser power has been turned on, so the top LED is lit solid. Also, the top toggle is in the up position so TBAD is bypassed. The fourth LED is blinking green (not obvious in the still photo) to indicate the bypass state.

A second keyswitch box is available in the blue-lidded APOLLO supplies bin on the observing level to allow the interlock shutter to be controlled by someone working in the dome (in which case they will definitely be wearing laser safety glasses, also available in the blue-lidded bin.) This keyswitch box attaches to a cable which emerges by the floor grating just below Agile; look for the tape with arrows to align the connectors with each other:

The upstairs keyswitch box has the same conditions as the downstairs box in order to open the interlock shutter, plus two additional conditions: the downstairs box needs to be turned on in order for the upstairs box to work, and also the upstairs box has to be in a good mood because it's flaky and sometimes it just doesn't want to work. You may need to coordinate with someone downstairs to open the interlock shutter for you.

2. Software

2.1 ATUI

ATUI is the APOLLO control interface inside TUI, equivalent to an instrument control window and accessible from the Instrument menu for APOLLO-enabled TUI installations. But in fact ATUI is a complex of multiple control windows, each accessed by different tabs along the top of the window.

2.1.1 Tabs along the top

Active/Main Tab

The first tab is listed as "Active" in some versions of ATUI and "Main" in other versions. In this document I mostly call it the Main/Active tab. Some ATUI versions have a bug where the name of the tab disappears anyway, so the name is not important, but the tab is. This is where most of the ranging work takes place.

Take Control

The Take Control button in the top right of the Main/Active tab is like giving yourself permission to control things through ATUI. When pressed, the label on the button changes to say Houston: Active Control. You can press the button again to rescind control. The primary purpose of this button is just to avoid accidental button-presses. More than one person can have control at a time, in which case they should be coordinating with each other to avoid command collision.

StartNubs / Listen / ConnDev / Get Status

These four buttons in the middle of the Main/Active tab are pressed, in order from top to buttom, to connect ATUI to the housctl and ICC programs and make sure those programs are talking to each other and the instrument. When planning to run anything from ATUI, you will usually start with these four buttons. If you or someone else did all four buttons recently and then disconnected, you may only need the Listen and Get Status buttons to re-connect your own ATUI without going through the rest of the initialization. If you were disconnected for longer than a couple of hours, it's safest to just use all four buttons.

Bottom section

The bottom half of the Main/Active tab has buttons, number entry boxes, and sliders which are primarily used while ranging. Some of these settings will be changed frequently, and some of them should never be changed; see the image below for reference and be careful!

Log output and houston command line

In the middle of the Main/Active tab is a command line which highlights with a cursor when you click on it, and just above that is a window showing 7 lines of recent output, where you can scroll up to see previous output. This is effectively a TUI Log window where the output is filtered to show only housctl messages, and the command line sends commands directly to the actor "houston" which is the housctl program. You can change the severity filtering and search for text with the "Replies" and "Find" controls just above the mini-log.

Run Statistics

The Run Statistics section at top left of the Main/Active tab has information about the successfulness of the most recent ranging run. You will keep an eye on some of these numbers during ranging to see if you need to change anything. Then, after each run, you will copy some information from this section into the paper observing log. Be sure to grab whatever information you need before starting the next run!

The only control button in the Run Statistics section is a button saying "Reset Everything for New Run." This used to be needed in an older version of ATUI to refresh the plots and statistics, but it's mostly defunct now and should not be needed. You can experiment with it sometime in between runs if you want to try.

Offsets

The Offsets section in the top right of the Main/Active tab includes controls for centering up targets, keeping them centered, searching for signal, and keeping the signal. Some of the controls are non-intuitive for observing specialists.

Four offset types are possible, chosen with mutually-exclusive checkboxes. It's important to make sure you are moving the right thing!

Boresight offsets are used for initial telescope centering before ranging begins

Raster Scope moves the telescope with Guide offsets, used during ranging to see if tracking problems accumulate over time.
Raster Optics adjusts the position of the receiver (aka "rx") optics; this is the "shoot ahead, look behind" adjustment that compensates for the moon's motion while the light is traveling to and back from the moon. Moving these optics will cause the moon to shift position on the guider so it's like offseting the telescope, but it also has an effect on signal strength.

Raster Beam adjusts the receiver optics and also offsets the telescope in the opposite direction. The ideal is that you get signal centered up, and then you can use Raster Beam to optimize signal strength and not worry about having to follow the tracking at the same time. In practice, it isn't quite simple. Raster Beam might be useful for certain tests, but Russet generally prefers to Raster Optics by themselves and then adjust the telescope position separately.

Once you have decided what kind of offset to use, the size of the offset is set in the Offset Increment box. Enter the desired value and then press return to make the pink go away. The maximum offset permitted through this interface is 10 arcsec, generally used for Boresight offsets during the initial centering. While ranging, the Offset Increment will usually be less than 1 arcsec. Moves larger than 1 arcsec will be Computed offsets with sounds and a countdown in TUI. Smaller offsets will be uncomputed and basically instantaneous (but remember the round-trip time to the Moon is about 3 seconds, so even an instant offset takes a while to change the returns!).

The three checkboxes on the right determine what coordinate system or "frame" the arrows will move in.

"Native" is the natural alt/az coordinate system of the Guide or Boresight offsets, where the > button increases az or X by Offset Increment, and the ^ button increases alt or Y.

"APD" is the frame of the APD science detector, displayed in the hitgrid at the top of the Plots window; the > button pulls the signal to the right on the hitgrid, and the ^ button pulls the signal up.

"CCD" is the frame of the STV guide camera, displayed in a browser window; the > button pulls the star or crater to the right, and the ^ moves the star or crater up.

Fun fact: Offseting in the wrong frame is the most common mistake to make while ranging!

If you get lost, you can use the Home button in the center of the arrows to return the Guide offset to 0, 0 (if Raster Scope is checked) or the Boresight offset to 0, 0 (if Boresight Offset is checked). If you accidentally moved the rx optics, the Go To Target button will bring them back to the desired shoot-ahead-look-behind position. Please use care not to press the "Commit Rxx, Rxy" button unless you're really sure what you're doing (described in the Operations section.)

Pointer Tab

Show / Hide craters

Numbers table

Graphic Interface

Month / Day / Hour / Minute forward and back

Slew Telescope (always for Now)

Polynomials and Vtarget

Plot Control Tab

The Plot Control tab is somewhat useful, but has bugs. Most notably, when you enter a number in a box and press return, the number will stay pink or perhaps change briefly to white and later back to pink. Some of the numbers seem to have no effect when changed, but others do take effect even though they stay pink. Some of the labels are incorrect, for example the APD Hitgrids Time Window says it shows a certain number of hits which implies incoming photons, but in fact it's showing the hits that came in during the displayed number of outgoing shots.

The default settings are reasonable for strong return signals and it's okay to leave this tab alone completely. In the case of weak signal it may be helpful to increase APD Hitgrids Time Window from 200 to 400 and the Return Time Histograms window from 200 to 800. Those are Russet's preferred settings.

Alarms Tab

The main body of the Alarms tab is like a TUI log window which shows APOLLO-specific warning messages of a certain severity level ("Error" by default). The severity level appears to be adjustable but may not actually change in some versions of ATUI. Use this window to check on error messages which went by too quickly to catch. Once you have decided if a message is or is not a problem, you can press the Acknowledge Alarms button which should cause the name on the Alarms tab to change from red (or invisible) to black, until another error message comes through.

Below the main section is another section dealing with blocks from space command. If there is no block, it will say "Time remaining in release: ### s" in green (or in yellow if there's only a short time remaining). If there is a block, it will say "Time remaining in blockage: ### s" in red. If no blockfile exists, it will say there are many nines of seconds remaining in the blockage. Whether there is currently a blockage or a release will also affect the color (or perhaps visibility) of the tab for Main/Active.

If there is a block currently in place, you will not be able to open the interlock shutter (green button on keyswitch box). Put a blockfile with valid times in houston:/home/apollo/daily/housctl.blk and press the readblock button on the left side of ATUI, or use the "unblock #" command to permit opening for a certain number of minutes.

Raster Tab

The raster tab is an attempt at automating a search raster which was never fully implemented and probably doesn't work. Best leave it alone.

STV Tab

The STV tab is mostly a visual display of the physical control box located in the observing level cabinet. This interface is used for controlling the APOLLO guide camera. [Add a link to an SBIG manual maybe?]. There are some additional buttons at the bottom of the page which haven't been tested and maybe don't do what you think they do. Or maybe they do.

Channels Tab

The Channels tab is very simple: A 4x4 checkbox grid, and a 4x4 grid of numbers. This is a reference and control for the hitgrid plot in the Plots window. If it seems like a particular pixel of the detector is always hot, you can come to this tab to identify which number channel that pixel is; for example, the bottom-left pixel is known as "channel 12." Then you can uncheck the misbehaving pixel in the grid of checkboxes so that its output will no longer be displayed in the hitgrid and won't make it impossible to see what the other pixels are seeing. Note that unchecking only affects the display of the hitgrid, it does not actually turn off one of the pixels of the detector. So if you do see misbehavior, you should also report which channel is misbehaving to someone who can do something about it.

Power Tab

The Power tab is just a bunch of checkboxes which allow you to turn on and off individual components of the APOLLO systems. This is generally only used for testing or troubleshooting. During normal operations, groups of components will be turned on and off at the same time as the system state changes from Idle to Warmup to Run to Standby to Cooldown, and so forth. There are also automated environmental checks in the housctl program which may turn heaters or fans or chillers on and off as needed. It's usually best not to mess with them unless you're trying to unstick something that's stuck, such as the GPS clock (checkbox #24 on the list).

Laser Tuning Tab

The Laser Tuning tab is used to control the main APOLLO Laser, also known as the Leopard laser.

ROOT Tab

The ROOT tab is used for connecting to and disconnecting from the separate python program which runs the Plots window. You will need to press the Connect to ROOT button before opening the Plots window, and press the Disconnect button before you try to close the Plots window. If you forget to disconnect, you might have to force-quit the python program.

ACS Tab

The ACS tab is used to control the Absolute Calibration System.

2.1.2 Control/Status buttons down the left side

Idle / Warmup / Standby

Run

Fidlun

Caltdc

Stare / Dark / Flat

Lpower

Lasercal

Cooldown

TR sync / clear / dark

Readblock

Spotter 1 / Spotter 2

2.1.3 The Plots window

[This is the latest I have: After clicking the CONNECT button in the Apollo/ROOT tab, in Terminal window: python3 Apollo_Plots_ROOT.py twm2.json. Need to know if this is going to be stable - what about directories?]
The plots that are most useful for ranging are on the left side of the window, to make it easier to observe with overlapping windows. Some of the other plots are useful during checkout and troubleshooting.

2.2 housctl

housctl is half of the APOLLO control program, which runs on the machine called houston. housctl takes input commands from the hub and sends them to various APOLLO subsystems, and housctl also controls many of the subsystems including environmental controls and the gps clock. If the housctl program is not running, ranging will not be possible and also the big Leopard laser could be in an unsafe state with either cooling or heating running without feedback control.

To check that housctl is running properly, login to observer@houston and run
$ ps aux | grep housctl

There should be four or five housctl processes running, depending on who else is connected. If more than six processes are running they may get hung up. If one or no processes are running, housctl is not working properly. In either event, housctl will normally be automatically restarted within a few minutes by a cronjob on houston, which also sends an alert email saying that it needed to do a restart.

However, sometimes a human needs to restart housctl, maybe for a software update or if housctl is running but not communicating correctly with the ICC program on cocoa. In this case, try to kill the old housctl process and restart the new process within a few minutes so that you do not collide with the cronjob which might try to restart housctl at the same time.

To kill the old housctl, login to root@houston and run:

$ ps aux | grep housctl

Identify the PID for the oldest process (lowest numbered, first on the list).

$ kill [PID]

This should kill all of the housctl processes running. Check to be sure:

$ ps aux | grep housctl

To restart housctl, working as root@houston:

$ cd /home/apollo/

$ nohup ./housctl &

$ ps aux | grep housctl

Get the PID for the oldest, lowest-numbered process (also make sure there are only four running, or you might have to kill and start over!)

$ renice -20 [PID]

2.3 ICC

The ICC is the other half of the APOLLO control program, which runs on the machine called cocoa. It handles the lunar ephemeris, science data, and interlocks. If the ICC is not running or if it is not communicating properly with housctl, it will not be possible to do ranging.

The most common symptom that the ICC is either stopped or not communicating with housctl is that when you initialize your TUI session with StartNubs / Listen / ConnDev / Get Status, you will see multiple error messages saying "NoneType object has no attribute 'call'." Sometimes the error will be "the target named apollo is not connected." If there is a single example of the error, you might be okay, or you might be safer restarting. If there are multiple error messages, you definitely need to restart. Most often it will be best to restart the ICC followed by restarting housctl, to make sure that both programs are talking to each other.

First check whether or not the ICC is running. Login to observer@cocoa and run
$ ps aux | grep -i icc
      If the ICC is already running, look for more signs that it is not running properly. If you are confident that it is not, use the PID to kill the process. Do not restart the ICC program if a copy is already running.

To restart the ICC, login to root@cocoa or, if already logged in, su root.

$ export DISPLAY=0:1

[NOTE: As of 2022, this command is for the physical cocoa machine. If the virtual cocoa is running, use 'export DISPLAY=:0' instead.]

$ cd /home/tmurphy

$ ./icc &

$ ps aux | grep -i icc

Sometimes it will be sufficient to restart only the ICC, but most often you may find it necessary to restart housctl right afterward to make sure the connection between the two programs is correct.

2.4 Miscellaneous scripts

2.4.1 TBAD monitor

The TBAD monitor script on cocoa logs the state of TBAD. If it is not running, the TBAD system can still operate and close the interlock shutter when a plane moves close to the laser beam, but the output of TBAD will not be logged and checkable. Once a day at 17 UT, a cronjob checks the status of the TBAD system, and if the monitor script is not running there will be an email alert.

After a reboot of cocoa, the monitor script may need to be restarted. To restart the TBAD monitor script, login to root@cocoa and
   $ cd /home/apollo/xponder
   $ ./xp_monitor.py &

After this, logging should resume. You can run a TSIM test to confirm.

2.2.2 Webpage server

cocoa hosts some webpages with important information about the status of the APOLLO systems, including two-day plots of temperature / coolant flow / GPS clock behavior and also tables showing all the temperature sensors, what systems are on, and error messages from housctl. The tables update every 10 seconds.

Direct access to these webpages is at Tables and Plots. However, these links are not open to the world and may only be accessible from onsite, in order to avoid burdening cocoa with web-server duties when its primary job is handling APOLLO data. Russet McMillan and Tom Murphy have been able to reach these pages with a tunnel using ssh forwarding, but other people from UCSD and NASA have been unable to make the tunnel work.

Therefore, the pages are echoed on the regular APO website here: Tables and Plots. This copy of the pages updates once per minute, and might need to be refreshed by hand in some browsers.

After a reboot of cocoa, the web hosting may need to be restarted. To restart the webpages, login to root@cocoa and
   $ cd /home/environment
   $ python htmlmonitor.py &

2.4.3 Generating quarterly schedule request

/home/apollo/ephem/pytools/longplan.py 04 01 2023

2.4.4 Generating notification files and emails

2.4.5 Generating crater slew commands

3. Operations
3.1 Periodic Communications and Checks
3.1.1 Summer of odd years: get new Letter of Determination from Laser Clearinghouse

Every few years (next up in July of 2026), renew the Letter of Determination with FAA. This is really a job for Nancy Chanover and/or Stephen Merkowitz, but the operator should be aware of it. The date of the LOD listed in the quarterly email to FAA should be updated when a new letter comes into force. Do not operate if the LOD is expired.

3.1.2 Quarterly: communicate with FAA and Space Command

Before the beginning of a new quarter, run the notify.py script on the APOLLO instrument schedule for the upcoming quarter.

Input file is created by going to the quarter schedule page and scrolling down to Instruments, APOLLO, click on Schedule, copy the table into a text file with a name like 2021_Q1.sched. Example file:

./notify.py 2021_Q1.sched

Example file to go to FAA:

Output files include 2021_Q1.faa (this has all the ranging sessions scheduled for the quarter, but FAA actually wants to be informed about one month at a time), 2021_Q1.csv (to be uploaded to space-track.org website), and 2021_Q1.ssc (defunct).

Before the beginning of the new quarter, log in to https://www.space-track.org/, go to Files - Upload, select destination /zLCH/Apache Point/PRM , then use Choose Files to find the .csv file on the local machine to be uploaded.

Before the beginning of a new quarter, copy the addresses from the .faa file into an email, subject line like 'Laser ranging at APO, March 2021', copy the email content from the .faa file but then remove the lines that are not for the correct month, and send email.

3.1.3 Monthly: TSIM test
Approximately once a month, test the TBAD plane transponder system using the TSIM aka "fake plane" in the dome.

With dome closed, point telescope at 20 deg alt. Lights on/off do not matter. Covers open/closed do not matter.

Take an extension cord into the pit in front of the telescope. There is usually an extension cord on the bottom shelf of the NICFPS cart or the bottom of the NA2 cabinet. Plug in the extension cord beneath the dome controls or beneath the phone, then run it over to the seam between the dome doors. At the bottom of the left dome door is a gray box with an amber warning light and a very short power cord. Plug this into the extension cord. The amber light should come on. This will not work with the dome open.

Open up the gray box (be careful, the latch likes to pinch fingers!) and look inside. On the top right side there is a toggle switch, which should be On, and a green LED that says "transmit" which should be flashing. It may flash quickly or slowly; either is normal. Confirm the switch and the flashing LED, then close the box again.

On a computer in the APO network:

$ ssh observer@cocoa

$ cd /home/apollo/xponder

$ touch bypass-dome
If this file exists in the directory, the TBAD system will turn on with the dome closed. Normally TBAD turns off when the dome is closed.

Wait at least 90s with the box plugged in AND the bypass-dome file existing, for a full test cycle.

$ rm bypass-dome

Unplug the gray box and return the extension cord to its proper place.

The TBAD output is stored in /home/apollo/xponder/log/ under a file named by the UT date. Optionally, you can check this file to see if the system is behaving normally. It should look something like this:
2024-04-16 03:29:57.412 s4243..BHFX.F2  ----- TSIM     104.00  20.00 B
2024-04-16 03:30:05.412 s4143..BHFX.F1  ----- TSIM     104.00  20.00 B
2024-04-16 03:30:07.295 o8888...H...AB  ----- OPEN     104.00  20.00 B
2024-04-16 03:30:08.007 o4141..BHFX.EB  ----- TSIM     104.00  20.00 B
2024-04-16 03:30:08.162 o4143..BHFX.ED  ----- TSIM     104.00  20.00 B
2024-04-16 03:30:08.318 o4145..BHFX.EF  ----- TSIM     104.00  20.00 B
2024-04-16 03:30:08.474 o4147..BHFX.F1  ----- TSIM     104.00  20.00 B
2024-04-16 03:30:08.630 o4151..BHFX.EC  ----- TSIM     104.00  20.00 B
2024-04-16 03:30:08.786 o4153..BHFX.EE  ----- TSIM     104.00  20.00 B
2024-04-16 03:30:08.942 s4155..BHFX.F4  ----- TSIM     104.00  20.00 B
2024-04-16 03:30:09.098 s4157..BHFX.F6  ----- TSIM     104.00  20.00 B
2024-04-16 03:30:09.254 s4161..BHFX.F1  ----- TSIM     104.00  20.00 B
2024-04-16 03:30:09.410 s4163.DBHFX.09  ----- TSIM     104.00  20.00 B
  
The first few columns after the timestamp indicate the "fake plane" changing the strength of its signal as it goes through the test cycle. You want to make sure that almost every line says TSIM, as shown here, and that there are no clusters with many OPEN lines in a row. A few intermittent OPEN lines are a normal part of the test. If you want, you can look at this log file sometime when the dome is open (APOLLO does not need to be in use), and you should see some real plane detections with transponder and location information. Remember that TBAD will not detect all planes, but only the ones near where the telescope is pointing.
2024-04-16 05:02:38.403 o3670...HF.FCB  -----          35.90  50.43 O
2024-04-16 05:02:38.414 o8DADB35C58AF81E3AA99E96729CD...HF..77 DF-17 35.90  50.43 O
   DF-17: PPass, ID ADB35C, Alt 34000, Lat=32.83397, Lon=-105.83564
2024-04-16 05:02:38.414 o1524...HF.FC7  34000          35.90  50.43 O
2024-04-16 05:02:38.436 o3670...HF.FCB  -----          35.90  50.43 O
2024-04-16 05:02:38.442 o8DADB35CEA428864015C080AF03F...HF..38 DF-17 35.90  50.43 O
   DF-17: PPass, ID ADB35C, MCP alt: 34016; Hdg: 0; 213.6 mbar; TCAS
2024-04-16 05:02:38.470 o3670...HF.FCB  -----          35.90  50.43 O
  
    
3.2 Preparation and Warmup
3.2.1 Few days before ranging: Request NOTAM
A NOTAM is a "NOTice to AirMen" which warns pilots to stay away from a particular area of sky. Our Letter of Determination requires us to issue a NOTAM for each ranging session. FAA has approved APO as a "station" to issue NOTAMs, but it can only be done from a browser running Microsoft Edge in Internet Explorer 7 mode, which means it has to be on a machine running a particular version of Windows. We have a virtual machine set up to do this, but communicating with the machine requires either being onsite or connected to APO by VPN.

A few hours up to a week before a ranging session (if the weather is anything better than hopeless), request a NOTAM by this method:

Use Safari or a VNC tool to connect to vnc://10.75.1.216 (first-guess password for screen sharing)

If you need to login to the machine, the password is on the list.

There should be a Microsoft Edge browser already open on the desktop; if not, open it.

The browser should already be connected to the NOTAM interface; if not,

In the top-right of the browser window, click on the "..." menu icon

Near the bottom of the menu, choose Internet Explorer Mode

Send the browser to https://nes.notams.faa.gov/nesv3/index.html

Tell the popup window to accept the certificate.

For login, the station name is APO-LASER and the user is RM. Password should be automatically remembered, also on the typed list, but it may change quarterly

Pro tip: if you login, and then as soon as you try to do something it asks you to login again, it probably means you are not in Internet Explorer mode!

From the menu on the left of the browser window, select Draft Notam. You should get a big text entry box.

Click on the Apply Template button above right of the entry box, which fills in almost all the info, then edit the date and time:

Go to the bottom of the text where it says YYMMDDHHMM-YYMMDDHHMM

Convert the schedule for the upcoming APOLLO run to UT date and time, with some padding at the start and end.

Enter the padded start/end datestamp in place of YYMMDDHHMM-YYMMDDHHMM

Below the text entry box, click the Preview button

Make sure there are no red error messages that need to be addressed. It is normal for ^^ to appear under the date.

Press the Return to Draft button.

Now press the Submit to USNOF button and say yes, you really want to.

A list will appear showing the work number; note the work number.

A few minutes to an hour later, click on NOTAM List on the left-side menu to make sure it was approved.

Example NOTAM request:
ZAB NM..AIRSPACE SUNSPOT, NM..LASER RESEARCH WI AN AREA 
DEFINED AS 5NM RADIUS OF 324649N1054913W (BWS098010) SFC-FL600. 
APACHE POINT OBSERVATORY. AT A TYPICAL ANGLE OF 45DEG, 
FM THE SFC, PROJECTING UP TO FL600 AVOID AIRBORNE HAZARD BY 5NM. 
THIS BEAM IS INJURIOUS TO PILOT'S/AIRCREW'S AND PAX EYES. 
ALBUQUERQUE /ZAB/ ARTCC TEL 505-856-4500 IS THE FAA CDN FACILITY.
2303250110- 2303250230
3.2.2 Hour before ranging: Set up observing log

Every APOLLO ranging session since 2005 has been recorded on paper logs. Maybe someday this will be modernized, but that day is not today.

Get a blank log form or print one out from here.[It's a PDF; link works in Chrome and Safari but not in Firefox?]

Go to the Pointer tab of ATUI (you do not need to initialize anything else inside ATUI, or even connect to the hub).

Press the Now button below the Moon graphic to make sure the ephemeris information is current.

Record the Phase, Libration (X,Y and total) and reflectors lit (1) or unlit (0) on the top line of the page.

Record the UT date (use the after-midnight date for winter sunset runs that start before 0 UT), APOLLO observer, Telescope obs-spec, and sky conditions on the second line.

Dates and notes for various warmup procedures will be written on lines of the log without regards to the column labels; when you start ranging, then notes for each individual run will be written according to the column labels.

Here is an example observing log from a successful run as a PDF or as an image:

3.2.3 Hour before ranging: Prepare ATUI and other windows

Many windows are needed to operate APOLLO:

A terminal window for opening ATUI and plots

Another terminal window for ssh to houston and cocoa

A browser window for viewing the instrument status and temperatures

Another browser window for viewing the guider video, which needs to be in a fixed position once the crosshairs are set.

A VNC window for controlling the acs-laser laptop

Regular TUI with, at minimum, Status and Chat and Slew and Sky windows

ATUI which is a whole bunch of windows combined into one

The ROOT Plots window, which may refresh itself or may spawn a new window with every run.

So, you'll want to get started early to set these windows up!

python3! Here are some instructions for starting the python3 version of ATUI through a docker container. Hopefully some of this can be wrapped/packaged/made easier, but as of April 26, 2023, here's what you have to do:

Make sure you already have ATUI-via-docker installed in your account on obsA or whatever computer you are using!

Open XQuartz (hopefully on dock, else look in Applications/Utilities)

Open Docker (hopefully on dock, else look in Applications). Docker desktop will launch.

Open a terminal window. In that window, enter:

xhost +

docker start atui3 [This will launch TUI with the ATUI window. Possibly you can also do this with the play-triangle button in the Docker desktop window.]

docker exec -it atui3 /bin/bash [This gives you a new command prompt.]

cd /workdir/python/TUIAdditions/Apollo

Now you can [optionally do some other things and then] go the ATUI ROOT tab and press the Connect to ROOT button. After pressing that button, you can't do anything else with TUI or ATUI until you open the plots window:

Back in your terminal window: python3 Apollo_Plots_ROOT.py twm2.json

Open ATUI and connect. Start in the Main/Active tab.

At top right of window, press Take Control
In middle of window, press StartNubs then Listen then ConnDev then Get Status

Wait 15s for Get Status to finish.

If you see an error (in the small log excerpt, or in the main TUI log window, or in the Alarms tab) saying "NoneType object has no attribute 'call'" that occurs once or a few times after the StartNubs, but stops recurring by the time you get to Get Status, then you're okay. If that error message continues to show up or you see another message "the target named apollo is not connected," then you may need to restart the ICC and/or housctl. Go do that, then come back to ATUI and try the four middle buttons again.

Note: if someone else already did the initialization using the four middle buttons, or if you disconnect from ATUI and come back a little while later (without restarting anything), you may only need to repeat the Listen and Get Status buttons. If you come back many hours later or if you restarted any software, repeat all four of the initialization buttons.

The status indicators along the left side should indicate that the system is IDLE.

Press the warmup button below the IDLE indicator. Note: In summer, you might want to delay this step until you are ready to do the ACS checkout!

Record the time of warmup in your observing log.

Wait 30s for all the warmup activities to finish, then switch to the Alarms tab to see if there is anything new. If all seems well, you can press the Acknowledge Alarms button to get the tab name back at least for a while.

If you have not done it already, open the plots window. Start by switching to the ROOT tab.

Press the Connect to ROOT button.

You may get a rainbow pinwheel while your cursor is in the ATUI window, until you...

Go to a Terminal window and [Russet laptops: 'aplots'; see instructions above for python3 machines] to open the APOLLO Plots Window.

You are now ready to proceed with other checks and warmup procedures.

3.2.4 Hour before ranging: check GPS clock
The GPS clock provides a reference for the more-precise Cs clock, so we prefer to have both clocks working. The GPS clock gets periodically jammed by exercises at nearby military installations; sometimes it manages to unjam itself, and sometimes not. So it should be checked and if necessary woken up more than half an hour before ranging begins (because it needs some time to settle after unjamming.

The simplest way to check the clock is to go to the webpages accessible through the Plots webpage and scroll down to the "GPS Clock Properties" near the bottom of the page. Make sure the timestamp at the right of the graph is the present time. If it is not, refresh the page (you may need to hold the shift key and press refresh) and make sure the blue line on the graph is not flatlined. If the graph is flat, you need to wake up the clock.

The clock can be woken up either through ATUI (preferred method), or through direct input to housctl.

To wake up the clock through ATUI, go to the Power tab:

Switch 24 (labeled GPS_clock) should have a checkmark

Uncheck switch 24 (Bug tip! Checkbox not working? You can instead do "power 24 0" in the command line of the Main/Active tab)

Wait one minute, then recheck switch 24 (or "power 24 1" in the command line)

Wait one minute, then refresh the Tables webpage; it should show numbers for gpstime, phase, and offset

Wait one minute, then update the Tables webpage; it should show new numbers for gpstime, phase, and offset

Go do something else for a while, then after half an hour check either Plots or Tables to make sure the clock is continuing to settle.

Here's what the plots for the clock look like when the clock has been stuck for a few hours (left), and then what it looks like a few minutes after being reset (right). Because of the huge jump in scale you can't see any details on the "after" plot, which means you have to look at the numbers on the tables webpage instead to see if small changes occur with each update.

As a backup method, you can use direct access to housctl with a login to observer@houston. This is not the best way to reset the clock but this access can be used for other things, so the method is described here:
$ telnet houston 5320
This gives you a direct interface to housctl so you can see log output and send commands; only one person can do this at a time, and be sure to disconnect when you are done!
Inside the telnet interface, watch the log output for lines every 10s that look like
    gps0 gpstime="F03  UTC  04/13/23 08:33:12"; gpsdac="F71 phase= 1.377E-08 s  offset= 3.490E-12  drift= 4.783E-12/DAY  DAC= -5439"
Are the phase and offset changing with at least every other report / every 20s? If so, the clock is fine. If not, you can try resetting it with a warmstart (more gentle) or with a power cycle (more likely to work).
For the warmstart:

Enter 'gps warmstart' directly into the telnet interface

Watch the clock output for the next minute or two to see if the phase and offset change and keep changing. If not, try the power cycle instead.

For the power cycle:

Enter 'power 24 0' directly into the telnet interface

Wait 30s

Enter 'power 24 1'

Watch the clock output for the next minute or two to see if the phase and offset change and keep changing.

Exit the telnet interface with ctrl-]

Note: You can also use the 'gps warmstart' method with the command-line interface in ATUI! But it's awkward because the window with the log murmur is very small and doesn't always show everything, so you might have to look at the webpage for clock phase and offset instead, and the webpage only updates once a minute. But it is an option.

If none of these methods works to unstick the clock, that may mean active jamming is still going on. Check back in half an hour and try again. You can try ranging even if the clock is stuck, but you may find no signal or find the signal at higher predskew than expected, or find that the predskew keeps changing. Any of these symptoms may mean that the ranging results are unreliable.
3.2.5 Hour before ranging: generate polynomial files
The polynomial files report the estimated distance of the Moon using a moderately complicated lunar orbit model. This is important since we only check a short window for the laser light returning from the Moon (about 40 cm of distance), in order to then refine the distance to 1 cm or less. So we have to start out with a pretty-good prediction.

The polynomial files make use of earth orientation data and predictions, so the files should be built no more than one day in advance of ranging, using the latest earth orientation measurements. The name of the polynomial file in use will be shown in the Active and Pointer tabs of ATUI, but it may be colored red or say 'None' until you slew to the position of one of the reflectors on the Moon. Once you slew to a reflector, if it still says 'None' or is red, that probably means the correct files are not in the correct place.

Login to observer@houston
$ cd /home/apollo/ephem/predict

This method uses no editor, but still needs cut-and-paste:

copy full contents from USNO website

$ cat >gpsrapid.daily

paste text

newline

ctrl-C to exit

OR emacs -nw gpsrapid.daily [if not automated; true from April 2022]

delete old contents [ctrl-@ at top, esc-> to go to bottom, then ctrl-w]

paste in full contents from USNO website

make sure file ends with newline

save file; exit editor [if emacs, ctrl-x ctrl-s then ctrl-x ctrl-c]

make a directory with UT date name, such as $ mkdir 2023-03-25

cd to new directory

make polynomials with a start time at least a few hours before ranging. Example commands:

$ ../bin/mkpoly 2023 03 25 00 00 0.0 [if session starts well after 0 UT]

OR $ ../bin/mkpoly 2023 03 24 20 00 0.0 [if session starts near 0 UT]

OR $ ../bin/mkpoly 2023 03 24 20 00 0.0 10.0 [if Moon is below 20 deg altitude for part of scheduled time]
output should end with lines like these:
 
Count for reflector [0, 1, 2, 3, 4, ctr] = [95, 95, 95, 95, 95, 95]
Congratulations, Fits created successfully
numbers in brackets should be in the range 40-120
If lower than 40, rerun with earlier start time or lower altitude limit

If higher than 120, rerun with later start time

Pro tip: If it claims moon is never above altitude limit when you're sure it is, make sure gpsrapid.daily file does not have spurious lines or characters! Delete files from the date-specific directory, then try again.

copy polynomials to daily directory:

$ cd /home/apollo/daily

$ rm PolysRefl*

$ cp ../ephem/predict/2023-03-25/PolysRefl* .
3.2.6 Hour before ranging: check space block files
Space Command, also known as "Laser Clearinghouse," is supposed to tell us when we can range without hitting a satellite. APOLLO is not capable of causing harm to satellite structure, but some satellites might have sensitive cameras looking down at the Earth that don't want to see a bright laser.

The normal procedure is that we send Space Command a schedule of when we plan to range; we will basically always be pointing at the Moon when we send the laser into the sky, so that's what is listed in the files we send them. These files are called "Predictive Request Messages," or PRMs. In response, Space Command creates "Predictive Avoidance Messages" or PAMs, which indicate blocks of time where the laser is and is not allowed to shine the laser at the Moon. Usually the block periods will be short (seconds to a couple of minutes), except sometimes when the Moon is near -5 degrees declination, there might be a slow-moving geosynchronous satellite passing near the Moon, so there might be multiple blocks of many minutes.

Since approximately 2019, Space Command has not been generating PAMs in response to the PRMs that we send them. Nevertheless, we should check each time to see if there are any PAMs available. To do this, first login to Space-Track.org [Note: we need to create a group account for this...]. From the top menu bar, choose Files - Download. It will show you a directory tree ending in the directory zLCH; click on the > to open that directory and wait a few seconds. Click on the > for Apache Point and wait a few seconds. Click on the > for PAM and wait a few seconds.

Does it say "-- Folder Empty"? Then go to ATUI Main/Active tab and in the command line at the center of the window, input 'unblock 90' to allow ranging for the next 90 minutes, or else adjust the time to the approximate end of the scheduled APOLLO session.

Does it have a file or files in it? Look for one with the current UT date and the word "moon" in the filename; click on that. Copy the portion of the file which looks something like this:
YYYY MMM dd (DDD) HHMM SS    YYYY MMM dd (DDD) HHMM SS      MM:SS
-------------------------    -------------------------    -------
2023 Mar 07 (067) 0651 00    2023 Mar 07 (067) 0658 13    0007:13
2023 Mar 07 (067) 0658 24    2023 Mar 07 (067) 0659 09    0000:45
2023 Mar 07 (067) 0659 13    2023 Mar 07 (067) 0710 16    0011:03
2023 Mar 07 (067) 0710 20    2023 Mar 07 (067) 0727 28    0017:08
2023 Mar 07 (067) 0727 31    2023 Mar 07 (067) 0829 00    0061:29
  
Login to houston and update the block file:

$ ssh observer@houston

$ cd /home/apollo/daily

Easy method with no editor:
$ cat >housctl.blk
paste in what you copied from the webpage
press return to end with a newline
ctrl-C to exit

OR use an editor:
$ emacs -nw housctl.blk

Delete the old info from the file

paste in what you copied from the webpage.

Save and exit the file [ctrl-x ctrl-s then ctrl-x ctrl-c]

In ATUI, press the "readblock" button on the left side to read in the file you just updated

Now if you look at the Alarms tab, it should either say "Time remaining in release" in green or it should have a fairly short "Time remaining in blockage" indicating that you will be allowed to range at the scheduled time. Additionally, depending on ATUI version, the color or visibility of the tab for Main/Active may depend on whether or not there is a space block in place.
3.2.7 Half hour before ranging: check out ACS

The ACS or "Absolute Calibration sytem" is a second laser with lower power and a much faster pulse frequency (80 MHz?) than the main laser. When the ACS laser is properly in phase with the Cesium clock, its fast pulses can be fed into the data stream for the lunar returns to provide a "comb" for calibrating the timing to very high precision. A lot of the checkout steps have to do with tuning the ACS pulses to match the clock, for instance by changing the temperature of a fiber-optic coil in order to change the path length. But for a lot of these processes Russet doesn't really understand what's going on, so these instructions are along the lines of "turn this knob until A lines up with B, and then you're good!"

Note: If it's summertime and you chose to delay pressing the WARMUP button, now is when you want to do that. You can check the description in section 3.2.3 for how WARMUP should proceed.

Start in a terminal window:

$ ssh observer@cocoa

$ cd /home/apollo/ACS

$ ./ip-pwr.py
This shows the status of 5 switches; all should be on except #2

$ ./ip-pwr.py pset 2 1
Next, go to the ACS tab of ATUI and check the box for PicoFYb DC Power at top right.
Open a VNC viewer (Safari browser is recommended, VNC Viewer may not work).

Connect to vnc://acs-laser.apo.nmsu.edu:5959

It will ask for a password.

Once connected, cancel and/or close any windows begging for updates. Do not press Okay, that will start an unwanted update!

Double-click the big red button icon to open TOPAS FemtoFiber smart

Press Menu at top left, choose Connect

Press the "Hardware Disable" button at left

In the Laser Current box at right, change "I set" to 356 mA and press Return; the small fake LED just above the entry box should turn on green.

Press the "Emission" button at left

The "Laser on" fake LED at the top should turn yellow, and the fake warning sign illuminates

Double-click the yellow padlock icon on the desktop to open PicoFYb-Phaselock. You may need to wait 10-15s for the program to connect before proceeding.

Look at the number in the "difference frequency" box at top right.

If the number is over 1000 and changing rapidly, go away for a few minutes and check back after it settles down.

If the number is a few hundred up to about 1000 and steady or changing slowly, proceed...

Select the "TEC temperature" tab at top right. Initial setpoint will be 25.85 C.

Change the setpoint and press return.

In summer, make the setpoint colder. In winter, make it warmer.

For a difference frequency of 600, change temp by 1 C, less for a smaller difference and more for larger.

The TEC may initially go in the wrong direction but should eventually figure out heating vs cooling

Select the "phase mismatch" tab and watch the difference frequency head towards zero. When it gets below 200 a line will appear on the plot next to the difference frequency.

If difference freqency increases instead of decreasing, change the TEC in the other direction.

If difference frequency levels off, change TEC again OR use Forward/Backwards buttons to adjust by 0.01 C.

Once difference frequency gets to less than about 60, go to ATUI ACS tab and check PicoFYb lock

Back in the VNC window, the plot at top is now grayed out. Instead, watch the "piezo voltage" plot at middle right.

You want the voltage to be near the green dashed line, between the two gray dashed lines.

Use the Forward and Backwards buttons to bring the voltage in and make it steady

Once it's more or less flat, press the 'loop open' button to close the loop

Press the R next to the red 'overflow flipflop' button near top left of window

Check the temperature in the log murmur at bottom right; record this and the time in your observing log.

The auto loop can only correct slow changes, so check back again every few minutes to make sure the voltage is staying between the lines, or bring it back if needed. You may need to re-close the loop and press 'flipflop OK' again after you bring it back manually.

After a while it should settle down and stop wandering and you can check less often, but don't forget it completely.

Leave the VNC window open with the Phaselock program on top so you can check it at a glance.

Go back to the ACS tab of ATUI.

Press "Set delays to nominal values"

They should be 0 55 71 123 ; if not, set them by hand and press return. [Last changed April 2022]

Just below the delays, set the [DAC Val] to 0 and press return

Press the "ADC Read" button. The second value should be not-too-high, less than 400

After a minute, press the "ADC Read" button again. It should be pretty close to the previous values; if not, wait a few minutes for it to settle before proceeding.

Press the "DAC Sweep" button. Watch the red and green sinusoids grow across the graph in the bottom of the window. Note the time of DAC sweep with the maximum red and green values in your observing log. If the DAC sweep is flat, you have a problem. Try the "AD5592 Recover" button or call for help.

Press the "Phase sweep" button. Watch the weird shapes grow across the graph. Try to mentally remove the square bumps, and the underlying shape should be flat-hump-flat-hump-flat-hump (three cycles). The graph should start on the flat portion shortly before the hump; if it doesn't, note the X value of that desirable position.
Here are two examples of the Phase sweep plot: on the left with older values for the delays, so that only narrow blips are interrupting the underlying shape, and on the right with more recent delay settings (still not the same as current) so there is a broader square wave almost hiding the underlying shape. Both examples are close to the desirable position and don't need an offset.

Enter the X value of the desirable position into the Phase bump box, including a + sign if it's a positive value.

Press return after entering the value, BUT this is one of the boxes with a bug where it remains pink. Watch the fake LED at the top of the window to see when the comman is done, or go to the Main/Active tab and look at the last command that went through to make sure it was the offset you wanted.

After the LSB Status LED turns green, press "Phase sweep" again. Hopefully it now starts on the flat portion shortly before the hump. If not, adjust again.

Once the phase sweep is adjusted, press the "Extremize bias" button and it should make a nice parabola and set the [DAC Val] number to the minimum.

Change the step size from the initial 200 value to 100, and repeat the "Extremize bias" step. The parabola is smaller now.

Finish ACS checkout by setting the attenuation to maximum with 'acs atten 60' in the Main/Active tab command line OR by entering 60 in the Attenuator [dB] box of the ACS tab.

There can be some variations in how the phase sweep looks. Sometimes the phase sweep will show an upward slope (never downward?). This may mean that the temperature hasn't quite settled on the fiber optic. Check the PhaseLock program, adjust temperature if needed, then try the phase sweep again after a few minutes and the slope should be shallower or perhaps gone. This is not believed to be a serious problem. You should be able to adjust the phase bump even when a slope is present.

Hopefully all these steps went smoothly. If they didn't, you might need to call for help or else skip using ACS for this session and try to get someone to work on it later. If you're ready to give up on ACS, you can go through the shutdown steps for ACS only in the Finish and cooldown section, but proceed with checkout and warmup of the main laser system. Turning off ACS if you're not going to use it will reduce the heat load inside Utah, which is desirable in warmer weather.

During a ranging run with ACS enabled, you can look over at some of the plots on the right side of the Plots window to see what ACS is doing. The top-right plot in particular, "Lunar Raw TDC," should show the desired reference comb with 11 teeth. This example shows a closed-dome run, so there are fids but no lunar returns.

3.2.8 15 minutes before ranging: Check APD

The Avalanche Photodiode, or APD, is the primary science detector for APOLLO. It is 15 pixels laid out in a 4x4 grid, with the top left pixel in the grid used for timing rather than light detection. There are gaps between the individual photodiodes, but each has a lenslet above it to focus light from a larger area and ensure no photons fall through without being detected. The lenslets are about a quarter arcsecond on-sky, meaning the size of the detector is just over 1 arcsec side-to-side, or about 1.5 arcsec diagonally.

Look in the middle of the ATUI Main/Active tab, just to the left of the StartNubs button. Make sure NStares = 5; Stare rate = 1000; Stare binning = 500; # of Darks = 10000.

On the left side of ATUI, press the Stare button.

Watch the Hitgrid and Stare Rate plots in the Plot window (top left and top middle) for ~5s.

If the x-axis of the Stare Rate plot counts up to 5 but the line is flat at zero and the hitgrid doesn't change, press the caltdc button and wait a few seconds for status to go back to STANDBY. [Note: if you already did ACS checkout, this step may be unneeded.]

Press the Stare button again, and now you should see positive counts on the Stare Rate plot.

Make sure BC1 eyelid is closed, and press the Dark button.

Wait ~30s until the hitgrid updates and the status goes back to STANDBY.

Press the Stare button again, and now you should see the counts on the Stare Rate plot average zero plus or minus some random noise, because an average dark is being subtracted from the rate.

Record the time of Stare/caltdc/stare/dark/stare in your observing log.

Now the APD is ready to go.

3.2.9 15 minutes before ranging: Set STV crosshairs

The APOLLO guider is a Santa Barbara Instrument Group (SBIG) camera known primarily as the STV, also called "CCD" or "guider" in some places. It has a physical control box and image display screen inside the observing-level cabinet, but the camera is nearly always used in remote instead of local mode. A graphical display in the STV tab of ATUI replicates the appearance and functionality of the physical control box.

Start by opening a browser window to video-35m. From the quad view, select video 4 at the bottom right. This is the feed from the APOLLO STV. If the state is WARMUP or STANDBY, the STV should already be on and probably already in Focus mode, which is the mode we usually operate in. You can tell it's in Focus mode from the square box moving across the bottom of the screen from left to right, showing the progression of frames. Set this browser window with the STV video feed somewhere visible off to the side of ATUI.

In the STV tab of ATUI, click on the white box with red text in the upper-right quadrant of the control interface. This is the display screen for control info of the STV. Sometimes the display does not update with the latest information, so clicking on the display will force an update. If the state is WARMUP or STANDBY and the video is running in its default startup state, updating the display should cause it to say
FOCUS
Norm 25ms, 1x Full

This means it's in Focus mode with 25ms exposure time and 1x gain.

Optional practice for getting used to the controls: Move your cursor below the display screen to the knobs which have pasted-on boxes saying "20" and a pair of arrows next to each knob. Click the up-arrow, and you should see the 25ms in the display screen change to 50ms. You can click on the display screen if desired to make certain it's up to date. 50ms or .10s (another up-arrow) are reasonable exposure times for a likely focus star.

To bring up crosshairs, click once on the square black button labeled "Display/Crosshairs" and then on the square black button labeled "Value." Now the video display in the browser should become static, showing crosshairs centered on the upper-left part of the image.

Holding down the control key on your keyboard, click on the right-arrow. This will move the crosshairs to the right by 20 steps (or whatever is in the white box pasted over the knob). Now the display screen should say something like DISPLAY (X,Y)=100,51. You can click on the screen to make sure the numbers are up to date. Use ctrl-right-arrow to make large adjustments until the X number is near 160 or so, then release the control key and use the right and left arrows to make single-pixel adjustments until the X position is 159 (remember to update the display, especially after changing directions!). Then use ctrl-up-arrow and single-pixel up/down adjustments to move the Y position to 72 (note the up-arrow makes the crosshairs move down on the image for higher Y numbers.)

Pro tip! Did you press Display/Crosshairs more than once? Did you press the Parameter button instead of the Value button? Is there now a weird menu showing up in the red & white display? Try pressing the black Focus button twice to go back to Focus mode. Once you see the little white square moving across the bottom of the image, try again. If you get into a really weird mode like a zoomed-in image, you can go to the Power Tab of ATUI and uncheck the STV box, wait a few seconds, then re-check the STV box to power cycle the camera. When the image shows a SBIG start screen, then press the black Focus button twice to get back to Focus mode.

Measured in December 2019 and periodically confirmed since then, (159, 72) is the sweet spot on the STV where light passes through to the APD science detector. The STV is looking through a filter close to the laser wavelength but not exactly on it, so the star does not disappear from the STV while it is on the sweet spot. The X,Y position for the sweet spot is stable unless the STV camera or the "snout" that feeds light to it gets disturbed, in which case you should update the sweet spot position using the method described in Troubleshooting and Maintenance.

Put a piece of tape over the crosshairs position in the browser video window, and mark it with a sharpie. Take care not to move that browser window again after marking it!

Put the guider back in Focus mode by pressing the square black button labeled Focus twice. Confirm that the white square is moving across the bottom of your video window. Now the STV is ready to go.

3.2.10 5 minutes before ranging: warm up main laser

The main laser needs to "flash" in order to warm up before ranging. If not warmed up, the laser may have unusual numbers for threshold voltage and output power, which could lead you to run the laser with the wrong voltage. In warm weather you might want to shorten the warmup time to 5 minutes in order to reduce the heat load inside Utah, but from October to April you probably want more like 10 minutes of warmup.

In ATUI, go to the Laser Tuning tab.

Press the Laser Powerup button.

Watch the display on the Continuum interface until it shows a shot count.

Record the shot count in your observing log, in the box near the upper right corner.

Press the Laser Warmup button right beneath the Powerup button.

Watch the Continuum interface until the first two red and green LEDs are both lit and circled in white to indicate they are flashing.

Record the time of "powerup" and "flashing" in a row of your observing log.
3.3 Ranging

3.3.1 Pointing and focus check on a star

Load the APOLLO catalog into your TUI Slew window. Check the ATUI Pointer tab (press the Now button to refresh the numbers) for the Moon's azimuth and altitude. The azimuth listed here is aviation azimuth, so you have to convert it to TUI azimuth, but a rough conversion is sufficient. (Hint: alternatively, look at Omea or IRSC for the general position of the Moon!) Choose a catalog star near the position of the Moon. When the observing specialist gives you slew permission, the first target you will slew to is this pointing star.

Standard APOLLO procedure is to use Boresight offsets for coarse initial centering, and then Guide type offsets for smaller tracking adjustments while ranging. This separation makes it possible to watch for tracking imperfections and patterns which might otherwise go unnoticed. However, in practice either Boresight or Guide offsets can be used for either purpose. DO NOT use ObjArc offsets with APOLLO, because those offsets are used to follow the Moon's nonsidereal velocity, so changing the ObjArc offsets may mess up the lunar pointing very badly.

In the STV tab of ATUI, first make sure it's in Focus mode by clicking on the white display window, or check the guide video for the little white square moving across the bottom of the screen. Then use the up/down arrows to adjust the exposure time on the STV so that you can see the star clearly but it isn't saturated. Note that the maximum exposure time in Focus mode is .25s, but if you keep pressing the up-arrow the gain will increase to make the star look brigher, until you get to .25s, 16x which is the maximum sensitivity setting. If you can't see a third-magnitude star at this setting, the clouds are too thick and you will certainly not be able to range to the Moon. Or maybe the pointing is off or something is obscuring the light path.

Adjust telescope focus to make the star look round and not-lumpy on the video image. Then go to the Main/Active tab of TUI and the "Offsets" section at top right of the window. Check the box for Boresight Offset and set the Offset Increment to a number between 1.0 to 10.0. Make sure the CCD frame is checked at the right side. Now clicking on the direction buttons will pull the star in the indicated direction on the STV image by the Offset Increment amount. Use the direction buttons to center the star and also to get a feel for how big is 1 arcsecond, so that you can do a rough visual estimate of the seeing. Since you are looking at green light, the seeing will rarely be better than 1 arcsec. 2 arcsec is bad enough to make ranging difficult, and 3 arcsec might make ranging impossible. Report the seeing value to the observing specialist so they can record it.

Note: There is a script for python3 machines which will do a quantitative measurement of the star size. This script is almost ready to go in spring 2023, but still a little slow and clunky, so it's probably more efficient to estimate the seeing by eye. But watch for more developments here!

Note the name of the pointing star in the observing log, the accumulated Boresight offset, and the focus value. Put your seeing estimate in the Comments column. Now you're ready to slew to the Moon.

3.3.2 Pointing check on a crater

In the Pointer tab of ATUI, press Show Craters to mark the positions of some smallish reference craters on the Moon. If Apollo 15 is in sunlight, you will usually start with "Nearby Knob" which is a small mountain rather than a crater, but fits in the same pointing-reference category. Nearby Knob is a small isolated bright spot with no misleading shadows and very useful for centering on... when it's in sunlight.

If the Moon is a narrow waxing crescent, you will probably have to use Webb way over on the west side, but if the Messier craters are illuminated, then Messier B is your best choice (small with accurate position). Craters Peirce and Luther have less-reliable positions, so only use those if you also cross-check against another crater. For a waning moon, Fra Mauro B and Milichius A have reliable positions; for a narrow waning crescent, try to cross-check Damoiseau E and Lohrmann A against each other if possible.

If the Moon is a narrow waxing crescent, use Webb, pointing just a little below center:

For a wider waxing crescent, use Webb first then center on Messier B:

During mid-lunation while it's illuminated, Nearby Knob is the best choice:

For a wider waning crescent, use either Fra Mauro B or Milichius A:

For a narrow waning crescent, use both Damoiseau E and Lohrmann A, pointing a little low on both:

Press Slew Telescope in the Pointer tab to move to the position of your selected crater (or mountain) on the Moon. Slews made from the Pointer tab will keep both Boresight and Guide offsets. You may find that the crater position disagrees with the pointer star by up to 10 arcsec; this is mostly because the proper motion of very bright stars is not always well measured. You can do a coarse centering at this time, but don't refine it too carefully just yet, because you have some other things to do first.

IMPORTANT!!! After going to the Moon, be sure to press the Go To Target button in the Offsets section of the Main/Active tab. After you press that button, down in the bottom portion of the Main/Active tab the numbers in the "Current rxx" and "Current rxy" boxes should now agree with the numbers right next door under the "Target Values" label. If you try to do ranging with the Current and Target values disagreeing by 0.2 or more, you will lose signal photons. If you try to do ranging with a disagreement of 0.5 or more, you may never find the signal. If you center on a crater with the Current values disagreeing with the Target values, and then later you remember and press Go To Target, it will shift your pointing. So press Go To Target while you are still at the centering-on-a-crater phase in order to avoid confusion and possible future trouble.

Another important note: Please be careful not to press the very-nearby Commit button instead of the Go To Target button. The Commit button will force Current and Target values to agree with each other not by moving the receiver optics to the Target position, but by re-defining the Current position. So if you Commit the wrong position, the optics position becomes untrue. Someone will need to go into the housctl logfiles and find out out which starting position got redefined to which ending position, then move the optics in the opposite direction by the same amount and Commit again. It's a nasty job, so don't do that. But if you do, tell the truth about it right away!

3.3.3 Laser threshold and power check

The next thing to do is check the laser threshold voltage and output power. Hopefully it has now been flashing for about 5-15 minutes (shorter in the summer, longer in the winter), so it's time to find out if the laser is happy.

In the Laser Tuning tab in ATUI:
Enter 2 or 3 in the box beneath the Measure Power button, and press return to change it from pink to white.
This is how many minutes the power measurement will last.

Press the Measure Power button

Wait until numbers and a plot line appear in the graph in the lower part of the window

Are the numbers exactly 0.0? The bolometer may be off. Press Stop button below Measure Power button, then go to the Troubleshooting section.

Are the numbers around 0 but with some noise? Move voltage up with the ^ button to the right of "Oscillator Voltage," but if you don't get measurable power within four button-presses, something may be wrong.

Are the numbers somewhere between 1 and 2 Watts? Proceed with threshold check:

Go to the buttons to the right of the "Oscillator Voltage" in the top part of the window, and press the v30 button. This moves the voltage down by 30V, but the number in the box to the left changes by a larger amount because the displayed number is ohms, not volts!

Note: After a restart of housctl, there may be a fake number (0 or 9999) shown in the box initially. Go ahead and change the voltage, and after that it will remember the actual value.

Did the power drop to near 0?

If not, use the v button to go down 5V at a time until the power drops.

Once the power drops, use the ^ button to go up 5V to the lowest voltage with non-zero power. This is the threshold.

Record the threshold value (in ohms, in the box to the right of "Oscillator Voltage") in your observing log

Now press the ^30 button to go up 30V from the threshold. This is the operating voltage.

Press the Clear Stripchart button to refresh the graph so you can see small changes.

In the Continuum interface, experiment with the clockwise-arrow and counterclockwise-arrow buttons to get maximum power.

These buttons are adjusting the second harmonic generator which changes the laser light from IR to green.

If the telescope altitude is higher than when it was last adjusted, the SHG probably wants to go clockwise (top button); if lower, it wants to go counterclockwise (second button).

Find the SHG setting that gives highest power and leave it there.

Record the 20-point average power (at top center of window) in your observing log, in Watts.

When the power measurement finishes (you can use the Stop button below Measure Power to end it early), the Continuum should say that it's in PGM2, which is what we want for ranging. Press the Shutter button in the grid of Continuum buttons, and watch the Shutter LED turn green; this opens the laser shutter that allows laser light into Utah.

Now go back to the Main/Active tab and refine the pointing on your reference crater until it's well centered. Ask the observing specialist to press the green button on the keyswitch box; this opens the interlock shutter that allows the laser light out to the telescope. Now you are ready for ranging.

3.3.4 First target: Apollo 15

Always start ranging with Apollo 15 because it's the biggest reflector with the strongest returns, and also (when it's in sunlight) has the easiest position to find. In the Pointer tab of ATUI, in the graphical interface, click on the red + marking the position of Apollo 15, then press Slew Telescope. Allowing for the possibility of different rotation and different shadows and whether or not you have an atmosphere in the way, the location of the landing site looks like this:

The second image above is from the APOLLO STV showing moderately good seeing around 1.4 arcsec, which is nearly perfect for ranging. If you have better seeing than this, you might want to consider defocusing the telescope 100 steps for your first ranging attempt, in order to make the laser beam larger on the Moon. Once you detect signal you should walk the focus back to best-focus, 50 steps at a time.

Pointing should be pretty good if your reference was Nearby Knob, but if you need to make an adjustment, you can do it from the Offsets section of the Main/Active tab of ATUI. Use Boresight offsets if you have not been ranging yet, or "Raster Scope" (Guide) offsets if you have already started ranging, but either way, if you are adjusting position by referring to the STV image, be sure that you are doing the offsets in the CCD frame.

If Apollo 15 is not in sunlight, you will just have to start from the position you measured on a reference crater, but you are expecting that you will have to start out with a search raster initially. If it's in sunlight with decent seeing conditions, you have a reasonable chance of acquiring signal right away. In preparation for a search raster, go to the Offsets section of the Main/Active tab, check the Raster Scope box, and set Offset Increment to 1.0 (maybe 1.25 in poor seeing conditions, or 0.75 in good seeing conditions).

Checklist before starting run:

Is laser shutter open? (Continuum interface in Laser Tuning tab)

Is interlock shutter open? (Green button on keyswitch box)

Is the system state in STANDBY? (Left side of ATUI)

Is the APD awake and ready to go? (Got near-zero but not flat-zero counts in last stare)

Is "# of shots" set to 5000-8000? (Lower section of Main/Active tab)

Is ACS off for first run? (LUN_EN and FID_EN boxes unchecked in ACS tab)?

Is ACS attenuation set high to begin with? ('acs atten 60' in command line of Main/Active tab)

Is ACS phase properly locked? ("piezo voltage" plot in VNC window)

You're ready!
Press the Run button on the left side of ATUI

Record the time, reflector, and # of shots in the first three columns of the observing log.

Your Plots window should refresh or maybe a new Plots window will open.

Wait 10s for TR mirror to spin up (plus an extra 5s if ACS is enabled)

In the Run statistics section of Main/Active tab, "Shots per second" should go up to 20 and "Fiducial Records" should begin counting up.

A spike should show up right away in the "Fiducial, subtracted" plot in the middle of the Plots window.

After 3 more seconds, occasional photons should start appearing in the "Lunar, subtracted" histogram and "Lunar Residual Scatter" stripchart plots on the left side of the Plots window.

If no clear signal shows up after about 10s, start doing a search raster with the Offsets in the Active window.

check Raster Scope

Native frame suggested, especially if no sunlight, but any frame can be used

Offset down; wait 10-20s after end of offset

Offset left and wait

Offset up and wait

Offset up and wait

Offset right and wait

Offset right and wait

Offset down and wait

Offset down and wait; should be near end of run now.

Optionally go back to Home for end of run.

Optionally increase # of shots (and press return!), and continue searching.

You got a signal spike!

First move the PredSkew slider at the bottom center of the Main/Active tab so the dashed lines in the plot bracket your signal

Record the Predskew in the Comments column of the observing log with "PS=###"

Look at the hitgrid in the Plots window; is the signal off center?

Check Current vs Target rx values in the bottom part of Main/Active tab, make sure they are close to each other

Check APD frame for your offsets!

Make Offset Increment smaller; 0.5 if you're barely at the edge of signal, 0.25 if you're close, 0.15 if it's almost right and you just want to tune it up.

Use arrow buttons to center signal on APD.

Now take a look at the "Registered Fiducial Photons" in the middle of the Run Statistics table. We would like this to be between about 1-1.5

If the fid rate is consistently higher than 1.5, change the "Target Diffuser Phase" in the bottom part of the Main/Active tab by subtracting 40 or 50, and press return.

If the fid rate is consistently lower than 1.0, change the "Target Diffuser Phase" in the bottom part of the Main/Active tab by adding 40 or 50, and press return.

Don't change the diffuser phase to less than 1100 or more than 1300 without consulting someone who understands it better.

You didn't get a signal, and now the run is over.

Record the sad run statistics before doing other stuff

Check Current vs Target rx values in the bottom part of Main/Active tab, make sure they are close to each other

Check STV (if in sunlight) or cloudcam to see if you are losing signal to clouds

Slew back to reference crater and adjust focus (or defocus if seeing is very good) and recenter. (Zero Guide offsets with Home button, then switch to Boresight offsets to re-center crater, then back to Raster Scope in preparation for next run)

On next run, try a larger Offset Increment, or spend longer on each position, or just hope for better luck.

You got a signal but then lost it when trying to center up!

Look at the "Lunar Return Rate" plot; it should show the Guide offset where the peak returns occurred.

Use Raster Scope and Native checkboxes, then offset with the direction arrows to adjust Guide offsets to be near where you got the peak returns.

Wait for more returns to come in, and then try again to center up, in the APD frame this time!

You got a signal, and didn't lose it, but it gets weaker when you try to move it to center, as if the APD is "hot" on one side.

Check Current vs Target rx values in the bottom portion of the Main/Active tab; are they close?

Press Go To Target to move rx optics to Target position.

Is it a single hot pixel? It might be malfunctioning. You can uncheck that pixel in the Channels tab of ATUI

If you're sure this is a real, persistent trend, adjust optics carefully!!!

Check Raster Optics

Offset Increment 0.25 at most

Press direction button to "pull" signal onto chip

Check Raster Scope while you watch to see if this works, so you don't move the optics accidentally.
Maybe do some more telescope offsets to center signal.

Not sure that it helped? Press Go To Target to move back to previous optics position.

Yes, it definitely helped? When you have it adjusted so you have good signal in the center of the chip, record the change in the RX Offset column of the Observing log and then press Commit Rxx, Rxy.

You got some signal, and now the run is over.

Record Yield in observing log (bottom of Run Statistics table).

Record Peak in observing log (from "Lunar Return Rate" plot)

Record Guide Offset in observing log

Add Comments to observing log if needed.

You're ready for the next run!

Somewhere around this time, maybe before your first run in summer or after your third run in winter, there will be an error message about Utah temperature exceeding 22 C. The error message looks scary, but we aren't really worried yet. Between runs, go to the Environment Tables webpage, be sure to refresh it, and look at the latest measurement of the Utah temp.

If it's 22 up to 25 C, keep ranging but check again between each run. If it exceeds 25 C, start shortening ranging runs to 3000 shots or leaving longer gaps in between runs. Consider ending your observations if you already got a circuit around the reflectors. You could also turn off ACS but keep going with the main laser. If Utah temp exceeds 27 C or if you start seeing laser dropouts where the the "Shots per second" in the Run Statistics table drops below 15 and doesn't come back up, it's time to end your observations. You might want to do a final power measurement before cooling down the laser, and talk to knowledgeable people about what could be done to improve cooling for the next observing session.

3.3.5 Repeat Apollo 15 with ACS

Running ACS will add an increased level of background noise. The minimum non-ACS signal level that will let you add on ACS and still be able to keep your signal centered is about 0.1 photons/shot in the "Lunar Return Rate" plot, or 0.2 if you feel less confident. So the signal level in your non-ACS runs will determine your decision whether to enable ACS. However, even if you are getting weaker signal than that, if you have any returns at all you still want at least ONE run with ACS which can then be used to calibrate other runs without ACS. Usually, you will enable ACS for a second run on Apollo 15 just because that's most convenient. If signal is weak now but you have some reason (the Moon is rising? The clouds are expected to clear?) to think that later runs will be stronger, you could postpone the one ACS run until later in the session. Only skip ACS if you decide to give up ranging without any usable signal at all.

You can start up a run with ACS enabled but attenuation high ('acs atten 60') and the signal will look like a non-ACS run so that you can get centered up. There will still be two differences: an ACS run has a longer delay at the beginning before laser light starts to come out, and an ACS run with high attenuation will not show "Peak" information in the Lunar Return Rate plot. (This is a bug - not a showstopper, but it's pretty annoying to have to keep the peak rate AND the position where you got that peak in your own mind.) Because of those differences, if you are sure you won't be using ACS on a particular reflector, it's best to go through the step of un-checking LUN_EN and FID_EN. If you think you might use ACS but you're not sure, start up a run with high attenuation, try to track the peak rate with your brain, and you can reduce the attenuation later to let the ACS comb come through. But we do need a minimum of 1200 ACS photons in each run for ACS to be useful, so make the decision whether or not to lower attention while you still have at least 3000 shots left in the run.

Checklist before starting first ACS run:

Is laser shutter open? (Continuum interface in Laser Tuning tab)

Is interlock shutter open? (Green button on keyswitch box)

Is the system state in STANDBY? (Left side of ATUI)

Is "# of shots" set to 5000? (Lower section of Main/Active tab)

Is ACS phase properly locked? ("piezo voltage" plot in VNC window)

Go to ACS tab and do Extremize Bias with a stepsize of 100. Record the DAC value in your observing log.

Check the LUN_EN box and wait for the LSB Status LED to change from yellow back to green.

Check the FID_EN box and wait for the LSB Status LED to change from yellow back to green.

Go to the active tab, and on the command line enter 'acs atten 60' to start out with high attenuation. Wait for command to complete.

Ready to start the run!

Press the Run button on the left side of ATUI

Record the time, reflector, # of shots (if you think you won't change them) in the first three columns of the observing log.

Record the DAC for the ACS over at the right side of the "Comments" column in the observing log. This indicates it is an ACS run.

Your Plots window should refresh or maybe a new Plots window will open.

Wait 15s for system to get up to speed

In the Run statistics section of Main/Active tab, "Shots per second" should go up to 20 and "Fiducial Records" should begin counting up.

A spike should show up right away in the "Fiducial, subtracted" plot in the middle of the Plots window.

After 3 more seconds, occasional photons should start appearing in the "Lunar, subtracted" histogram and "Lunar Residual Scatter" stripchart plots on the left side of the Plots window.

You were already centered on Apollo 15, so hopefully a spike will show up in the histogram right away.

Spend a short while getting the signal centered on the hitgrid with Raster Scope offsets in the APD frame.

Once signal is centered but before the run is up to 2000 shots, lower attenuation:
In the command line of the Main/Active tab of ATUI, enter 'acs atten 28' if your returns are strong

Or 'acs atten 32' if your returns are on the weak side, or somewhere in between

Record the shot count and the new attenuation value in the Comments column of the observing log.

ACS photons will raise the background in the histogram and stripchart.

On the hitgrid, ACS photons are more likely to show up in the center.

Even though background is subtracted, ACS photons will increase the Yield in the Run Statistics table and the Peak in the "Lunar Return Rate" plot.

If you have enough signal, try to stay centered on the hitgrid by paying attention to the OUTER pixels rather than the inner ones, because the inner ones are more affected by ACS.

If you don't have enough signal, just trust the blind tracking.

Keep an eye on the "# of ACS photons" at the bottom of the Run Statistics table. If # of ACS photons is not going to reach 1200 before the end of the run, either lower attenuation more OR increase the # of shots in the run.

When the run is finished, record normal stuff in your observing log

Record Yield from bottom of Run Statistics table).

Record Peak from "Lunar Return Rate" plot

Record Guide Offset if you were able to follow tracking; leave blank if you used blind tracking

Add Comments to observing log if needed.

3.3.6 Other reflectors

The decision of what reflector to observe next will depend a lot on moon phase and conditions. It is desirable to get returns from as many reflectors as possible, and add ACS whenever the return signal is strong enough to support it. Russet usually makes a clockwise circle around the reflectors on the Moon, but it isn't necessary to do them in this order. If you are doing more than one circuit, however, you may want your second circuit to follow the same path as the first in order to get the longest spacing between two runs on each reflector. And in most cases it's desirable for the last run to be back at A15, in order to get the longest time baseline on the reflector with the strongest signal.

Expected returns from Apollo 15 vary mostly depending on conditions in Earth's atmosphere (clouds and seeing). In clear conditions with good seeing, peak signal rates from A15 may exceed 1 photon per shot, especially for moon phase around 100-140 and 220-260. Signal will be weaker near full moon (phase 150-210) even with good atmospheric conditions, but keep going even if it's a struggle because measurements near full moon are very valuable and only our system can successfully obtain those measurements!

Lunokhod 2 (the rover for mission Luna 21) is the weakest reflector, usually returning 1-5 percent of the Apollo 15 signal level. It rarely gives any returns in sunlight, and never gives anything at moon phases between about 140 to 220; but data from that reflector is very valuable because it's so rare. If L2 is in the dark, try for it unless seeing is bad and signal from A15 is very weak. If L2 is in sunlight, it may be worth trying if conditions are very good, if signal is very strong from A15, or if it's only just in sunlight. If you're going to try for L2, it's a good idea to go there immediately after A15 when pointing is optimized. Expect signal to appear at a predskew about 30-50 lower than the signal from A15; you can even move the predskew slider before beginning the run.

In most conditions, Apollo 11 is the second-strongest reflector after A15 and may be the second one you try if the situation is not promising for L2. The predskew for A11 should agree well with the predskew you found on A15.

Apollo 14 gives comparable returns to Apollo 11, usually a little bit weaker but sometimes stronger at moon phase near 90. The predskew for A14 is sometimes a little bit higher than A15, maybe by about 20. Start out with the A15 predskew and then adjust if you find the signal is a little off to the side.

Lunokhod 1, the rover for mission Luna 19, is the most variable reflector. In the dark it is sometimes almost as strong as A15, and data from this reflector is also very rare and therefore very valuable. In the sunlight L1 may not give measurable returns, but it's usually worth trying at least once (don't bother if the moon phase is near full, about 140-220). L1 also has a very low predskew, 60-80 lower than the predskew for A15, so you can move the predskew slider before starting your run. You may have to do an extra search raster to pick up L1, or you might have success by searching primarily in the azimuth direction (up/down arrows if you are using the APD offset frame, or right/left arrows if you are using the Native frame).

The areas around A11, A14, and L1 have few landmarks. The positions of A15 and L2 are more recognizable, but it can still be tricky to center up because the field rotates depending on where the moon is in the sky, and the shadows will look different at different moon phases. So in most cases you will rely on your previous centering from a crater or another reflector as a starting point. Here are images of each of the reflector sites taken near full phase with the Moon near the meridian in the south (with uncertainty for some of them):

Russet's preferred algorithm for choosing which reflectors to observe:

Signal from A15 was strong! (peak >0.5 photons/shot):

Is L2 in the dark? Then L2 with ACS enabled, but might keep attenuation high for the whole run. Then A11 and 14 with ACS, and L1 probably with ACS depending how it went on the other reflectors, and lastly back to A15 with ACS. For second circuit, adjust ACS depending on how the first circuit went.

Is L2 in the sunlight but not that close to full moon? Then try L2 with ACS not enabled, A11 and A14 with ACS, L1 probably with ACS depending how it went with the other reflectors, and lastly back to A15 with ACS. For second circuit, adjust depending on how the first circuit went.

Close to full moon phase? Then skip L2 and go to A11 and A14 with ACS, try L1 without ACS but maybe give up early if not finding anything, and lastly back to A15 with ACS. Second circuit probably just the three Apollo reflectors, adding ACS depending on how the first circuit went.

Signal from A15 was fine (peak 0.2-0.5 photons/shot):

Is L2 in the dark? Then L2 with no ACS. Then A11 and 14 with ACS enabled but might keep attenuation high, and L1 probably with no ACS, and lastly back to A15 with ACS.

Is L2 in the sunlight? Then try L2 once without ACS but maybe give up if not finding anything, A11 and A14 with ACS enabled but might keep attenuation high, L1 probably with no ACS, and lastly back to A15 with ACS. For second circuit, adjust depending on how the first circuit went.

Close to full moon phase? Then skip L2 and go to A11 and A14 with ACS, try L1 without ACS but maybe give up early if not finding anything, and lastly back to A15 with ACS. Second circuit probably just the three Apollo reflectors, adding ACS depending on how the first circuit went.

Signal from A15 was weak (peak <0.2 photons/shot):

Is L2 in the dark? Maaaybe try for it without ACS, or maybe skip it for the first circuit and then decide whether to try on second circuit. A11 and A14 probably with no ACS (could add ACS on second circuit if signal is better than expected). L1 without ACS and be prepared to stop early if no signal. Then back to A15 probably with no ACS, unless the signal improves.

Is L2 in sunlight? Then skip it and go to A11 and A14 probably with no ACS (could add ACS on second circuit if signal is better than expected). L1 without ACS and be prepared to stop early if no signal. L1 without ACS and be prepared to stop early if no signal. Then back to A15 probably with no ACS, unless the signal improves.

Close to full moon phase? Skip both Lunokhods and circle the three Apollo reflectors, probably without ACS at first but maybe adding ACS later if conditions approve.

3.3.7 Finish and cooldown

When you are nearing the end of the APOLLO observing session, hopefully after at least one run with signal and at least one run with ACS, you can usually continue ranging right up until the end time the observing specialist gives you. If something weird happened during the past hour, you might want to end 2-3 minutes early so you can do a final laser power check (but you can skip the threshold check and SHG adjustment), just to record any change in performance from the initial power check. This final power measurement should be done with the telescope at the altitude of the Moon, so before you hand the telescope back to the observing specialist. But if everything is performing well, you can skip the final power check.

If you got any useful data, you do want a Lasercal at the end. This can be done at any telescope altitude and it does not send light into the dome, so you can do it while the observing specialist is starting the instrument change. You want the laser shutter open (the LED in the Continuum display in the Laser Tuning tab) and the interlock shutter closed (ask the observing specialist to press the red button or just turn off the keyswitch), and # of shots set to 5000 even if some of your ranging runs used a different shotcount. Set these things up and press the Lasercal button on the left side of ATUI, and occasionally glance at the progress messages showing up at the very bottom of the ATUI window to make sure the Lasercal is progressing. It will take 2-3 minutes. Make a note in your observing log that you did the Lasercal (usually the last thing in a typical observing log).

If you used ACS during the run, you can power that down while the Lasercal is running.

Enter 'acs atten 60' in the command line of the Main/Active tab, and wait for that to complete.

Go to the ACS tab and uncheck LUN_EN (wait for LED to turn green)

Uncheck FID_EN (wait for LED to turn green).

In your VNC window to the acs-laser laptop, press Open Loop.

In the ACS tab of ATUI, uncheck PicoFYb Lock.

In the VNC window, close the Phaselock program

Press the Emission button in the Toptica program and close that program

Close Connect on your screen sharing.

In the ACS tab of ATUI, uncheck PicoFYB DC power

In the ssh window to cocoa that you forgot you still had open (or in the cocoa:/home/apollo/ACS/ directory if you need to reconnect), enter './ip-pwr.py pset 2 0' to turn off the main power to the ACS laser.

Once the Lasercal completes, you can press the Cooldown button at the left side of ATUI. This will spend a while cooling the main laser and Utah and the chillers, so the state will say COOLDOWN for about 10 minutes before it changes to IDLE. During this period the "Laser power" LED on the keyswitch box will say that the laser power is still on, but it is safe to quit and disconnect before the whole process finishes, and let housctl take care of everything.

Before you exit TUI, go to the ROOT tab and Disconnect. If you forget this step, closing the plots window will be a pain and it might need a Force Quit.

Oops, did you forget about powering off ACS until after you pressed the Cooldown button? No big deal; it took care of most of the power-off steps for you, but you might find some messy open windows the next time you VNC to acs-laser. And you do still need to do the final './ip-pwr.py pset 2 0' on cocoa, which is the only thing that Cooldown is not able to turn off for you.

4. Troubleshooting and Maintenance

4.1 Viewing/adjusting pulse train and build-up time

First, gather the equipment you need to take upstairs:

The APOLLO oscilloscope from the bottom storage drawer in the rear lab.

The flat black box holding burn paper and miscellaneous tools, from the cabinet in rear lab.

A BNC cable from the box labeled "coax cables" in the cabinet in the rear lab.

The black electronics box labeled "FAST PHOTODIODE" from the cabinet in the rear lab

Take these tools upstairs along with a laptop that can run ATUI. Set the oscilloscope on a cart or chair or box just outside of the sliding bars behind Utah, and get an extension cord to plug it in.

Get laser safety glasses out of the bin for everyone present. The lead person should use the Laser alignment glasses, which have a dark red frame instead of a black frame. This pair of glasses lets a small percentage of the green laser light through, so the person wearing them will be able to see the adjustments they are making. Our other safety glasses block all of the laser light.

Move the telescope to high altitude (82-83 for a short person, 80 for a tall person) so that you can access Utah, then make sure a stop button is active.

Start up ATUI and initialize with Take Control, StartNubs, Listen, ConnDev, and Get Status. (Warmup and the plots window are not needed for laser optimization.)

In the Laser Tuning tab, press the Laser Powerup button (wait for completion and shot count) and the Laser Warmup button. Let the laser flash for 5-10 minutes to warm it up.

Open the top and middle panels of the phone booth/TARDIS. Find the BNC cable coiled in the right side of the phone booth below the capacitor banks. Plug this cable into the EXT TRIG port on the lower right of the oscilloscope.

Use the BNC cable you brought upstairs to plug the fast photodiode into CH1 of the oscilloscope.

Has the laser been flashing long enough to warm up? In the Laser Tuning tab of ATUI, check the threshold, measure power, and optimize the SHG just as you would before ranging. With the telescope at high altitude, SHG probably wants to go several jumps clockwise (top button).

Get a roll of tape from the cabinet at the top of the stairs. Open the left door of the Utah box, and use the tape to hold the door open by attaching it to the phone booth.

Put on safety glasses before opening the laser cavity. The laser should not be flashing; confirm this is true, and then you can take off the safety glasses if desired, but keep them ready. Move the cavity door out of the way.

At this point, there are several different things you can do to check or optimize the laser health. Here is a description of different tasks approximately in order of how common they are:
Improve power using steering mirrors:

Check beam using burn paper:

Check/optimize pulse train and build-up time:
Use tape to secure the Fast Photodiode sensor-side up (keep tape off the sensor!) on top of the oscillator box, right next to the shutter but not blocking the shutter. Use more tape to hold up the BNC cable so that it will not pull down on the Fast Photodiode.

Measured laser power is exactly 0.0 and flat - turn bolometer paddle on

Measured laser power is around 0 with some noise - check flip mirror feeding light to bolometer

Checking APD on STV position

Pushed a wrong button and put STV in a weird state

Checking vtarget position / "fid flower": Focus mode - Setup - up/down until Filter - want "No"

Poor telescope pointing

Error message: "Shot time out of range: 103.488018 not in [100.322917, 100.475694]"

Error message: "Lunar frc mismatch"

No laser power

No laser in dome / no fid returns

Yes fids but no lunar returns

Yes fids, but shot rate is dropping below 18

Adjusting rear mirrors

Error message: "Failed to set initial ACS delay values during warmup" means pirx is powered off

Error message: "Can't open CAMAC" take top off Utah, wiggle cables on CAMAC crate

APOLLO Operator's Guide

Table of Contents

1. Hardware

3. Operations

4. Troubleshooting and Maintenance