Developer Guide

The Watcher CSC is implemented using ts_salobj.

The fundamental objects that make up the Watcher are rules and alarms. There is a one to one relationship between these: every rule contains one associated alarm. It is the logic in a rule that determines the state of its alarm.

Each rule monitors messages from remote SAL components (or, potentially, other sources). Based on that logic the Watcher sets the severity of the associated alarm. Rules are instances of subclasses of BaseRule. There are many such subclasses.

Each alarm contains state, including the current severity, whether the alarm has been acknowledged, and the maximum severity seen since last acknowledgement. Alarms are instances of Alarm.

Other Classes

Model manages all the rules that are in use. It is the model that uses the watcher configuration to construct rules, construct salobj remotes and topics and wire everything together. The model also disables rules when the Watcher CSC is not in the ENABLED state.

In order to reduce resource usage, remotes (instances of lsst.ts.salobj.Remote) and topics (instances of lsst.ts.salobj.topics.ReadTopic) are only constructed if a rule that is in use needs them. Also remotes and topics are shared, so if more than one rule needs a given Remote, only one is constructed.

Since rules share remotes and topics, the rule’s constructor does not construct remotes or topics (which also means that a rule’s constructor does not make the rule fully functional). Instead a rule specifies the remotes and topics it needs by constructing RemoteInfo objects, which the Model uses to construct the remotes and topics and connect them to the rule.

TopicCallback supports calling more than one rule from a topic. This is needed because a salobj topic can only call back to a single function and we may have more than one rule that wants to be called.

Rules are isolated from each other in two ways, both of which are implemented by wrapping each remote with multiple instances of RemoteWrapper, one instance per rule that uses the remote:

  • A rule can only see the topics that it specifies it wants. This eliminates a source of surprising errors where if rule A if uses a topic specified only by rule B then the topic will only be available to rule A if rule B is being used.

  • A rule can only see the current value of a topic; it cannot wait on the next value of a topic. That prevents one rule from stealing data from another rule.

Writing Rules

Contributing

lsst.ts.watcher is developed at https://github.com/lsst-ts/ts_watcher. You can find Jira issues for this module using labels=ts_watcher.

Python API reference

lsst.ts.watcher Package

Functions

get_filtered_topic_wrapper_key(topic_key, ...)

Get a key for a filtered topic wrapper.

get_rule_class(classname)

Get a rule class given its name.

get_topic_key(topic)

Compute the key unique to a topic.

run_watcher()

Run the Watcher CSC.

write_and_wait(model, topic[, timeout, verbose])

Write data and wait for it to be processed by the topic callback.

Classes

Alarm(name[, log])

A Watcher alarm.

BaseEssRule(*, config, name, ...[, ...])

Check one kind of ESS data, e.g.

BaseFilteredFieldWrapper(model, topic, ...)

Base class for filtered field wrappers.

BaseRule(config, name, remote_info_list[, log])

Base class for watcher rules.

FieldWrapperList()

A sequence of field wrappers.

FilteredEssFieldWrapper(model, topic, ...)

Track a field of an ESS telemetry topic, with a particular sensor name.

FilteredTopicWrapper(model, topic, filter_field)

Topic wrapper that caches data by the value of a filter field.

IndexedFilteredEssFieldWrapper(model, topic, ...)

A filtered field wrapper for an array field, with metadata indicating indices of interest.

MockModel([enabled])

MockOpsGenie([port])

A mock OpsGenie service to support Watcher escalation in unit tests.

MockPagerDuty([port])

A mock PagerDuty service to support Watcher escalation in unit tests.

MockSquadCast([port])

A mock SquadCast service to support Watcher escalation in unit tests.

Model(domain, config[, alarm_callback, log])

Watcher model: constructs and manages rules and alarms.

PollingRule(config, name, remote_info_list)

Base class for watcher rules that poll for data.

RemoteInfo(name, index[, callback_names, ...])

Information about a remote SAL component.

RemoteWrapper(remote, topic_names)

Simple access to the current value of a specified set of topics.

RuleDisabledError

Raised by BaseRule's constructor if the rule is disabled.

ThresholdHandler(warning_level, ...[, ...])

Compute severity for a rule that involves one float value with multiple threshold levels.

TopicCallback(topic, rule, model)

Call rules and/or wrapper callbacks when a topic receives data.

WatcherCsc([config_dir, initial_state, override])

The Watcher CSC.

Class Inheritance Diagram

Inheritance diagram of lsst.ts.watcher.alarm.Alarm, lsst.ts.watcher.base_ess_rule.BaseEssRule, lsst.ts.watcher.filtered_field_wrapper.BaseFilteredFieldWrapper, lsst.ts.watcher.base_rule.BaseRule, lsst.ts.watcher.field_wrapper_list.FieldWrapperList, lsst.ts.watcher.filtered_field_wrapper.FilteredEssFieldWrapper, lsst.ts.watcher.filtered_topic_wrapper.FilteredTopicWrapper, lsst.ts.watcher.filtered_field_wrapper.IndexedFilteredEssFieldWrapper, lsst.ts.watcher.testutils.MockModel, lsst.ts.watcher.mock_opsgenie.MockOpsGenie, lsst.ts.watcher.mock_pagerduty.MockPagerDuty, lsst.ts.watcher.mock_squadcast.MockSquadCast, lsst.ts.watcher.model.Model, lsst.ts.watcher.polling_rule.PollingRule, lsst.ts.watcher.remote_info.RemoteInfo, lsst.ts.watcher.remote_wrapper.RemoteWrapper, lsst.ts.watcher.base_rule.RuleDisabledError, lsst.ts.watcher.threshold_handler.ThresholdHandler, lsst.ts.watcher.topic_callback.TopicCallback, lsst.ts.watcher.watcher_csc.WatcherCsc

lsst.ts.watcher.rules Package

Classes

ATCameraDewar(config[, log])

Monitor ATCamera dewar temperatures and vacuum.

Clock(config[, log])

Monitor the system clock of a SAL component using the heartbeat event.

DewPointDepression(config[, log])

Check the dew point depression.

Enabled(config[, log])

Monitor the summary state of a CSC.

Heartbeat(config[, log])

Monitor the heartbeat event from a SAL component.

Humidity(config[, log])

Check the humidity.

Hvac(config[, log])

Monitor HVAC SAL topics for any alarming states.

MTAirCompressorsState(config[, log])

Monitor the summary state of the two MTAirCompressor instances.

MTCCWFollowingRotator(config[, log])

Check that the MT camera cable wrap is following the camera rotator.

MTDomeAzEnabled(config[, log])

Monitor the MTDome azEnabled event for any alarming states.

MTDomeCapacitorBanks(config[, log])

Monitor the MTDome capacitor banks for any alarming states.

MTForceError(config[, log])

Monitor the actuator force error of main telescope M2 is out of the normal range.

MTM1M3Temperature(config[, log])

Monitor M1M3 mirror temperature.

MTMirrorTemperature(config[, log])

Monitor the mirror temperature of main telescope M2 is out of the normal range.

MTMountAzimuth(config[, log])

Monitor MTMount azimuth.

MTOutClosedLoopControl(config[, log])

Monitor the main telescope M2 is out of the closed-loop control.

MTTangentLinkTemperature(config[, log])

Monitor the tangent link temperature of main telescope M2 is out of the normal range.

MTTotalForceMoment(config[, log])

Monitor the total force and moment of main telescope M2 is out of the normal range.

MTVibrationRotator(config[, log])

Monitor the main telescope vibration event based on the rotator detection.

OverTemperature(config[, log])

Check for something being too hot, such as hexapod struts.

PowerOutage(config[, log])

Monitor the PDUs for signs of a power outage.

ScriptFailed(config[, log])

Monitor the status of the ScriptQueue.

Telemetry(config[, log])

Monitor the presence of telemetry from a SAL component.

UnderPressure(config[, log])

Check for low pressure.

Class Inheritance Diagram

Inheritance diagram of lsst.ts.watcher.rules.atcamera_dewar.ATCameraDewar, lsst.ts.watcher.rules.clock.Clock, lsst.ts.watcher.rules.dew_point_depression.DewPointDepression, lsst.ts.watcher.rules.enabled.Enabled, lsst.ts.watcher.rules.heartbeat.Heartbeat, lsst.ts.watcher.rules.humidity.Humidity, lsst.ts.watcher.rules.hvac.Hvac, lsst.ts.watcher.rules.mt_air_compressors_state.MTAirCompressorsState, lsst.ts.watcher.rules.mt_ccw_following_rotator.MTCCWFollowingRotator, lsst.ts.watcher.rules.mtdome_az_enabled.MTDomeAzEnabled, lsst.ts.watcher.rules.mtdome_capacitor_banks.MTDomeCapacitorBanks, lsst.ts.watcher.rules.mt_force_error.MTForceError, lsst.ts.watcher.rules.mt_m1m3_temperature.MTM1M3Temperature, lsst.ts.watcher.rules.mt_mirror_temperature.MTMirrorTemperature, lsst.ts.watcher.rules.mt_mount_azimuth.MTMountAzimuth, lsst.ts.watcher.rules.mt_out_closed_loop_control.MTOutClosedLoopControl, lsst.ts.watcher.rules.mt_tangent_link_temperature.MTTangentLinkTemperature, lsst.ts.watcher.rules.mt_total_force_moment.MTTotalForceMoment, lsst.ts.watcher.rules.mt_vibration_rotator.MTVibrationRotator, lsst.ts.watcher.rules.over_temperature.OverTemperature, lsst.ts.watcher.rules.power_outage.PowerOutage, lsst.ts.watcher.rules.script_failed.ScriptFailed, lsst.ts.watcher.rules.telemetry.Telemetry, lsst.ts.watcher.rules.under_pressure.UnderPressure

lsst.ts.watcher.rules.test Package

Classes

ConfiguredSeverities(config[, log])

A test rule that transitions through a specified list of severities, repeatedly.

NoConfig(config[, log])

A minimal test rule that has no configuration and no remotes.

TriggeredSeverities(config[, log])

A test rule that transitions through a specified list of severities, repeatedly, when manually triggered by test code.