How do you monitor your system?

Because our system is small and run entirely by volunteers, we do not have a staffed 24x7 network operations center. The only system monitoring that we do is automatic. Monitoring provisions include:

• What's Up. We use What's Up Gold from Ipswitch. It pings each of our important network nodes, both upstream to our ISP (and to its ISP) and downstream to routers in each of our subscribers' homes. In this way we monitor:
  • our ISP's connection to the Internet backbone
  • our frame relay T1 line
  • our point-to-point microwave link from the T1 line to our neighborhood
  • our DSLAM
  • the DSL links to each of our subscribers (how we do it)
  • the IP connectivity to the DSL routers of the subscribers
  If What's Up detects a failure to respond, it logs the event and sends a text page to predetermined destinations such as cell phones and email addresses of system volunteers.
• MRTG. We use Multi-Router Traffic Grapher to graph traffic through our system. We separately graph the traffic on a number of links including:
  • our frame relay T1 line
  • our point-to-point microwave link
  • the link from the microwave system to the DSLAM
  • the DSL links to each of our subscribers
  We inspect the graphs regularly to see if any of our links comes close to being full. In this way we would know if we needed to increase capacity, for example on our feed to the Internet backbone. You can see excerpts from our MRTG configuration file.
• UPS self-tests. Our uninterruptible power supply tests itself every two weeks and emails the results of the test to system volunteers. If the UPS batteries were to require replacement we would thus receive early warning.
• DSLAM and UPS environmental monitoring. We monitor the UPS and DSLAM environment. You can see temperature and humidity and line voltage trends. Temperature readings are taken inside the UPS (UPS battery temperature) and in the cabinet that contains the DSLAM and UPS (cabinet ambient temperature).
• Microwave link monitoring. The point-to-point microwave system automatically steps down its data rate (from 3M bps down to 1M bps) if conditions degrade the wireless connection. We monitor and graph the data rate.
• Security monitoring. The Dillon office of Oppedahl & Larson LLP, which contains a router linking the frame relay T1 line to one end of the point-to-point microwave system, is alarmed with perimeter and motion sensors. The DSLAM cabinet in the barn is also alarmed. Violation of either alarm results in text pages to Coop volunteers.
• Qwest network operations. The provider of our frame relay T1 line (Qwest) monitors the line continuously. If Qwest detects a loss of service, it contacts us.
• What's Up Gold can receive SNMP traps, and our DSLAM can send SNMP trap messages to annunciate system events. At some point we plan to configure our DSLAM and our What's Up Gold to monitor and annunciate selected SNMP traps.