Content Skip to content.
San Diego State University

Network and Applications Status

2014 Outages

2014 Reported Outages

 

November 2014

 

 

  • AztecLink

    • Unplanned outage.
    • Date: 11/22/14 - 11/24/2014
    • Actual Length of outage 1d 4hrs
    • Contact: ETS Help Desk X45261

 

  • Shibboleth Single-Sign-On

    • Unplanned outage.
    • Date: 11/07/14
    • Actual Length of outage 8:00am-9:15am
    • Contact: Marcus Jeffers Ext. 45432

 

  • AztecLink Identity Manager System Updates

    • Planned outage.
    • Date: 11/07/14 (6:00am to 9:00am, 3.0 hours)
    • Actual Length of outage 1:41
    • Contact: Marcus Jeffers Ext. 45432

 

October 2014

Library servers

  • 10/09/2014
  • All Library servers affected by power outage
  • No access
  • Duration - 10:40 am to 12:17 pm
  • Contact: Infosys@library.sdsu.edu

 


September 2014

 

Voice mail server 

  • 9/28/2014
  • Voice mail server affected by data center upgrade
  • limited access to messages during upgrade
  • Duration - 6am to 8:30am (2.5 hours)
  • Contact: ksulliva@mail.sdsu.edu

 

Core router chassis reboot 

  •  9/11/2014  
  • Reboot AD/LSN core. 
  • administration including firewalls, industrial technology, physics-astronomy,  physics,  love library,  love library addition, scripps cottage, east commons, west commons, aztec athletics, chill plant, hepner hall, exercise and nutritional sciences, physical sciences, communications, arts and letters, engineering, art north, art south, facilities services, biology , engineering lab, life science south, psfa , hardy tower, life sciences north, storm hall
  • Duration - 10 minutes
  • Contact: austin@mail.sdsu.edu

 

 Core router blade reboot

  •  9/9/2014  
  • Reboot blade 2/2 in LSN core.  Brief outage in multiple buildings. 
  • Duration - 3 minutes
  • Contact: austin@mail.sdsu.edu

 


August 2014

 

Skillport Outage

  • Sporadic technical difficulties and connectivity issues.
  • Date: 08/05/2014 
  • Duration: 8:00am - noon (approximately 4 hours)
  • Contact: fvlahos@mail.sdsu.edu

Campus Voicemail Outage

  • Unplanned outage 
  • Date: 08/05/2014
  • Duration: 10:25am - 10:45am (20 minutes)
  • Contact: pelphrey@mail.sdsu.edu 

Wireless Controller Upgrade

  • TNS performed an upgrade to the campus wireless controllers. This update caused all wireless APs to reboot 2-3 times. Customers lost wireless connectivity for 2 minutes or less per reboot.
  • Date: 08/02/2014
  • Duration: 7am - 8am, Intermittent outages during this time.
  • Contact: jmcgovern@mail.sdsu.edu


July 2014


Network Redundancy Testing

  • TNS completed network redundancy testing as required by the CSU CNI (Campus Network Initiative) network core upgrade project. During this testing, some buildings experienced outages lasting up to an hour. Also, the network cores were rebooted at 2am causing 7-10 minute outages per reboot. Servers such as WWW and Rohan were briefly down at approximately 10:30pm.
  • Date: 07/15/2014
  • Duration: 10pm - 3am, Intermittent outages during this time.
  • Contact: jmcgovern@mail.sdsu.edu

DHCP Server Upgrade

  • TNS upgraded the campus DHCP servers. Maintenance lasted longer than expected, but actual service downtime was minimal (less than 10 minutes) due to redundant systems.
  • Date: 07/12/2014
  • Duration: 7am - 1pm
  • Contact: jmcgovern@mail.sdsu.edu

EBA 2nd Floor Outage

  • 7/10/2014  
  • 2nd floor EBA network down due to equipment failure.  We are currently cofiguring a replacement.
  • Duration 09:35am - 11:30am
  • Contact: austin@mail.sdsu.edu

Border Router Reboot

  • TNS applied a software upgrade to the campus border routers. During this upgrade, both routers were rebooted. These devices were in a redundant configuration so downtime was minimal (less than a minute).
  • Date: 07/04/2014
  • Duration: 12am - 2am, Less than one minute of downtime per reboot.
  • Contact: dibona@mail.sdsu.edu


June 2014


Skillport E-Training Down

  • Skillport E-Training site down for maintenance, upgrading to Skillport version 8
  • Date: 6/30/14
  • Duration: 24 hrs site up 7/1/14
  • Contact: fvlahos@mail.sdsu.edu

Wireless Controller Upgrade

  • TNS applied a critical security patch to our wireless controllers. During this upgrade wireless connectivity was distrupted intermittently while wireless access points rebooted.
  • Date: 06/15/2014
  • Duration: 7am - 7:50am
  • Contact: jmcgovern@mail.sdsu.edu

Data Center Network Maintenance

  • TNS migrated all data center network gear to the new Alcatel-Lucent network. During this migration data center servers and services were offline. Wireless connectivity was also affected intermittently.
  • Date: 06/01/2014
  • Duration: 6am - 5pm
  • Contact: jmcgovern@mail.sdsu.edu


May 2014


Voice Mail System Upgrade

  • Voice Services upgraded the voice mail system software.
  • Date: 05/24/2014
  • Duration: 9am - 6pm
  • Contact: ksulliva@mail.sdsu.edu 

BFA.SDSU.EDU Hardware Change

  • The bfa.sdsu.edu web server and all of it's virtual domains were migrated to new hardware.
  • Date: 05/23/2014
  • Duration: 4pm - 4:49pm
  • Contact: requests@mail.sdsu.edu 

Core Switch Maintenance

  • TNS worked with Alcatel-Lucent engineers to upgrade the campus core switches. This upgrade was performed to fix a known issue in the switches' operating system. During this maintenance window, each core switch was rebooted twice. Each reboot lasted for approximately 8 minutes per switch.
  • Date: 05/18/2014
  • Duration: 5pm - 12am
  • Contact: jmcgovern@mail.sdsu.edu 

Campus Power Outage

  • An SDG&E utility failure caused power issues in many of the buildings in the south and south west portions of campus. These power outages shutdown wired network switches and wireless access points.
  • Date: 05/4/2014
  • Duration: 5am - 3pm
  • Contact: jmcgovern@mail.sdsu.edu 


April 2014


Partial Network Outage

  • Some network users on both the wired and wireless networks were unable to receive an IP address due to a reconvergence of the routing table on two core network routers. Also, some wired network users could not reach a subset of local servers for the same reason.
  • Date: 04/30/2014
  • Duration: 5:56pm - 6:15pm
  • Contact: jmcgovern@mail.sdsu.edu 

Google Apps(Google Groups)

  • A change implemented by Google impacted the display options for Google Groups.   A temporary workaround provided by the Help Desk provided users with the appropriate display options.
  • Date: 04/25/14-4/29/2014
  • Duration: 4 days
  • Contact:  etshelpdesk@mail.sdsu.edu

SDSUcard Maintenance Outage


Core Router Upgrade

  • TNS upgraded the campus core routers to resolve a known issue in the Alcatel-Lucent software. Due to a flaw in the new software, the routers had to be reverted to their previous configuration.
  • Date: 04/20/2014
  • Duration: 6:00am initial outage. Connectivity sporadic until 11am.
  • Contact: jmcgovern@mail.sdsu.edu 

Wireless Router Migration

  • TNS migrated wireless services to a new set of Alcatel-Lucent routers.
  • Date: 04/5/2014
  • Duration: 6:00am initial outage. Limited connectivity on the wireless network until 10am.
  • Contact: jmcgovern@mail.sdsu.edu 


March 2014


Campus Network Outage

  • After a spanning tree reconvergence, the Alcatel-Lucent core routers failed to learn new MAC addresses due to a bug in their operating system. This is a known issue and is being addressed in a future update to Alcatel AOS 7.
  • Date: 03/27/2014
  • Duration: 9:30am initial outage. Connectivity fully restored at 5pm. Intermittent issues during that time with many clients completely down. Wireless was operating normally except for two outages at 2:30pm at 4pm lasting for 7 minutes each time.
  • Contact: jmcgovern@mail.sdsu.edu 

Power Outage

  • A power outage in some campus building caused a disruption to one of the ETS datacenters. Customers lost network connectivity as systems failed over.
  • Date: 03/19/2014
  • Duration: 3:51pm - approximately 15 minutes, 6pm - approximately 5 minutes
  • Contact: jmcgovern@mail.sdsu.edu 

SDSUCard Planned Outage

 

Google Apps

  • Google App Sync, Talk and drive slow and in some cases erroring with the error code 502.
  • Date: 03/17/2014
  • Duration: 9:30am - 12:30pm
  • Contact: etshelpdesk@mail.sdsu.edu 

Firewall Maintenance

  • The campus connections to the internet were briefly down due to border firewall maintenance.
  • Date: 03/8/2014
  • Duration: 7:00am - 8:00am
  • Contact: security@mail.sdsu.edu 

DHCP Maintenance

  • DHCP services were restarted briefly to apply fixes related to the 2/28 outage.
  • Date: 03/2/2014
  • Duration: 6:00am - 6:30am - A few brief service restarts during this window.
  • Contact: jmcgovern@mail.sdsu.edu 


February 2014


DHCP Issue

  • For a brief period on Friday morning a server which gives out IP addresses failed. The backup server failed to take over after the first failure leading to a short outage.
  • Date: 02/28/2014
  • Duration: 8:50am - 9:20am
  • Contact: jmcgovern@mail.sdsu.edu 

Network Wired and Wireless Outage

  • Unplanned outage on west side of campus affecting multiple buildings.
  • Date: 02/26/2014
  • Duration: 10:30am - 11:10am
  • Contact: austin@mail.sdsu.edu 

SDSUCard Transactions Outage

  • Planned maintenance.
  • Expected intermittent outages due to UCO Data Center network core upgrade.
  • Date: 02/23/2014
  • Duration: 6:30am - 7:30am, with many intermittent outages through 10pm.
  • Contact: emislang@mail.sdsu.edu

SIMS/R

  • Planned - Taken offline as part of UCO Data Center network core upgrade.
  • Date: 02/23/2014
  • Duration: approximately 13 hours (7:00am - 8:00pm)
  • Contact: simsr@mail.sdsu.edu

WebPortal

  • Planned - Taken offline as part of UCO Data Center network core upgrade.
  • Date: 02/23/2014
  • Duration: approximately 13 hours (7:00am - 8:00pm)
  • Contact: webportal@mail.sdsu.edu  

Data Center Upgrade Related Outages

  • TNS began the process of upgrading a large portion of the network equipment that is used in the campus data centers.  This upgrade caused various user facing issues such as lost connection to certain servers, lost wired network connections, lost WiFi connections, and an inability to login to SDSU_Wireless even if the WiFi network name was present.
  • Date: 2/23/2014 
  • Duration: 7am - 9pm (intermittent issues at various times during this time frame)
  • Contact: John McGovern, jmcgovern@mail.sdsu.edu

Wireless Network Outage

  • A problem with campus WiFi authentication servers caused WiFi users to not be able to join the SDSU_Wireless network.
  • Date: 2/21/2014
  • Duration: 7:00am - 10:05am
  • Contact: Skip Austin, austin@mail.sdsu.edu




    January 2014

    Network Outage

    • BFA users reported network outage. 
    • Date: 01/27/2014
    • Duration: approximately 2.25 hours (10:45am - 1:30pm)
    • Contact: Skip Austin, austin@mail.sdsu.edu

    Google Apps Outage

    • Google Apps experienced an unknown technical issue.
    • Date: 01/24/2014
    • Duration: approximately 1 hour from 1120-1220
    • Contact: ETS Help Desk, etshelpdesk@mail.sdsu.edu

    azteclink.sdsu.edu Outage

    • azteclink experienced a hardware failure.
    • Date: 01/10/2014
    • Duration: approximately 5 hours from 0915-1415
    • Contact: ETS Help Desk, etshelpdesk@mail.sdsu.edu

    SMTP.sdsu.edu slowdown

    • smtp.sdsu.edu experienced high spam traffic.
    • Date: 01/07/2014
    • Duration: approximately 4 hours from 0915-1315
    • Contact: Oudom Ky, ok@mail.sdsu.edu

    Storm-Nasatir Hall Wired Network Outage

    • The primary network subnet in Storm-Nasatir was not receiving network addresses.
    • Root cause is still being investigated.
    • Date: 1/6/2014
    • Duration: Early morning - 1:30pm
    • Contact: John McGovern, jmcgovern@mail.sdsu.edu

    Partial Campus Wired Network Outage

    • An unplanned network outage was caused by a failure of the core network switch that serves the south west portion of campus. Later that day, another core switch failed which serves parts of the northwest side of campus failed.
    • Both switch failures were caused by a bug in the switches' operating system that caused a disruption in the communication with their partner network devices.
    • This outage impacted all wired and wireless traffic in these areas.
    • Date: 1/3/2014, 1st switch failed at 6:30am, 2nd switch failed at 6:10pm.  
    • Campus building were back online by 11pm on 1/3.
    • All switches had to be rebooted late in the day on 1/4 to apply an operating system patch.
    • Contact: John McGovern, jmcgovern@mail.sdsu.edu