Anne Bennett's CV: Experience

Software infrastructure

I kept everything running smoothly. General system administration 1990-2011, IITS & ENCS

I did all the standard sysadmin stuff: Not to mention crawling under the floor. installing, configuring, and patching a Unix O/S, installing third-party applications (open-source or commercial), managing user accounts, dealing with tape backups, configuring network and printing services, writing or modifying Perl and shell scripts and C programs, designing and implementing small (SQL) relational databases, dealing with NFS, configuring ntp, creating web sites (and managing SSL certificates) and web pages (HTML, CSS, PHP). The idea is to keep everything running smoothly! I am a highly competent troubleshooter, though I take pride in minimizing the need for such, since I value quiet reliability over spectacular saves. At IITS from 1990 to 2003, I worked extensively with SunOS, Ultrix, and Tru64Unix (formerly Digital Unix, formerly OSF/1), and had a chance to occasionally at least look at other Unix flavours, such as RISC/OS, early versions of Solaris, and Irix. Since then (at ENCS) I've worked mostly with RedHat derivatives (including RHEL 3 itself as well as Scientific Linux 5), and a bit with Solaris (8, 9, and 10), though occasionally with other Linux distributions. At home I've been running NetBSD since the mid-1990s. I've also worked in a rudimentary way with "appliances" such as NetApp fileservers (ONTAP), APC UPSes, and Perle console servers.

I made major system changes with minimal user-visible impact. Clyde and Alcor upgrades 1993-1995, IITS

While even very early in my career I performed upgrades of hosts and even entire labs, these projects were my first where a highly visible production system had to undergo a major change, and where there was very little tolerance for downtime. In 1993 I coordinated a seamless transition of Clyde (DNS, mail relay, terminal server support, logging, time) from RISC/Ultrix to Alpha/OSF. We managed the actual moment of transition to the new machine with a downtime of less than one minute. In 1995 I planned the transition of Alcor (user interactive login and mail service) from RISC/Ultrix to Alpha/OSF; this project also merged formerly separate undergraduate and research communities onto one platform, and, for many of the undergraduates, marked a switch from VMS to Unix. The project was completed correctly and on time, and we were well prepared to support the sudden very large influx of additional users.

I helped create a system for software installation. Software layout 1993-2003 IITS, 2003-2004 ENCS

At IITS I helped create a software installation scheme and tools to support it, which allows software to be installed with minimal interference with the operating system (in particular so we can patch the O/S without clobbering the software), and to be documented thouroughly so that others can easily duplicate the installation. In 2003 I adapted that scheme (and those tools) for ENCS's multi-architecture, NFS-based environment.

I implemented automated system consistency checks. Management tools and consistency checks 2003-2010, ENCS

I improved the maintenance of filer export files by writing tools to generate them from simple patterns, greatly reducing the frequency of NFS export errors. I wrote many scripts and database views to warn us of anomalous conditions, preferably before they caused problems: shares and exports are identical on main and mirror filers, mirrors and host snapvaults are running on schedule, no unexplained world-writable files in our O/S or software areas, no world-writable user home directories, hosts database and accounts database internally consistent, hosts database consistent with inventory data, no ownerless files on the filer.

I planned the deployment of services to new hardware. Capacity and deployment planning 2003-2011, ENCS

I guided the allocation of services to hardware after major hardware acquisitions (2003 and 2008), and tracked the migration of services to new hardware. I provided reports on this allocation as needed by management. I encouraged the efficient use of hardware resources.

I improved redundancy and failover mechanisms for several services. Service redundancy improvements 2005-2010, ENCS

I advocated and planned for all services to run on at least two machines. Where possible I wrote code to assist in failover. The latest incarnation (2010) uses shared IP addresses; I wrote code for a machine to "safely" acquire the shared address, and to start certain services only if it is the current designated master for the service. I made it much easier for the Applications folks to build and operate redundant systems.

I arranged to send only relevant NAGIOS reports to cellphones. SMS alert system 2007-2010, ENCS

Using procmail and Perl, I arranged for NAGIOS alerts to be formatted and sent to analysts' cellphones, but only outside working hours, outside the monthly maintenance window, and omitting alerts for not-so-critical events. This has allowed us to react very rapidly to serious problems such as compromised accounts, power and cooling outages, or outages on critical services, without actually having any staff on duty 24/7.

Human infrastructure

I assisted users whose needs were not met by the Service Desk. User assistance 1990-2011, IITS & ENCS

Help Desks cannot handle all requests; when necessary I have provided advice on professor research computing needs and handled general user requests and questions. Doing so competently keeps users happy.

I helped draft departmental policies describing services offered. Policies 2006-2009, ENCS

I assisted the Director in writing policies on various topics including computer provisioning, user accounts, software licenses, and diskspace entitlements. Clear policies help manage user expectations, guide departmental priorities, and reduce conflict.

I helped interview candidates, and trained and mentored colleagues. Hiring and training 2005-2008, ENCS

I have been a member of hiring committees for analysts in my group, reviewing CVs, preparing interview questions, and participating in interviews. I like to come up with the right questions to find out what motivates someone, and whether they can learn quickly and interact clearly with others. I have also been responsible for training or mentoring several people, and proposed a 3-month training and evaluation plan for new analysts. We want to hire good people and help them develop.

I performed most of my group's project planning. Leadership 2003-2011, ENCS

I am the system administration group's de-facto project planner, and sometimes supply planning assistance to colleagues outside our group. I generally track our multi-person projects, and design frameworks to help us turn our procedures into a routine that reduces errors, such as keeping a file with the list of tasks planned for our next maintenance window, including "boot tests" of any systems which have undergone significant changes since their last reboot. I write a fair amount of our documentation, and organize it so that other people can more easily contribute. In group discussions, I keep the big picture in mind, and try to make sure that whatever decisions we make on small matters move us closer to our larger goals.

Physical infrastructure

I planned resources use in our new machine room. Plan for new machine room layout 2005, ENCS

The Faculty of Engineering and Computer Science needed to vacate two very small "machine rooms" (wiring closets with makeshift cooling) in its former quarters, and redistribute its server farm optimally among three machine rooms in a new building and one older machine room in a building nearby. I designed the new layout, planning for failover server pairs in different buildings where possible. I had to deal with outdated, inaccurate, or incomplete server room plans, and juggling space and power requirements. The new machine room layout optimizes redundancy; it would allow us to lose the primary machine room completely and still have most of the infrastructure for day-to-day operations back online within about a day.

I coordinated the move of our server farm into new facilities. Coordination of machine moves 2005, ENCS

In order to move the servers out of the old building, I planned the sequence of equipment moves, including failovers between redundant servers at the right time and in the right way, coordinated the work of our own technicians to disconnect, disassemble, reassemble, and reconnect our servers on schedule so that external movers could move the equipment at pre-assigned times, and kept the Network Group informed of our plans and progress so that switches could be configured and ready for us. I also supplied information for users about expected outages (for example, it takes a couple of hours to fully fall back to a mirror fileserver), and information for our own Inventory group as to which equipment had been moved where. The equipment moves went smoothly and on schedule, and most user-visible downtime was limited to the time required for filer mirrors to resynchronize.

I created a database to track the location, power, and use of our servers. Hosts database 2005, ENCS

To be able to accomplish these planning and tracking tasks at all, I designed, documented, and implemented a "hosts database", including importing and cross-referencing selected parts of the Inventory database, which required negotiating common keys with the Inventory group. This database continues to facilitate our system administration tasks by helping us track O/S revisions, host responsibilities, equipment locations and rack layouts, power supply (breaker panel, circuit, UPS, and host correlations), and so on. Certain changes cascade through the use of database constraints, and views are available for frequently used reports and for sanity checking the data.

I analyzed and presented data concerning cooling emergencies. New building teething pains, part 1 (cooling issues) 2005-2009, ENCS

Cooling losses have occurred often, due to everything from human error to a design flaw in the fire control system, and we have had to arrange to be notified quickly, to have procedures in place for mitigating the problem while awaiting a fix, and to supply data to allow our management to apply pressure on the Physical Resources department. Several times I have intervened remotely to prevent temperatures from reaching critical levels. I also created temperature graphs from UPS probe data to demonstrate to management the need to react quickly, and I documented response procedures.

I increased our ability to survive power failures. New building teething pains, part 2 (power issues) 2005-2009, ENCS

We also had frequent power outages, including outages of generator-backed power while regular power was available! I coordinated an inventory of floor power positions and a reallocation of power to optimize redundancy (power from both regular and emergency circuits) for every rack, which resulted in fewer service outages due to power failures. I wrote or improved tools to gather and report temperature and projected runtime data from all UPSes, to help us react more efficiently during a power or cooling emergency, by shutting down hosts which would otherwise crash, and shutting down low-priority hosts and large heat producers to keep the core part of the installation alive longer.

I improved the management of our machine rooms. Space/power planning, general room management 2003-2011, ENCS

I tracked and assigned rack and floor space for new equipment, as well as power circuits for new UPSes. I recommended and tracked electrical work when upgrades were needed. I tracked tasks required for the installation of new equipment (inventory, rack-mounting, power cabling, network cabling and switch configuration, console cabling and configuration, clean-up, filing of documentation). All of this eliminated delays due to poor planning. I coordinated a clean-up of the machine rooms (especially the old one), got rid of unneeded material, labelled racks and hosts clearly, sorted and boxed spare parts, helped re-run cabling neatly, and helped ensure the presence of needed tools. As a result the machine rooms are easier to work in, which reduces errors and time wasted. It's also more pleasant and professional looking, which provides good PR when VIPs visit - on one occasion I created foam core rack ends to make the room look neat for an important photo-op!

Computer security

I helped design a security checklist for hardening Unix systems. Hardening Unix systems 1991-2003 IITS

During the 1990s I contributed to the IITS Unix Group's guidelines on hardening Unix systems; this checklist was used before allowing a system to be used in production. Nowadays systems usually ship with a much more conservative configuration, and further tightening is done by scripts or configuration management systems. At the time, though, it was a breakthrough for us to systematize our configuration procedures and help ensure consistency and security of configuration.

I contributed document clarifications to the draft ssh procotol version 2. IETF "secsh" Working Group 1994-1997, IITS

I contributed (mostly document clarifications) to early versions of "draft-ietf-secsh-userauth", which eventually became RFC 4252 ("The Secure Shell (SSH) Authentication Protocol"). At the time I was involved in implementing the use of ssh within my department, to combat password sniffing.

I investigated incidents and provided data and analysis. Incident investigations 1995-2010, IITS & ENCS

I investigated incidents (be they anomalies detected by the sysadmins or reported from outside) and complaints (again internal or external) of abusive behaviour or suspicious computer behaviour. I provided data requested via subpoena or by the authorities via Legal Counsel, and provided informal assistance where appropriate to neighbour sites. I provided effective, discreet, and trustworthy interventions.

I contributed material and editorial assistance toward RFC 2350. IETF "grip" Working Group 1996-1997, IITS

I contributed to RFC 2350 (1998), both with editorial assistance, and by testing the proposed framework using my own institution as an example, showing what would be required to create an incident response team at Concordia.

I advocated for improving the University's computer security practices. Security advocacy 1997-2003, IITS

I am a co-author of Computer Security at Concordia: Past Problems, Proposed Plans (1998); which educated management about the need to improve computer security, and which received a fairly wide readership at the time, even outside the University. I was also a member of the programme committee for the 2002 and 2003 FIRST conferences, which raised the profile of our institution and made external contacts for incident response.

I co-created a manageable file integrity checking system. Remote file integrity checking 1991-2010, IITS & ENCS

In 1991 I installed the COPS security on IITS Unix systems, and created and installed a "remote" version of crc.chk. "Remote MD5" (rmd5) was co-authored with Steven Winikoff and based in part on concepts from "COPS-1.04+". It checks the integrity of the files on a remote system, works in a way that is difficult to detect by an intruder trying to cover up their traces, and is extremely difficult to tamper with without detection. It continues to monitor IITS Unix Group production systems, while a slightly modified version is now in use in ENCS, and reports daily on unexpected file changes on any Unix host managed by the System Administration Group, allowing us to quickly correct system administrator errors which might have led to problems later if not caught early. So far, we've caught no intruders, though!

I authored sections of the University's Policy on Computing Facilities. Policy on Computing Facilities 1998-1999, IITS

I am the author of the Forbidden Activities List and a contributor to the Concordia Policy on Computing Facilities. The policy received a fair amount of input from me; in particular, items 4, 5, 7, 32, 33, and 34 contain text that I proposed. The policy clarifies the scope of responsibilities of system administrators and explicitly addresses unsolicited mailings.

I adapted and improved a script to monitor for vulnerable web servers. Monitoring for vulnerable web servers 2002, IITS

IITS runs daily scans of the Concordia network for vulnerable web servers using a Perl script I adapted from a summer 2001 Unisog mailing list posting by David Dandar (Old Dominion University, Norfolk, Virginia, USA) of a script to detect IIS web servers vulnerable to the Code Red worm. Over time, I added detection of other vulnerabilities (to Nimda, for example), defaced machines, machines using abusable "FormMail" software or unrestricted proxies, and hosts advertising Kazaa file transfer services. The script allowed us to intervene quickly if a vulnerable or compromised web server appeared on the University network. I received several messages informing me that my script was in use and appreciated at sites from Auckland, New Zealand to Waterloo, Ontario. (Nowadays, this type of test would be covered by Nessus.)

I scripted a report of infections from mailer virus scanner logs. Reporting virus-infected hosts 2002-2003, IITS & ENCS

While implementing virus scanning of e-mail at the relays, I caused the software to log sufficient information so that virus-infected hosts could be identified. I created a Perl script to extract the hostnames and virus names of all such Concordia computers, making it possible to arrange to notify the owners of infected computers of the need to intervene, and thus reduce virus infestations at the University. We use an improved version of my script at ENCS.

I monitored security vulnerability mailing lists for items relevant to us. Vulnerability research monitoring 2003-2010, ENCS

I monitored the SANS @RISK security vulnerability mailing list (and a few others as appropriate) for information affecting our installations, and summarized so that we could quickly intervene to close vulnerabilities on our systems.

I represented the University in a province-wide security forum. RISIUQ 2005-2009, ENCS

For some time, Concordia had no official computer security person, and no representation on RISIUQ (a working group of Québec university computer security people). Based on my reputation in the local sysadmin community, RISIUQ asked me to represent Concordia, and IITS did not object (despite the fact that I had moved to ENCS). I was able to report information, concerns, and trends in Québec university computer security.

Directory services and networking

I provided DNS expertise and ensured its reliability. Hostmaster and/or DNS resource person 1992-2011, IITS & ENCS

In 1991 I converted SunOS hosts to use the newfangled DNS! For some years, I performed "hostmaster" duties (routine data changes). After I handed that off, I remained the "DNS resource person" in IITS, and later in ENCS: I installed, configured, and maintained ISC bind, configured the DNS servers (zones, access control, etc.), researched DNSSEC, performed any large and potentially disruptive data updates, coordinated obtaining offsite secondaries, wrote small tools to facilitate data management, and answered questions from colleagues. I ensured complete reliability of this essential infrastructural service.

I helped manage the University's network. Substitute network administrator 1992-1994, IITS

I helped administer the Concordia network (mostly IPv4, some Appletalk), and was able to configure Cisco routers and write packetfiltering rules. I have a good understanding of how networks function, though little recent practical experience managing them.

I guided the implementation of AD and MIT Kerberos. IITS Active Directory implentation: project resource person 2002, IITS

I gave considerable assistance to the project manager in terms of understanding how the technology could be made to fit into the existing and proposed network infrastructure (DNS, central authentication, account management), and identifying and tracking tasks to be performed. My assistance helped make it possible to have this large project ready on time. In parallel with the Active Directory implentation, my colleagues and I put up an MIT Kerberos server "under the radar", which turned out to save the University lots of money when it was discovered that we were not licensed to use AD for authentication of the Portal in the way that had been projected.

I designed a DNS layout using sortlists to deal with multi-homed services. DNS system design, part 1 (multi-homed network) 2004, ENCS

I designed a system that used sortlists on the resolvers to ensure that clients got the best address first for multi-homed services. This greatly simplified the configuration of clients by making it possible to specify the same hostname everywhere for fileservers (especially in automounter maps), mail servers, and NIS servers. I also wrote a Makefile that checked for errors before trying to reload a zone, eliminating data errors and failed updates.

I integrated RADIUS with our switches, UPSes, and console servers. RADIUS server integration 2006, ENCS

I configured the OSC Radiator RADIUS server to authenticate network users as requested by switches implementing the IEEE 802.1X protocol, and wrote local code to properly log activities despite having queries encapsulated within a TTLS tunnel, and also to direct queries to the correct PAM service despite such encapsulation. Note also that the RADIUS server runs within a chroot not only to protect the system against programming errors, but also to allow the use of PAM for RADIUS without affecting the operating system's configuration.

I helped streamline the management of DHCP data. DHCP system design 2008-2010, ENCS

I assisted the Network Group in designing and implementing a more reliable and automated way to manage DHCP data, with better redundancy, leading to fewer DHCP data errors, less time spent managing the data, and smaller likelihood of service outages.

I improved the robustness of our DNS master system. DNS system design, part 2 (multiple masters) 2009-2010, ENCS

In order to ensure that we could lose one of two machine rooms and still continue operations, we had to ensure that we had a DNS master in each room, where we could make DNS data updates that would propagate correctly to all of our secondaries, onsite and offsite. I created an out-of-band (rsync-based) DNS data and configuration replication mechanism, which is controlled with a Makefile, and which keeps the "backup master" synchronized and usable at any time.

I helped plan and implement a transition to a new network design. "New network", project planning and tracking 2009-2010, ENCS

I participated in designing the Network Group's "new network", and wrote a project plan for its implementation, taking dependencies into account. I tracked the tasks, and helped implement the "host-based" parts of the project (new IP addresses, changed routing), as well as some of the DNS changes. I assisted the Network Group in ensuring that major network changes were implemented with negligible user-visible impact.

I modified our DNS data design to make more extensive use of views. DNS system design, part 3 (de-multi-homed, views) 2010, ENCS

When the Network Group "de-multi-homed" our network and started moving significant portions of it (including clients, which need access to the public Internet) into private network space, I removed the sortlists, but facilitated our use of views to avoid leaking RFC1918 addresses outside our network by judicious use of the $INCLUDE directive, Makefiles, and code to auto-increment serial numbers when one of the INCLUDEd files had changed. The Network Group was happy that DNS updates became less error-prone.

E-mail and Netnews

I ran the netnews service during a period of rapid growth. Newsmaster 1991-1999, IITS

I was Concordia's newsmaster That was fun! in the years when the universities were still the netnews transmission backbone (early to mid 1990s). As newsmaster of one of the top few hundred Sometimes in the top 200, actually. sites worldwide at the time, I kept the news service running relatively smoothly for our own users through a period when the news volume was doubling annually. I changed machines twice, both times to new architectures, and the second transition was transparent to the users. I also helped neighbouring sites by providing advice, and I kept statistics of the RISQ's (provincial research network) netnews traffic as seen by our server. I forged excellent working relationships with colleagues at many sites throughout Canada (and a few outside it). When necessary, I wrote reports to management on the status and future of the service. Finally, in 1996 and 1997, I participated in the IETF "nntpext" Working Group on extensions to the NNTP protocol.

I ensured the reliability of the mail relay and delivery services. Postmaster 1991-2011, IITS & ENCS

I ensured the smooth running of the mail relay and delivery services, first for the University, then for the Faculty of ENCS. This involved project planning, system design and implementation, configuration, and, as well as documentation (internal and for users), staff training, troubleshooting, performance and availability monitoring, user assistance, and assistance to staff members in other departments. The mail system when I started was MMDF; I switched it to sendmail in 1991, and to sendmail.mx (to use DNS MX records!) later the same year.

I prevented the bypassing of Majordomo mailing list security. Majordomo mailing list protection, phase 1 1996, IITS

In 1996 I designed a mailing list protection scheme which maintained the confidentiality of Concordia Majordomo subscription lists, and also prevented anyone from bypassing the moderator's approval. Because of the way the Majordomo mailing list manager stores its subscription lists by default, an attacker can obtain a subscription list by querying the sendmail daemon (unless EXPN is disabled, which may not be desirable in all circumstances). Also, the moderator's approval can be bypassed by a canny attacker, who mails directly to the post-approval address. In pre-8.12 versions of sendmail, by making a separate copy of sendmail for use only by Majordomo, I was able to hide the subscription list from the sendmail listener daemon completely, and thus prevent both attacks.

I introduced procmail-based spam control at the per-user level. Spam control, phase 1: user level 1996, IITS

Later superseded by the Concordia central de-spamming service, the Alcor procmail spamcatchers were recognized in the Linux Journal in 1998 (see Publications and Other Contributions for details). This first generation of anti-spam filters at Concordia also drew many messages of thanks from all over the Internet, where people made their own copies of it for their personal use. I was the technical half of the team that created these filters; Neil Schwartzmann (then of the Faculty of Fine Arts) provided data in the form of new spam sources to be blocked every week, which I translated into procmail recipes. Note that this approach, while it was useful in its day, did not scale well either in terms of system performance or in terms of the maintenance required to keep the recipes up to date. However at the time, for people who enabled their use, they greatly reduced the amount of spam in the main inbox.

I eliminated unauthorized mail relay from hosts which I managed. Unauthorized mail relay elimination, phase 1 (per-host) 1997, IITS

Before anti-relay "check_*" rules started to come standard with sendmail, I implemented such rules on the hosts which I administered, and put up a web page containing my annotated ruleset. Thus I stopped my hosts from being used to relay spam, and made useful information available to the sysadmin community.

I extended the anti-relay system to protect the entire campus. Unauthorized mail relay elimination, phase 2 (central) 1997, IITS

The unauthorized use of all of Concordia's e-mail facilities to relay spam to other sites was stopped in 1997 when I coordinated a centralized e-mail anti-relay scheme for Concordia. Transparent relaying of mail to all client systems, which ensured the acceptance of the scheme by the university's de-centralized sysadmin community, was accomplished using a mailertable automatically generated from DNS data on our domains. This implementation stopped all hosts at Concordia from being used to relay spam (in particular those not managed by professional sysadmins), without requiring any special per-host configuration.

I designed the Concordia University Broadcast E-mail service. CUBE (Concordia University Broadcast E-mail) service 1999-2000, IITS

I planned and implemented a service to allow University authorities to send e-mail announcements to groups of students, which improved communication with students.

I centralized spam and virus control on a per-host basis. Spam (and virus) control, phase 2: host level 2001, IITS

Starting in November 2001, all mail passing through Alcor was scanned for viruses. The virus scanner (Sophos with sophie) was initially connected to the mail system using AMaViS, but as of November 2002, MIMEDefang was used instead, as it also provided facilities to connect to spam-identifying software (such as our choices, SpamAssassin and Razor 2). Implementing the spam and virus detection function at the mail host level instead of at the delivery to the end recipient simplified the configuration from the user point of view; the number of virus-related calls to the IITS Help Line dropped dramatically after the implementation of virus scanning.

I redesigned Majordomo mailing list protection when sendmail changed. Majordomo mailing list protection, phase 2 2002, IITS

sendmail version 8.12 made major changes to the mail submission process, so I redesigned my scheme in January 2002 to instead define and use a "regex map" in sendmail.cf to restrict access to the sensitive addresses, thus maintaining the protection described above.

I extended spam and virus control to protect most of the campus. Spam (and virus) control, phase 3: institutional 2002, IITS

By 2002 (one year later), I had extended the virus and spam scanning service to all three IITS mail relays. This protected most of the campus against e-mailed viruses, and reduced the amount of spam faced by our users.

I automated updates to the e-mail virus control system. Automated download of Sophos virus signatures 2002, 2005, 2008, IITS & ENCS

I adapted a Perl script to receive e-mailed announcements of new virus signature files available, download and install them, and restart the virus checking daemon. This allowed us to detect and block virus-laden e-mail as soon as possible, reducing the amount of staff time lost to dealing with virus infections.

I designed new systems for mail relay and delivery. Mail relay and delivery (design) 2003, ENCS

My first assignment when I joined the Faculty of Engineering and Computer Science in 2003 was to design a mail relay and delivery system which would not only implement anti-virus and anti-spam scanning similar to that which I had deployed at IITS, but also allow mail for all of its different subdomains to be delivered centrally, while still allowing for special per-domain aliases as needed. In addition, mail from thousands of desktops had to be delivered to the right place. I accomplished this using almost exclusively free and open source software: sendmail, sophie, MIMEDefang, SpamAssassin, Razor 2, University of Washington's IMAP server, and procmail. Only the virus scanning engine (Sophos) was commercial. The net result was an inexpensive, flexible, and reliable mail system which met the Faculty's needs.

I planned the transition to new mail relay and delivery systems. Mail relay and delivery (transition) 2003-2004, ENCS

Merging mail services for several departments required gathering and merging user and alias lists for all subdomains, as well obtaining and analyzing mail configurations from widely disparate mail software systems. Mail continued to work seamlessly during the transition from the various departmental mail servers (including several Unix systems, a Windows system, and a VMS system) to the central servers, and the transition did not require complex reconfiguration on the part of most users.

I automated Mailman list membership management. Mailman list membership management 2005, ENCS

In 2005 I wrote Perl code to automatically synchronize the membership of Mailman mailing lists based on user data from our employee and student NIS netgroups, and in 2006 I added an exception mechanism for manual additions. As a result, it is no longer necessary to maintain the announcement lists explicitly; they just track the data that is maintained anyway for other purposes.

I implemented spam control for Mailman lists and RT queues. Spam control for Mailman lists and RT queues 2005, ENCS

Also in 2005, I wrote code snippets (procmail recipes and small Perl scripts) that allow us to de-spam Mailman lists and RT (trouble ticket system) postings. This has greatly reduced time spent cleaning spam out of RT queues. A beneficial side effect has been the short-circuiting of auto-generated mail loops when people or programs use RT queue addresses incorrectly in e-mail.

I analyzed and increased the performance of our spam control system. Spam control efficiency 2006, ENCS

In 2006, I improved our sendmail's use of DNSBLs to reject more traffic before the expensive scanning process kicks in, I implemented several sendmail features (greet pause, bad recipient throttle, and connection rate control) likewise, and I added the envelope sender to the Received lines to enhance our de-spamming capabilities. In MIMEDefang, I added user existence check callbacks (to avoid accepting mail for non-existent users), and implemented a cache to reduce the performance penalty of this check. All this made the de-spamming much more efficient, improved mail turnaround time, and delayed the need to buy faster hardware.

I automated rule updates for the spam control system. Spam control maintenance automation 2006, ENCS

In 2006, I upgraded to the latest SpamAssassin and implemented its automatic rule update service. As a result, it was no longer necessary to manually track spam rule updates.

I improved failover for mail delivery/IMAP/webmail. Improved failover for mail delivery/IMAP/webmail 2008-2009, ENCS

Failing over to the fallback mail delivery server formerly required many steps of human intervention. In consultation with the application specialist for webmail, I documented the procedure and then scripted it; now it is possible to move the services between hosts painlessly and without the intervention of the webmail administrator.

I prevented e-mailed responses to known phishing messages. Phish response prevention 2008-2010, ENCS

In an attempt to educate users, I crafted an announcement to discourage them from falling prey to identity theft, especially attempts to phish their ENCS credentials. This had only limited success, so I created a "known phish list", and code to prevent e-mailed responses to known phishing addresses. I automated a multiple-times-daily download of shared phish address data from other sites, which was added to our own list. All this reduced the frequency of ENCS credential phish victimizations.

I coded an early-warning system for compromised webmail accounts. Phish/spam detection and response 2008-2010, ENCS

Despite all our efforts, some users still give away their passwords, which are then used by spammers, usually from our webmail service. I created a "webmail preferences monitoring tool" to advise us within 15 minutes of any suspicious profile changes (such as setting an offsite Reply-To address), and I created and documented a procedure to shut down such activity, including creating or adapting tools to kill webmail sessions, and to show and delete user profiles. As a result we are able to intervene much more quickly to detect and shut down the abuse of stolen credentials.

I created a system to sort Service Desk mail into appropriate RT queues. Help Desk incoming mail sorter 2009, ENCS

At the Director's request, I wrote code to classify incoming mail to the Help Desk according to whether the originator is a student, staff, or faculty member (based on account management data), and to submit it to the appropriate RT queue in consequence. The scheme includes an exception mechanism to handle known offsite addresses and other misclassifications. The result was as anticipated: user satisfaction with the Help Desk services increased as their requests were more likely to be directed correctly in the first place.

User account management

I coded LDAP and Kerberos account management tools. Account management back-end code for AD and Kerberos 2002, IITS

I implemented code to manage users in Active Directory and Kerberos (add, delete, change password, etc.), based on directives received from a central account management system. This allowed these new directory services to be provisioned in the same way as our other computer accounts.

I led a tem to integrate computer accounts from five departments. Account Integration Task Force 2003, ENCS

I led a six-person team tasked to integrate the user accounts of two (and eventually all five) member departments of the Faculty of Engineering and Computer Science. This involved merging NIS directory entries, diskspace on the filer, and application configurations for users' Unix and Windows environments, including resolving username and UID conflicts, and where a fair amount of data (such as who the account owners were) was initially missing. Despite the complexity of the task, the short timeframe, and the initial mutual suspicion of the team members (who were themselves being "merged" not entirely willingly), we accomplished the task on time, and with no disasters and very little inconvenience to the users. In addition, we forged a strong working relationship that formed the core of the current System Administration Group.

I designed an account management database. Account management database 2003, ENCS

While untangling outdated and incomplete data for thousands of users, I designed an account management database. While less than half of the original design is implemented thus far, even in its current state, that database has become the heart of all calculations for automatic student account creations and automatic privilege calculations for all users (based on employment status, course or program registration, department membership, teaching assignments, and so on). It also allows us to track ownership of "special" accounts generated for guests, for course management, for administrative roles, and so on. The database includes data integrity monitoring views, so any errors we introduce are reported the next day (and fixed posthaste). The system includes fairly complex code to mirror in data from the Faculty Information System while preserving constraint relationships within the database — that is, it updates existing records in place and adds new records as needed, but where relevant it does not delete old records, so that associated usernames do not lose their "ownership". In addition, rigorous error checking is performed on incoming data, which avoids entering inconsistent data, and regularly flags problematic data from our primary sources (which can then be corrected).

I revolutionized our account management code and procedures. Account management automation 2004-2010, ENCS

I improved the automation of account creation and expiry, as well as the rotation of per-semester resources (such as course web pages): the code is now more independent, covers more cases, and is much more modular (re-uses code). Even after the first iteration, the start of term stopped being a chaotic nightmare for two weeks, and instead represented an only slightly heavier workload for analysts. Account deletion when the user is no longer entitled to privileges is now automatic, thus avoiding security issues related to incorrect retention of privileges. We can usually detect when a student becomes an employee, and avoid incorrectly deleting such accounts. We now automatically track whether users have the right disk quota with respect to their status. The remaining manual procedures are well documented. The net result is a user account management system that is mostly automated, requires manual intervention only for infrequent actions, and assigns and removes privileges automatically.

I improved our filer usage stats and over-quota reporting. Over-quota warnings and filer usage statistics 2009-2010, ENCS

I improved our over-quota warning system to better recognize the start and end of "incidents" (to avoid unnecessary multiple warnings), to log its actions, to give users clearer messages, and to be more reliable. The Help Desk reported more reliable performance, especially for over-mail-quota incidents. I wrote reporting code to allow for better decisions with respect to diskspace capacity planning and allocation.

Web programming and page authorship

I installed Gopher and Archie, precursors to the web and search tools. Archie client installation, Gopher implementation 1991-1994, IITS

In 1991, I installed client software to use Archie, a search engine for ftp files, which was the grandfather of modern search engines.
In 1994, I implemented a Gopher server for our department, with indexing and log analysis; it was a precursor to web-based methods of sharing information with the University community.

I pioneered web-based information sharing at the University. Web site design and web page authorship 1995-2003, IITS

I was a major contributor to the Alcor web site, for which I created the original basic design (both content and visual). I also wrote a significant fraction of the material on this site, which was one of the University's first uses of the web to provide information to its user community.

I designed a safe environment for user web pages using chroot and cgi-wrap. Safe multi-user web server 1996, IITS

I was a main designer for the safe user home pages and CGI scripts, implemented with a combination of a chrooted environment for the web server and the user web pages, proper configuration of httpd, and the use of cgiwrap. This set-up allowed thousands of users to run their own CGI scripts without endangering the system, other users' web data, or even their own home directory data. Note that all of these concepts are now in use in ENCS not only for our user web pages, but for several major web-based applications as well.

I created "hole-in-the-chroot" to connect fragile apps to a web server. Hole-in-the-chroot 1999-2000, IITS

I wrote a set of tools which made it possible to run a web server and CGI scripts in a "chroot" environment (for safety), and yet still communicate (in a protected and restricted way) with possibly vulnerable applications outside the "chroot"; these tools were used to safely make available the University's financial reports to authorized users. The first first generation system was presented as a refereed paper at the 13th FIRST conference as Securing web-based applications with hole-in-the-chroot (2001). The second generation tools were used to manage accounts for the Wireless network and to manage access to ISP services for a Concordia-Sympatico partnership.

I automated dead link checks for my group's web pages. Web link checker 2006, ENCS

I automated checks for dead links on our group's web pages, helping us keep information up to date.

In addition, before 1983, I held jobs as a pressing plant worker, a waitress, a parking lot attendant, a demonstrator at a science fair, and a babysitter!

IT positions (1990-now)