Professional Projects
Professional Projects
Grouped by category below are some of the projects I have worked on. The list is not complete, but serves as a good overview of my activities and duties.
Networking
- full BGP ASN using two ISP homes at colo facility with full Linux rack
- QoS over routed IPSEC for VOIP with several branch offices, offshore team
- converge separate data, phone and video networks into one using IP techs
- enterprise-wide Multicast routing and switching for multimedia with PIM-SM
- extensive VPN and firewall ACLs for collab with offshore devs and partners
- migrate VPN+firewall from Checkpoint FW-1 to Linux StrongSWAN and iptables
- move inter-office WAN links from leased lines to IP routed IPSEC links
- move from trunked, non-redundant to meshed, redundant LAN switching
- implement EAP-TTLS WLAN NAPs with RADIUS backends authing out of PAM/LDAP
- multi-site Postfix delivery routed directly out of user-configured LDAP
- Qmail delivery using script-generated member lists from Netscape LDAP
- Sendmail delivery out of flat, static alias files
- implement semi-sophisticated UCE controls on Postfix mail exchangers
- migrate SMTP routing from one-site MX (single hub) to multi-site MX/hubs
- multiple authenticated SMTP techniques implemented for novpn offsite relay
- migration of all tools that access mailstore from mbox to maildir format
- most all IMAP and POP software packages run in production over the years
Services
- run all setup and maintenance of hosted webapp/db instances for customers
- custom Syslog strata, loghubs regex-filtered logs to readonly NFS exports
- enterprise-wide fine-grained access control with POSIX ACLs, also to NT
- join auth database silos at each site into distributed company-wide auth
- migrate enterprise from Netscape/Solaris LDAPv2 to OpenLDAP/Linux LDAPv3
- migrate SMB file and print services from NT to UNIX SAMBA
- implement dynamic DNS via DHCP for enterprise-wide hostnames to all users
- all-site traffic link statistics gathering via SNMP with MRTG, Cricket
- network and service monitoring + alerts with Netsaint (aka Nagios)
- split name service into routable & RFC1918 zones on separate name servers
- reverse PTR delegation for DNS on both classful and classless nets
Automation
- implement role-based server provisioning system over network using Cobbler
- one-command instantiation of webapp customer instances (Apache/BEA/Oracle)
- scripted, RAID-image based provisioning of different server classes
- redundant, replicated directory services (DNS, LDAP, NIS, NT) company-wide
- enterprise-wide by-arch automounting based out of NIS, later LDAP
Database
- set up Oracle 9i RAC cluster on Veritas clustered filesystem
- implement clustered PostgreSQL on DRBD using LVM snapshots for backups
- track database maintenance and clone-to-dev events and store in database
- work with postgres developers to fix regression bug in sql query optimizer
Hacking
- implement shell script library for code reuse, highly factored scripts
- standard parsing of shell script args, options, and rcfile, autogen usage
- write syscall probe script using SystemTap to prove IO behavior of webapp
- custom web UI for users to select from multiple webmails, IMAPs, POPs
- hack open source web-based support tool to support LDAP logins
- expose Spamassassin filter configuration via custom web interface to LDAP
- custom web UI for config of email virus scanning using Vexira for UNIX
- custom source hacks to Qmail to make email shunt for legal case
- hack open source LDAP admin tool with customized web UI to full LDAP tree
Stuff
- convert real machine to KVM instance using a loopback copy: kpartx, grub
- maintenance of all development toolchains for all UNIX platforms
- IO latency tools, disk top, find stuff with disk IO accounting tools
- management of company SCM data in CVS and SVN, quasi-"release manager"
- all UNIX hosts moved from pam_unix and NIS to pam_ldap with LDAP
- migrate all Solaris-based services to Linux servers
- scores of Solaris, HP-UX, AIX installs for developers
- production Plone site for company information portal, import all data
- pilots of many webmail, ticketing, bug tracking, CMS, timesheet packages
- maintain and customize Bugzilla instance for use by development team
Administrative
- "Agile Systems Administration" sprints, estimates, burndowns, feedback
- several major IP/transport carrier negotiations (TWTC, Sprint, C&W)
- work with Sprint engineers on QoS queueing policies for WAN links
- consulting and recommendations to several small shops on contract basis
- give regular IT user orientations for all new employees
- work with several hosting providers to study cost scenarios for SaaS
- perform evaluation and interoperation tests with NetApp Filer
- formulate cluster split brain permutation matrices to rule out corruption
- custom enterprise LDAP schema created, custom schema additions
Backups
- Legato controlled AIT-2 changers at five sites
- AMANDA controlled AIT-3 changers with holding disk and RAIT at three sites
- sophisticated shell scripts for dump, rsync, RAID and LVM based snapshots
- implement dedicated backup net to increase frequency and offload data net
Facilities
- out-of-band server+power management via serial console, modem+network, PMU
- maintain Nortel Meridian PBX, CAT3 cube wiring, toneouts, et cetera.
- oversaw several office moves, 3 of these involving IT machine rooms/racks
- wire a new ~65-person office by hand using CAT5 110-blocks, punchdowns
- multiple data centers / colo sites / cabinets and racks wired