Software: Puppet, Terraform, Apache, Nginx, HAProxy, CloudStack, memcached, LogStash, mcollective, make, postfix, Cacti, Nagios, Graphite, collectd, SNMP, DNS/Bind, DHCP, rsyslog, RabbitMQ
Operating Systems: Linux (Fedora, RHEL, CentOS, Ubuntu, Debian), OSX
Databases: MySQL, Postgres
NoSQL DBs: Redis, ElasticSearch, Memcached
Virtualization: KVM, VMWare
Cloud: AWS, Azure, DigitalOcean, CloudStack, some GCP
Container Orchestration: Nomad, ECS, Kubernetes
Text Editor: Vim, VS Code
Source Control: Git, SVN, CVS
5⁄2016 - Present: Principal Engineer, Graymeta, Inc.
- Built a SaaS version of our core platform integrated with K8S. Integrated Stripe as the payment processor.
- Designed and implemented the backend system for ingesting the 2018 Royal Wedding video stream, processing the feed through human moderated facial recognition (powered by AWS Rekognition) before being distributed by CloudFront to an audience of 1M+ worldwide live (2 minute delayed feed) users. This was known as the Sky News Who’s Who app. Flew to London to provide boots on the ground support for the day of the event.
- Authored Terraform modules for infrastructure management. Authored and published a module to the Terraform registry for enterprise customers to deploy our product inside their AWS environment.
- Built a new CI pipeline. Testing and deployment automation with Docker, Jenkins, and Puppet
- Managed deployments across various IaaS providers: AWS, Azure, SoftLayer, Oracle Cloud
- Built CloudFormation templates that included RDS, ElasticSearch, Application Load Balancers, Autoscaling Groups, CloudWatch logs, ECS clusters, EFS filesystems, etc. to automate deployment of our application via the AWS Marketplace.
- Designed, rebuilt, refactored, and debugged our product’s Golang codebase to add features, operationalize existing code, and refactor/improve the codebase.
- Refactored our back end batch processing system to so it could run inside containers under multiple container orchestrators on multi-tenant container clusters leading to an overall reduction in spend on infrastructure.
11⁄2015 - 5⁄2016: Senior Systems Administrator, Raytheon at NASA Jet Propulsion Laboratory
- Linux systems administration in a 1400+ node, heterogeneous environment.
10⁄2013 - 11⁄2015: Software Engineer, Limelight Networks
- Refactored internal virtualization management control plane.
- Modernized and maintained asset management application (forked and heavily customized version of Rackmonkey). Built a test suite, then refactored backend code using test suite to ensure backwards compatability.
- Helped to monitor a large scale network with thousands of nodes spread across 50+ data centers around the world. Completed migration off of a 2001-era monitoring system.
- Maintained and refactored legacy code/systems
- Developed new self-service portals for infrastructure management
- 100% remote employee
4⁄2012 - 9⁄2013: Systems Administrator, Blackstar Marketing, LLC.
- Deployed Puppet, Puppet Dashboard, and PuppetDB for configuration management to existing infrastructure.
- Deployed mcollective for orchestration. Wrote custom mcollective plugins to deal specifically with our needs.
- Deployed LDAP for centralized authentication.
- Deployed centralized logging via rsyslog and LogStash with Kibana as the web frontend
- Abstracted existing domain purchasing code. This allowed us to purchase domains from any registrar that exposes a domain purchasing API.
- Redesigned and redeployed the Nagios implementation. Nodes automatically added/removed themselves via Puppet.
- Deployed a Jenkins environment for continuous integration/builds. All builds output rpm packages as artifacts.
- Responsible for packaging all code into RPMs.
- Deployed Graphite + Tasseo + Bucky + Collectd for real time metrics collection and dashboards.
6⁄2011 - 4⁄2012: Sr. Systems Administrator, The Walt Disney Studios
- Deployed Puppet for configuration management to existing infrastructure.
- Helped to design and deploy a CloudStack and VMWare based private cloud.
- Designed and built a cloud based, auto-scaling, video transcoding application wrapped around ffmpeg to demonstrate how to leverage the capabilities of our private cloud. Web front end was built in PHP. Backend processes were built in Python. Utilized RabbitMQ as the messaging middleware.
09/2010 - 6⁄2011: Enterprise Systems Engineer, Specific Media, Inc.
- All aspects of 24⁄7 live site operations
- Managed Puppet infrastructure through major version upgrades. Refactored Puppet code to leverage new features.
- Tied together disparate pieces to build a fully automated bare-metal to production server build system using RequestTracker(RT) with the AssetTracker plugin as the system of record. DNS, DHCP, kickstart, and puppet tied into the system.
- Refactored and automated internal DNS infrastructure
11⁄2009 - 08/2010: Operations Engineer, Hangout Industries, Inc.
- All aspects of 24⁄7 live site operations
- Deployed Puppet for automated management of machine configurations.
- Refactored existing infrastructure; refactored all application configurations; added additional hardware into a mixed Linux/Windows environment.
- Performed configuration and testing of in-house .NET applications on a Linux/Apache/mod_mono stack to allow us to migrate existing applications from Windows servers to Linux servers resulting in a cost reduction and stability increase.
- Load tested infrastructure using a combination of off the shelf solutions(apache bench, jMeter, MySQL bench, etc.) and homemade applications. Compared different hardware, OS, and application configurations to meet anticipated demand requirements. Helped developers analyze slow query logs and refactor existing database structure for improved performance.
- Built a data warehousing system to load raw application log files into a MySQL database for our analytics team for in depth analysis. Developed a suite of business intelligence reports that ran on a nightly basis to feed information to management.
12⁄2006 - 11⁄2009: Software Build Engineer/Associate Site Operations Engineer, Disney Interactive Media Group
- Primary operations team contact for Pirates of the Caribbean Online(POTCO), an MMORPG.
- Build and release manager for all POTCO game services. Game client was built on Win32 and OSX platforms. Game server was built on RHEL platform.
- Retooled the build process for POTCO to make it easier to understand, manipulate, and train new publishers. This new process was then back-ported to Toontown, another MMORPG developed by our studio.
- Migrated the build process for POTCO into a continuous integration server.
- Authored the Windows client installers for POTCO and Toontown
- Day to day operations of the game infrastructure including server configuration, maintenance, monitoring, and backup of application log files and databases.
- Developed custom Sitescope and Cacti monitoring utilities.
- Developed a standardized suite of log collection and archival tools designed to be used cross-product, cross-deployment.
- Responsible for diving into hundreds of gigabytes of log files to study player in-game behavior and provide feedback and data to developers. Condensed the log files into a format that was easier to manage and provided this to the developers for their own manipulation.
- Developed automated daily reports based on parsing application log files and querying a bug reporting database to indicate overall product stability. This information helped the product developers to prioritize daily activities.
- Developed tools to aid our technical services/customer support teams to mine data when issues about individual customers came up.
12⁄2004 - 11⁄2006: Sort Product Development Engineer, Flash Products Group, Intel
- Developed C code to test the functionality of flash memory chips during wafer level sort.
- High volume data analysis and interpretation skills were employed to tune defect screens and monitor process variations to ensure that quality devices are delivered to the customer.
- Built and maintained data analysis tools, specifically SSAT (Sort Spatial Analysis Tool). This tool automated much of the day to day manual data extraction process in addition to providing a tool to spatially analyze data for wafer level trends and defects.
- Developed a set of Perl modules to do spatial analysis within an individual flash memory device based on the block architecture of the specific product. Helped to integrate these modules into a larger data analysis tool (TableTool).
- Repurposed an aging server to run large batch data analysis jobs. Built a web frontend for job submission and tracking.
- 2017 AWS Certified Solutions Architect - Associate
- 2013 Puppet Certified Professional
- 2000-2004: University of California, Davis, Bachelor’s of Science, Computer Engineering
- AWS: This is My Architecture - Real Time Celebrity Identification at the Royal Wedding Using Amazon Rekognition Video
- Hashiconf 2017 - “Backend Batch Processing With Nomad” Slides / Video
- PuppetCamp Los Angeles 2012 - “Integrating Cloudstack with Puppet” Slides / Video
- Cloudstack Collaboration Conference 2012 - “Running Puppet on CloudStack Instances” Slides / Video