DevOps Case Study: Trek10 & Off Grid Electric

  • Home
  • DevOps Case Study: Trek10 & Off Grid Electric

Powering the World with Faster Development on ECS

About Off Grid Electric

Off Grid Electric power the world with clean, transformative energy in the next decade. Operating in Tanzania & Rwanda today, they have created a distributed solar model, known locally as Zola, to provide electricity to each and every household in the off-grid world. They have been featured in prestigious media outlets including The Economist, Forbes, and the Financial Times and are backed by SolarCity and the Omidyar Network as well as global aid foundations. Learn more about Off Grid Electric at www.offgrid-electric.com.


The Challenge

Off Grid Electric relies on a large custom Ruby on Rails software suite to fuel their hyper growth, running their core payments platform and all internal operations. Their early AWS adoption included static EC2 configurations and a mostly manual deployment pipeline. This was creating critical strains on their growth, including:
  • A lack of redundancy and scalability
  • An unstable systems operations model that included many manual and error-prone updates and deployments
  • A development process that could not keep up with the speed of the business: release of critical features was often delayed by unrelated features bundled into a large release
  • A desire to adopt Docker but concerns around increasing complexity with DIY cluster management technologies

The Solution

Trek10 leveraged Docker & the EC2 Container Service to build Off Grid Electric environments that deliver a dynamic, highly flexible, and completely code-defined infrastructure. With a single command, developers can now deploy new feature-branched environments to an Auto Scaling ECS Cluster. Staging & production deployments are fully automated based on an immutable Docker container, and are also Auto Scaling to add redundancy and scalability.

"The value of moving from an every two-week release cycle to deploying 14+ times a week cannot be overstated enough. notes Dan Mayer, Director of Software Engineering at Off Grid Electric. "And, beyond isolating and speeding up feature development, the new DevOps driven deployment stacks enable new ways to ensure the success of major changes."

The foundation for ECS Auto Scaling is a serverless approach that not only scales the ECS services based on CPU load, but also scales the ECS cluster up and down based on the number of required tasks, making for a truly elastic cluster for Docker containers. The whole solution is completely automated and code-defined with heavy use of interrelated CloudFormation templates, and Dockerfiles solve the configuration management challenge and obviate the need for additional provisioning tools.

The Results

Critically and fundamentally, the branch-based and self-service dynamic development environment dramatically speeds up their software team’s ability to respond to business needs and roll out new features and drive the company’s rapid growth. Other key benefits include:
  • A safer way to roll out infrastructure changes across Off Grid’s development, staging and production environments
  • Smaller and faster deployments that no longer cause downtime
  • Automated environment builds now also create the database backend (including a production database snapshot)
  • QA teams and product managers can now test new code on a feature-by-feature basis
  • Auto Scaling in response to traffic providing greater capacity and infrastructure costs are anchored to usage
  • Surge response time is 2x faster, capacity is no longer teetering on the edge
  • Redundancy and load balancing mitigate the risk of hardware failures resulting in downtime
  • App Server crashing has been reduced by a factor of 6 and Off Grid expects to reduce occurrences to zero
  • Background jobs are now more reliable
  • Data processing no longer effects app performance (no longer dropping 600+ daily requests and dozens of payments)
  • Improved log access and system monitoring (Sumo Logic & DataDog implemented as part of Trek10 CloudOps)

Tools & Services Used

Docker
Docker Containers on AWS
Jenkins Deployment Pipeline
jenkins.io
Amazon EC2 Container Service (ECS)
aws.amazon.com/ecs
Amazon EC2 Container Registry (ECR)
aws.amazon.com/ecr
Serverless Deployment API
AWS Lambda & Amazon API Gateway
Serverless ECS Auto Scaling Solution
AWS Lambda, Amazon CloudWatch, & Amazon Simple Notification System (SNS)
AWS CloudFormation
With Troposphere for Template Creation
Amazon Relational Database Service (RDS)
aws.amazon.com/rds
SumoLogic
For Log Aggregation
DataDog
For System Monitoring
Cohesive Networks VNS3
Virtual Networking Appliance