DEVOPS; A TESTER’S BEST FRIEND

DEVOPS; A TESTER’S BEST FRIEND

DEVOPS; A TESTER’S BEST FRIEND TOM JANIK – AMERICAN FAMILY INSURANCE DEVOPS MANAGER – RELEASE MANAGEMENT KEN ROBERTSON – AMERICAN FAMILY INSURANCE SQA TEST MANAGER – SPECIALIZED TESTING OBJECTIVE OF SESSION: WALK THROUGH THE AMERICAN FAMILY JOURNEY TO BECOMING A COMPANY THAT OPERATES IN A CULTURE BASED IN DEVOPS

IN THE BEGINNING: RELEASE FLOW DEVELOPMENT LEG DEV INT SIT LEG DEV INT QA RELEASE LEG DEV INT QA PERF PROD Deploys done by development teams, could happen at any time No stability requirements, no triage of deployment / environment issues 4 year development cycle using Amgile SDLC (selective Agile) Development “Leg” (flow), SIT “Leg”, 2 month effort in Release “Leg”

IN THE BEGINNING: SIT TESTING LOCAL/DEV Development and Unit Testing DEV/INT QA Functional Testing Test Data Creation TDC Elevation 1 SIT Elevation 2 TDC Elevation n Manual testing Testing led by development teams, lacked integration coordination 2 year SIT cycle with multiple code elevations No formalized automation or performance testing No shared test data, test scenarios, or documented financial testing Limited communication between development and testing – throw it over the wall mentality

FORMING: THE DEVOPS REVOLUTION DEV DEVELOPMENT LEG 1 INT QA DEV DEVELOPMENT LEG 2 INT QA DEV DEVELOPMENT LEG 3 INT QA RELEASE LEG DEV INT QA PERF PROD D: Release C oordination, Automation, Performance, and Manual Testing Teams O: C hange Management, Build and Deploy, Software C onfiguration Management, Infrastructure C oordination Teams Specific deployment windows, DENT environments, Floating QA environment Branching and merging strategies begin, targeted deploys to PERF (testing left) 3 parallel development (“Legs”) streams, Monthly maintenance release “Leg” Press ESC to c ontinue

FORMING: SIT TESTING CYCLES LOCAL/DEV Development and Unit Testing DEV/INT/DENT Functional Testing INT AIT (Application Integration Testing) Phase 1 QA HIST Phase 2 Phase 3 Test Data Creation QA SIT (System Integration Testing) Introduction of DENT and QA HIST tiers Cycle 1 Cycle 2 Cycle n. Point to point integration testing coordinated by development teams Application integration coordination being introduced 6 month SIT duration – 3 months financial infrastructure testing and 3 months for financials Documented financial testing in collaboration with controllers

ACTIVITY: DEVOPS WHAT’S IMPORTANT TO YOU? Using the index card in front of you write down 3 things that are most important to you about DevOps. (1:00 Minute) Gather in groups of #-# at one of the posters on the wall. (1:00 Minute) Come up with something common between everyone in the group and write that on top of the poster as your group name. (1:00 Minute) Come up with one common thing from everyone’s index cards and draw (no words) that one common thing on the rest on the space on the poster. (1:00 Minute)

STORMING: A CULTURE CHANGE TO DEVOPS DevOps is the practice of operations and development engineers participating together in the entire service lifecycle: from design, through the development process, and then into production support Specifically DevOps is characterized by: Operations staff making use many of the same techniques as developers for their systems work Developers incorporating more operational concerns in their planning and coding (designing for operationalization) DevOps can be a team, a department, an effort, a movement, but most importantly it is a culture!

KEY TENETS OF DEVOPS Theory of Constraints (“The Goal”) The 3 Ways (“The Phoenix Project”): Systems thinking Amplified feedback loops Continuous experimentation and learning Plan Opera Archit te FEEDBAC ect K Releas e Dev Test Lean principles of manufacturing (“TPS: Toyota Production System”) People, then process, then tools (Men, Methods, Machines, Measurements) Collaborative development (Scrum, Agile, Kanban) Environment provisioning, quicker access to new code for testing Continuous deployment, integration, testing, delivery, feedback, and improvement

THE NEW DEVOPS CULTURE Quality Assurance Release Automation Release Management DevOps Data Team Services Team Environment Management Application Reliability

Value Stream Mapping Planning (Enterprise) Agile Project Tracking Small Batches Increased Frequency Speed to Defect Resolution Standardization Application Instrumentation Loosely Coupled Architectures Design for Operations Incl. Graceful Degradation Lean Principles Micro services Branching by Abstraction /Toggling The Automated Delivery Pipeline (DevOps) Continuous Integration Environment Management Deployment Orchestration Automated Testing Removal of Process Debt Build Quality Into the Process Architecture, Design & Development Developer Commit Removal of Technical Debt Minimal Viable Product Version Build Automation Basic Test (incl. static code quality, security, performance, and code coverage) Package Deploy to Software Asset Repository Governance Checks Feedback Mechanism Infrastructure Integrations Trust Baselined / Modelled Environments Test Data Propagation Automatic Configuration Infrastructure-asCode Automation On-Premises/ Cloud/Hybrid Virtualization Loosely Coupled Architectures Dev in Ops Collaboration / Break Down Silos Ops in Dev Collective Ownership Automated Deployment Development Integration Acceptance Performance Production On-Demand Environment Provisioning Layered Testing (GUI, API, Backend) Full System Testing (Functional, UX Verification, Regression, Security, Performance, Compatibility, Acceptance) Environment Sanity Tests Application Resiliency Testing (Chaos Monkey, etc.) Non-Functional Sanity Tests Infrastructure Resiliency Testing Production Deployment Deployment Phased Rollout Rollback Plans APM & Analytics Communication Feedback Mechanisms Common Goals based on Customer Satisfaction Culture (Foundational) Continuous Experimentation Blameless Postmortem Never Done Dashboards/ Scorecards Trends Reports Notifications ALDM Monitor/Measure Everything Continuous Process Improvement Common Metrics Measuring Quality Measuring Efficiency Visible Status Continuous Improvement Openness & Visibility

Value Stream Mapping Planning (Enterprise) Agile Project Tracking Small Batches Increased Frequency Speed to Defect Resolution Standardization Application Instrumentation Loosely Coupled Architectures Design for Operations Incl. Graceful Degradation Lean Principles Micro services Branching by Abstraction /Toggling The Automated Delivery Pipeline (DevOps) Continuous Integration Environment Management Deployment Orchestration Automated Testing Removal of Process Debt Build Quality Into the Process Architecture, Design & Development Developer Commit Removal of Technical Debt Minimal Viable Product Version Build Automation Basic Test (incl. static code quality, security, performance, and code coverage) Package Deploy to Software Asset Repository Governance Checks Feedback Mechanism Infrastructure Integrations Trust Baselined / Modelled Environments Test Data Propagation Automatic Configuration Infrastructure-asCode Automation On-Premises/ Cloud/Hybrid Virtualization Loosely Coupled Architectures Dev in Ops Collaboration / Break Down Silos Ops in Dev Collective Ownership Automated Deployment Development Integration Acceptance Performance Production On-Demand Environment Provisioning Layered Testing (GUI, API, Backend) Full System Testing (Functional, UX Verification, Regression, Security, Performance, Compatibility, Acceptance) Environment Sanity Tests Application Resiliency Testing (Chaos Monkey, etc.) Non-Functional Sanity Tests Infrastructure Resiliency Testing Production Deployment Deployment Phased Rollout Rollback Plans APM & Analytics Communication Feedback Mechanisms Common Goals based on Customer Satisfaction Culture (Foundational) Continuous Experimentation Blameless Postmortem Never Done Dashboards/ Scorecards Trends Reports Notifications ALDM Monitor/Measure Everything Continuous Process Improvement Common Metrics Measuring Quality Measuring Efficiency Visible Status Continuous Improvement Openness & Visibility

NORMING: WHO DOES WHAT? The Automated Delivery Pipeline (DevOps) Continuous Integration Environment Management Deployment Orchestration Automated Testing Version Build Automation Basic Test (incl. static code quality, security, performance, and code coverage) Package Deploy to Software Asset Repository Governance Checks Feedback Mechanism Infrastructure Integrations Baselined / Modelled Environments Test Data Propagation Automatic Configuration Infrastructure-asCode Automation On-Premises/ Cloud/Hybrid Virtualization Loosely Coupled Architectures RELEASE ENVIRONMENT RELEASE Automated Deployment Development Integration Acceptance Performance Production On-Demand Environment Provisioning Layered Testing (GUI, API, Backend) Full System Testing (Functional, UX Verification, Regression, Security, Performance, Compatibility, Acceptance) Environment Sanity Tests Application Resiliency Testing (Chaos Monkey, etc.) Non-Functional Sanity Tests Infrastructure Resiliency Testing Production Deployment Deployment Phased Rollout Rollback Plans APM & Analytics Communication Feedback Mechanisms APPLICATION QUALITY ASSURANCE Dashboards/ Scorecards Trends Reports Notifications ALDM Monitor/Measure Everything Continuous Process Improvement Common Metrics Measuring Quality Measuring Efficiency Visible Status

NORMING DEV INT QA DEV INT QA DEV INT QA DEV INT QA Release management, Better defined schedules 2 week release activity cycle, Bi-monthly maintenance releases Overlapping release cycles, Same branching strategy as development legs

NORMING: QUARTERLY RELEASES 6 Months Summer CTP Code Freeze Performance Testing Development and Functional Testing Prod P2 SIT Testing Release Data Creation DEV m INT QA Functional Code Freeze Final Code Freeze 2 2 Weeks 6 Months

NORMING: FULL RELEASE SCHEDULE DEVELOPMENT LEG 1 6 Months Summer Summer Winter DEVELOPMENT LEG 2 Spring Fall Spring RELEASE LEG m m m m m m M m 2 m m m m M m m m m m M m m m m m M

NORMING: TESTING LOCAL/DEV DEV/INT/DENT INT Development and Unit Testing Functional Testing Functional Integration Testing Phase 1 QA HIST Phase 2 Phase 3 TDC SIT QA Cycle 1 Cycle 2 Cycle 3 Program level test lead for coordination of functional and integration testing Phased / cycle approach to functional integration and SIT testing 10 week SIT duration with data creation critical path constraint removed Introduction of forced month ends for financial testing

TESTING BENFITS FROM Continuous integration builds DEVOPS Gated deployments based on testing results Automated unit testing Automated smoke testing Automatic rollback for failed deploys Environment stability Quicker, higher quality code delivery Ability to feature toggle and/or deploy dormant code Automated deployment notification and release notes Better merging strategies ease regression testing requirements Exposure into development process

NORMING: TOOLS Maven for configuration GIT for code management CA Harvest for build and deploy JUnit / GUnit for continuous integration testing Jenkins for automated deployments CA Dev Test (formerly Lisa) for automated testing Word for requirements (moving toward TFS) Excel for testing plan (moving toward TFS) Excel release plans Puppet for IaaS CA Service Desk for change management and configuration item repository Artifactory for Enterprise Data Management Solution (EDMS)

QUESTIONS? THANK YOU TOOL APPENDIX FOLLOWS

APPENDIX A: TOOL RESEARCH Tool AccuRev (Micro Focus Software) Agile Software Factory (Grid Dynamics) Amazon ECS Ansible (Red Hat) Apache Ant Name Ansible Artifactory Asgard AWS Awsbox Category Configuration Management Application Release Automation Container Management Configuration Management, Application Release Automation Build Management Trial No No No Yes Yes Description An agent less orchestration engine that can automate application and systems. JFrog’s open source binary repository management product with support for Apache Maven artifacts. A netflix web interface tool for AWS Cloud Management and application deployment. A cloud computing platform offering ready to use web services which are loosely coupled. A featherweight DYI PaaS on top of awscloud EC2 for deploying node apps. http://www.devopsbookmarks.com/ https://xebialabs.com/periodic-table-of-devops-tools/ Site borland.com griddynamics.com aws.amazon.com ansible.com ant.apache.org