Testing in the Digital Age: DevOps, ChatOps and Automation to support Testing презентация

Helping clients transform their testing through
INNOVATION, COACHING and LEADERSHIP
Our CLIENTS
Want to be agile rather than follow Agile dogma
Have a pragmatic approach and are focused on delivery
Want a solution that fits, not a badly fitting suit.
Intelligent Definition and Assurance
Slide

Agenda
Digital Transformation
Digital is a Business Initiative
New Model Testing
Shift-Left
Collaborative Requirements
Continuous Delivery and DevOps
The Tools Landscape
ChatOps
Test Automation – Ambition and Reality
Digital Testing Strategy
Surviving Continuous Delivery and DevOps.
Intelligent Definition and Assurance
Slide

Introductions
Say hello to your team
Give your team a name (nameyourapp)
Please try to login to our Slack site:
https://chatopsdemo.slack .com
Username: [e-mail]
Password: p4ssw0rd
My card has the number xx to use
What do you want to talk about?
Intelligent Definition and Assurance
Slide

This class is about thinking for yourself
Pressure on testers – greater than ever
The IT industry is responding to Digital – a business initiative with new approaches
Shift-left, DevOps, continuous, pervasive automation, experimentation, analytics
These approaches are not standardised, they are changing and new tools appear every week
I can’t give you a ‘formula’ – but I can give you some ideas and thinking tools.
Intelligent Definition and Assurance
Slide

Digital Transformation

Revolution!
Lots of hype around “Digital” bombarding business and IT
Digital Transformation programmes are affecting business across all industry and government sectors
There is no doubt that it also affects people in their daily lives.
Intelligent Definition and Assurance

Scope
Digital includes traditional IT but includes:
Mobile anything
The Internet of Things
Autonomous vehicles
Our home, workplace, public and private spaces
Robots (physical)
Bots (software)
Artificial Intelligence, Machine Learning, Deep Learning (not limited to corporates)
All our lives will be affected.
Intelligent Definition and Assurance

Digital Transformation
Internet users:
2005: 1bn, 2010: 2bn, 2014: 3bn
Affordable technological advances:
Small devices – huge range; local/power networking
AI finally delivers in the form of Machine Learning
Virtual and Augmented Reality-based systems
Robotics, drone technology and 3D printing
Almost all businesses have committed to these technologies…
… and they are calling it Digital Transformation.
Intelligent Definition and Assurance

Unprecedented project ambition
Digital visionaries are promise a lot:
No human intervention in your systems (Autonomous Business Models)
Marketing messages created, sent, followed up and changed almost instantly
Full range of data from the smallest locale to global in all media formats at your disposal
Autonomous drones, trucks and cars can transport products, materials and people.
Physical products needn’t be ordered, held in stock and delivered at all – 3D printing removes constraints
Etc.
Intelligent Definition and Assurance

The most complex systems ever?
Space Shuttle (2.5m parts) used to be the most complex machine ever built by man
Nimitz class supercarrier has a billion parts
Thousands of interconnected systems
Crew of 5-6,000 people
Could be compared to an average town - afloat
Compare with a “Smart City”.
Intelligent Definition and Assurance

A Smart City is…
“… an urban development vision to integrate multiple information and communication technology (ICT) and IoT solutions in a secure fashion to manage a city’s assets – the city’s assets include, but are not limited to, local departments' information systems, schools, libraries, transportation systems, hospitals, power plants, water supply networks, waste management, law enforcement, and other community services.” (Wikipedia)
Intelligent Definition and Assurance

Smart Cities: the largest systems ever built?
Nodes and endpoints: from a million to billions
Bigger, more complex than anything before
Connected citizens and many of the systems:
Move in the realm of the city and beyond
Interact in unpredictable ways
Citizens are not hand-picked like the military; crooks, spies and terrorists can usually come and go as they please.
Intelligent Definition and Assurance

Systems of Systems and Ecosystems
Unlike ships, smart cities are very vulnerable to attack
Every human carries a mobile system – a phone
Every car, bus and truck connected - some driverless
Every trash can, streetlight, office building, power point, network access point is a Machine to Machine (M2M) component of a Digital Ecosystem
“A Digital Ecosystem is a distributed, adaptive, open socio-technical system with properties of self-organisation, scalability and sustainability inspired from natural ecosystems”.
Intelligent Definition and Assurance
Slide

Systems of Every Scale
Scale of Digital ranges from the trivial to the largest systems ever attempted
Home automation product (10-30 components)
Factory automation, monitoring and management (several thousand components)
Smart City (probably millions)
“A Digital Ecosystem is a distributed, adaptive, open socio-technical system with properties of self-organisation, scalability and sustainability inspired from natural ecosystems”.
Intelligent Definition and Assurance

Systems with Social Impact
Digital systems will have a social impact on all citizens who encounter them
Huge consequences as systems become more integrated with the fabric of society
Systems already monitor our every move, our buying, browsing and social activities
Bots push suggestions of what to buy, where to shop, who to meet, when to pay bills to us minute by minute.
Intelligent Definition and Assurance

Law enforcement
E.g. CCTV monitors traffic, people and asset movement and our behaviours too
Goal: Prevent crime by identifying suspicious behaviour and controlling the movement of law enforcement agents to places of high risk
But these systems have the potential to infringe our civil liberties
Legal frameworks are behind the technology.
Intelligent Definition and Assurance

Ecosystems of Ecosystems
The span of Digital covers commerce, agriculture, health, government, the media in its all forms, the military …
Digital affects all industries
A systems view does not do it justice
It might be more appropriate to treat Digital systems as…
“Ecosystems within Ecosystems”.
Intelligent Definition and Assurance

Digital is a Business Initiative
Agile (an IT initiative) has taken 15 years to get half way
Digital?

Digital and Marketing
Digital is the buzz-phrase of the moment
Social, Mobile, Analytics, Cloud (SMAC)
Consumerization of IT
Buzzword bingo*
Agile laggards (gov, large companies) playing catch-up?
Marketers in charge: Chief Digital Officers etc.
S/W development at the pace of marketing.
* https://zoomph.com/blog/top-20-digital-marketing-buzzwords/
Intelligent Definition and Assurance
Slide

Frequent/regular s/w delivery is critical
Mobile users expect apps to change almost daily
New features, offers, opportunities all the time
Users use the best apps to get what they want
Not necessarily the best supplier
Businesses are in an APPS RACE
Speed of delivery is...
Partly about pro-action
Partly about survival.
Intelligent Definition and Assurance
Slide

Automation (not just test) is critical
Business wants IT responsiveness (true agility)
Not necessarily 100s of releases/day
Rapid, regular turnaround from ideas to software delivery
With continuous integration/deployment, DevOps, developers can now promise Continuous Delivery
Testers need to provide Continuous Assurance
"Automation through the (shortened) lifecycle"
But effective test automation success is still hard to achieve.
Intelligent Definition and Assurance
Slide

"Automation is the future!"
Heard that before?
What exactly is possible and impossible with automation, right here, right now?
Are Continuous Delivery and DevOps the route to success?
Could testing be the bottleneck that prevents success?
How do testers work in high-paced, automation-dominated environments?
In the rest of this tutorial, I want to explore these and other questions.
Intelligent Definition and Assurance
Slide

The old ways won't work in the future
We need a New Model of Testing (free from logistics)

Forget Logistics
(for the time being)
Document or not?
Automated or manual?
Agile v waterfall?
This business or that business?
This technology v that technology?

ALL Testing is Exploratory
We explore sources of knowledge ...
... to build test models ...
... that inform our testing.

Models Pub Quiz
How many models can YOU name?
Twitter: @paul_gerrard
Paul Gerrard
[e-mail]
gerrardconsulting.com
Intelligent Definition and Assurance
Slide

Models are innate, essential, human
Intelligent Definition and Assurance
Slide

Judgement, exploring and testing
Testing
(the system)
Our model(s) are adequate
Our model(s) are not adequate
Exploring (sources)
Judgement
Creates test
models
Uses test
models
We explore sources of knowledge to build test models that inform our testing
We don't yet know what the system should do.
We can't test yet
(learning what it should do)
We believe that we know what the system should do.
We can (automate) test.
(learning what it does do)

Judgement, exploring and testing
Testing
(the system)
Our model(s) are adequate
Our model(s) are not adequate
Exploring (sources)
Judgement
Creates test
models
Uses test
models
We explore sources of knowledge to build test models that inform our testing
BTW – Do Developers explore the same way? I think so.
Intelligent Definition and Assurance
Slide

Exploration process
Exploration
Test
Models
Enquiring
Challenging
Sources:
People, documents,
experience, system under test
Modelling
Test Models:
Can be documented
or mental models
Predicting
System under test
Intelligent Definition and Assurance
Slide

Testing process
Testing
System Under Test
Refining
Informing
Applying
Interpreting
Test
Models
Revise the System
Logging
Revising
More exploring
Reporting
Intelligent Definition and Assurance
Slide

New Model Testing
29 page paper: http://dev.sp.qa/download/newModel
Intelligent Definition and Assurance
Slide

Some Consequences
There are others

Relation to
TDD and BDD?
TDD is not really testing
BDD is modelling using stories

Test automation from a different perspective
Automation efforts fail too often
Automation uses different test models

Capabilities
Enquiring, Modelling, Predicting, Challenging
Informing, Applying, Interpreting, Refining
Reporting and Logging

Analysis, enquiry and elicitation
Modelling
Creation of custom models, using heuristics, guesses, brainstorming, ideation, creative thinking
Custom test design techniques
Comparison of models, value, advantages, disadvantages, compromises
Identification, validation and use of oracles
Predicate logic and proof
Hypothesis and inference
Socratic method
Rapid Review and Inspection techniques
Test case design
Test models and the meaning of coverage
Testing as controlled experiment
Observation, Note taking, recording
A very different skillset
Basic data analysis and statistics
Decision-making with incomplete data
Computer forensics
Fault tree analysis
Failure diagnosis
Bug advocacy, triage processes and negotiation
Meaningful software and test metrics
Visual presentation of data
Reporting and presentation skills
Understanding stakeholders
Test analytics
Risk management, risk-based testing and decision-making
Critical Thinking
Interpersonal skills
Dealing with uncertainty/fallibility
Intelligent Definition and Assurance
Slide

Testing Career Development
(speculative)
Foundations
Technical
Management
Strategic
Test Strategy
Project Intelligence
Test Assurance
Exploration
Forensics
Interpretation
Scripting/
Programming
Test Automation
Technical (Excel, SQL, OS utils etc)
Stakeholder management
Analytics & visualisation
Managing uncertainty
Critical Thinking
ISTQB, TMMi etc...
Supplier Management
Test Process Management
Methodology
Intelligent Definition and Assurance
Slide

Shift-Left

Shift-left
Teams redistribute responsibility for testing and collaborate more
Shift-Left can mean:
Developers take ownership for their testing
Testers get involved earlier, challenge requirements, share examples with users and devs
No test team and no testers
There is no 'one true way' of course.
Intelligent Definition and Assurance
Slide

Shift-Left is not new
Shift-Left is mostly about bringing the thinking about testing earlier in the process
So, all we do is get involved earlier and ask awkward questions?
Is it really as simple as that? Well, not quite.
Paul Herzlich's W-Model 1993
Intelligent Definition and Assurance
Slide

Shift-Left is Not New
“Test early, test often”: a mantra for 25 years
The underlying principle:
Sources of Knowledge should be challenged or tested
In a staged project, this might involve formal reviews
In an Agile project, ‘three Amigos’ tester, developer and user/BA suggest scenarios (or examples) that challenge requirements before code is written
Shift-Left is really “thinking about testing earlier”.
Intelligent Definition and Assurance
Slide

What is Driving Shift-Left?
Several changes in the market are at play
It started around some years ago but has taken till recently to 'click' in the marketplace
Behaviour Driven Development (BDD)
Continuous Delivery (CD)
DevOps
Frequent experimentation in production systems
SMAC, or Social, Mobile, Analytics and Cloud
Intelligent Definition and Assurance
Slide

Shift-Left – it’s all about feedback
Testers provide feedback – whenever possible
Get involved early – as early as you can
Challenge through example
Software development is knowledge acquisition
Knowledge is gathered throughout the project and evolves over time
The goal is to assure this knowledge and to ensure it is trusted before it is frozen in code
Shift-Left is not a threat; it is an opportunity to make a bigger, better contribution.
Intelligent Definition and Assurance
Slide

Discussion: How often do apps change?
How often are these apps released?
Facebook
LinkedIn
eBay
Amazon
NatWest bank
Lloyds Bank
Nationwide Bank
etsy
Google search
your app
Stats from appannie.com
Release frequency
2 weeks (mostly)
7-14 days
Monthly
2-4 weeks
4-6 weeks
1-3 months
2-8 weeks
2 weeks
4-6 weeks
Intelligent Definition and Assurance
Slide

Collaborative Requirements

Collaborative requirements
Testers can make a huge contribution to the requirements phase
In most contexts, user or business stories are used to capture them
For the purpose of the next few slides:
'Requirements' could mean documented or undocumented sources of knowledge
A 'story' summarises a system feature and illustrates it use with scenarios.
I‘ll use the Gherkin (Cucumber) format to illustrate its use (heard of Cucumber?)
Intelligent Definition and Assurance
Slide

Using Stories to Articulate/Validate
Requirements
Chapter 4 of the Business Story Pocketbook
Download here:
https://tkbase.com/resources/viewResource/19

Intelligent Definition and Assurance
Slide

Story Header
Feature: ship orders
As a orders clerk
I want to acknowledge and ship the order
So that we fulfil a book order
Scenario: ship a single book from stock
Given I select a valid order
And the ordered book is in stock
When I choose ‘acknowledge and ship’
Then order status is changed to ‘shipped’
And an address label is printed
Structured stories (other variations exist)
Key word
Story text
Each Story has multiple Scenarios
Scenarios can be data driven
Intelligent Definition and Assurance
Slide

Anatomy of a business story header
Note that roles can sometimes vary, but it is often better to reference ‘personas’ that have multiple roles.
Personas could be “18 year old male gamer” or “65 year old female retired nursery school teacher” for example.
The story brings together these aspects so we can view the feature from different viewpoint to explore it
Intelligent Definition and Assurance
Slide

Anatomy of a scenario
The parallel with test cases is obvious:
given=precondition(s), when=steps, then=outcome(s)
A scenario maps directly to a test case – but we haven’t used the word test yet.
If I do – stop me.
Intelligent Definition and Assurance
Slide

Stories may have many scenarios
Feature: Ship an Order
In order to fulfil a book order
As a orders clerk
I want to acknowledge and ship the order
Scenario: ship a single book from stock
Given I select a valid order
And the ordered book is in stock
When I choose ‘acknowledge and ship’
Then order status is changed to ‘shipped’
And an address label is printed
Scenario: advise a book is out of stock
Given I select a valid order
And the ordered book is out of stock
When I choose ‘message the purchaser’
Then Enter message to purchaser advising the order status
And an email is sent to the purchasers email address
Scenario: advise an item is discontinued
Given I select a valid order
And the ordered book is discontinued
……
Etc. etc.
Intelligent Definition and Assurance
Slide

We use examples, scenarios to challenge requirements here
New Model Testing
Intelligent Definition and Assurance
Slide
Scenarios are examples of behaviour

Exercise: what needs definition?
Jot down the terms to define
A customer order will have a unique order reference a customer identifier, an order date and a required-by date.
Each order generates multiple order items each of which has a unique ID, a product identifier, an order quantity, a unit price, a unit weight, a promised delivery date and required-by date.
The total order price, earliest and latest delivery date, total weight will be calculated from the items on the order.
Do you see any ambiguities?
Intelligent Definition and Assurance
Slide

Exercise: what needs definition?
A customer order will have a unique order reference a customer identifier, an order date and a required-by date.
Each order generates multiple order items each of which has a unique ID, a product identifier, an order quantity, a unit price, a unit weight, a promised delivery date and required-by date.
The total order price, earliest and latest delivery date, total weight will be calculated from the items on the order.
Intelligent Definition and Assurance
Slide

What other questions might you ask?
'customer order will have…' means what?
unique order reference’ unique in what context?
Order date’: date placed, or date recorded or other date to be defined?
Each order generates’ – means what?
Can an order have zero items? Is there a limit to how many items?
Can order quantity be negative, non-zero, is it a decimal or integer?
Must dates be weekdays or can they be weekends? Are there any date rules to apply?
What currency are prices in?
What units is weight measured in?
Is deliver-date the last required-by date or is it calculated some other way?
Other definitions? ‘Customer’, ‘required-by’, ‘weight’, ‘unit’, `promised’, ‘total (price)’, ‘product’
Where do these pieces of data come from? Where are they stored? Where do they ‘go to?
Intelligent Definition and Assurance
Slide

Typical questions to ask of a requirement
What do the nouns and verbs actually mean?
What features are being described here?
What outcomes do these features provide?
What scenarios (normal, extreme, edge and exceptional) should they cope with?
Are all outcomes predictable from the text?
Are all outcomes unambiguous?
Is anything (definitions, features, scenarios or outcomes) missing?
Definition
Feature
Outcome
Scenarios
Prediction
Missing
Ambiguity
Intelligent Definition and Assurance
Slide

D e F O S P A M
Definitions
Features
Outcomes
Scenarios
Prediction
Ambiguity
Missing
Identify the terms and define
One story per feature
One scenario per outcome
One scenario per scenario
Can’t predict? Scenario + guess outcome
Two stories with conflicts
Add a story as a suggestion
See “Business Story Method”, page 41
Intelligent Definition and Assurance
Slide

Exercise: D e F O S P A M
Use DeFOSPAM to find problems with this requirement
Create some examples or scenarios to illustrate problems.
The calculator will accept three inputs: a number A, an operator O and another number, B.
The calculator will validate the numbers A and B as valid in the range -1000,000,000 to +1000,000,000
The operator may be one of the following: "+" (plus), "-" (minus), "*" (multiply) or "/" (divide)
The calculator will perform the calculation according to standard arithmetical rules
The calculator will print the result as a real number with up to 20 significant digits.
Intelligent Definition and Assurance
Slide

Did you find any anomalies?
Do you want to share them?

Continuous Delivery

From traditional delivery…
… to Continuous Delivery
Intelligent Definition and Assurance
Slide

The Deployment Pipeline
Automated Unit tests
Automated Acceptance Tests
Manual User Tests?
Intelligent Definition and Assurance
Slide

Introducing Continuous Delivery
Rigorously scoped features
If a feature can't be squeezed into the next release, split it into smaller features
Validated requirements – thru example
BDD/Three Amigos/Collaborative requirements
Gherkin given/when/then scenarios
Validate the requirements
But can also become automated tests.
Intelligent Definition and Assurance
Slide

Introducing Continuous Delivery 2
Test-Driven Development – all tests pass
Commits must work – broken builds or tests in CI are the teams top priority
Everything under change control
(Almost) Everything automated
Rapid/regular feedback requires rapid/regular reaction.
Intelligent Definition and Assurance
Slide

Continuous Delivery in pictures
http://continuousdelivery.com/2014/02/visualizations-of-continuous-delivery/

If it Hurts, Do it More (Often)
Tasks (like testing) provide feedback; regular rapid feedback means that problems can fixed quickly
If we do a hard activity more frequently, We get better at it, maybe enough to automate it.
Intelligent Definition and Assurance
Slide

OK, we’re going to implement CD
Boss: “We’re want to automate EVERYTHING!”
Break down the test activities you are involved in day to day
Which ones can be automated?
And which ones can’t?
Discuss in your group, make a list of each.
Intelligent Definition and Assurance
Slide

Exercise: what test activities can be automated and what cannot?
What can be automated?
What can’t be automated?
Intelligent Definition and Assurance
Slide

It’s not about automation,
it’s about trust
A Deployment Pipeline depends on a Trusted Requirements Pipeline

If a requirement drives today’s delivery it must be trusted
Continuous delivery is a hungry beast that eats requirements
Requirements must be trusted to be mature, complete and coherent enough to deliver the business value envisaged by stakeholders
TRUSTED, NOT PERFECT
Intelligent Definition and Assurance
Slide

Tests, Automation and Trust
There is debate around:
The meaning of checking and testing
The reliance we can place on testers, on checks and automation
In most environments we can't place all our faith in automated checks
We need to look more closely at our testing.
Intelligent Definition and Assurance
Slide

Four types of checks?
Checks that can be automated by developers as part of their component-level check-in and Continuous Integration processes
Checks that can be automated (typically by system testers) to exercise API-level, link or end-to-end transactions
Tests that can perform compatibility checks to demonstrate compatibility across browsers, operating systems, platforms
Tests that can only be performed by a human.
Intelligent Definition and Assurance
Slide

Typical questions and objections
When do we create the test automation?
There are too many system tests, how can we automate them all?
When we find bugs, there isn’t time to rerun our systems tests
…
Intelligent Definition and Assurance
Slide

Shifting tests left
User/acceptance tests
GUI/System Tests
API/Service tests
Unit Tests
Intelligent Definition and Assurance
Slide
Move manual user and system tests earlier and automate

New feature and regression testing
Intelligent Definition and Assurance
Slide
Test new code
Regression test
existing code
Tested once as new code, and many times regression tested
Automated tests

Development Patterns

Three development patterns
Structured
Agile
Continuous
Goal-Based
Hi-Process
Autonomous
Intelligent Definition and Assurance
Slide

Intelligent Definition and Assurance
Slide

Profiles of the three patterns
Intelligent Definition and Assurance
Slide

Not three patterns;
There are many
You have to work out your own hybrid approach that suits your organisation

DevOps
an overview

Exercise: How long does it take?
Stages between dev-done and Go Live e.g:
Integration
Installing
Testing
Staging/Pre-Prod
Go Live Go Live
Discuss in your team:
How long for each stage in your environment?
Which stage takes longest?
Why does it take so long?
Intelligent Definition and Assurance
Slide

Slow Release, Deployment and Infrastructure Change
We're interested in Continuous Delivery
We've shifted left
Testers get involved earlier
Our requirements are better
Devs write more tests and automate them
Continuous Integration is our safety net
We can deliver in days, but it still takes weeks to deploy – what's going on?
Intelligent Definition and Assurance
Slide

Deployment process 1
Does this sound familiar?
Integration – "nothing works"!
First time developers code is merged it's clear they haven't been talking to each other, or testing
Installing – "sometime, something somehow"
A flaky, manual installation on new machines... Doesn't work. Days or weeks to get it running.
Intelligent Definition and Assurance
Slide

Deployment process 2
Testing – "it could be so much better!"
Some functionality is available; but doesn't quite match the requirements – which are changing
Staging/Pre-Prod – "why are we doing this?"
Closer to production environment, but not exact; flaky – in different ways; "you have 48 hours"
Go live – "just one more test..."
Testers spend an afternoon 'just checking' before the users are let loose
Intelligent Definition and Assurance
Slide

Intersection of Dev, Ops and QA
Intelligent Definition and Assurance
Slide

Definition
"DevOps is a development methodology with a set of practices aimed at bridging the gap between Development and Operations, emphasizing communication and collaboration, continuous integration, quality assurance and delivery with automated deployment”, Wikipedia
Intelligent Definition and Assurance
Slide
* Based on a review of definitions of DevOps from 238 papers

The ‘endless DevOps loop’
Intelligent Definition and Assurance
Slide
Why is it a figure of eight?

What is DevOps?
The DevOps ‘movement’ (for want of a better label) is progressing rapidly
Like many other moves in IT
the speed of adoption moves faster than the definition of the approach itself
There is a lot of debate: “What is DevOps?”
Sounds familiar?
"What is DevOps", The Agile Admin, http://theagileadmin.com/what-is-devops/
Intelligent Definition and Assurance
Slide

Is DevOps about tools or people?
At one level, the goal of DevOps is to speed up the delivery pipeline using automation
But automation still requires governance
Automation cannot complete tasks without human intervention
A DevOps process is meaningless without considering the human factor
Tools do some heavy-lifting, but people make the process work – or fail.
Intelligent Definition and Assurance
Slide

So DevOps is just Dev and Ops working more closely?
No, it's not that either
The handoffs between automated processes often involve other processes – usually testing
Automated tests are created by Developers and Testers
The output of these tests must provide sufficient information for processes or people, to transition between stages in the pipeline
Testing provides the assurance that DevOps processes deliver successfully and reliably.
Intelligent Definition and Assurance
Slide

DevOps - CAMS
Culture
Develop Dev-Ops relationship, communications, collaboration, bring Dev closer to the customer
Automation
It's pervasive: builds, deployments, testing, monitoring, self-healing, system rollouts, configuration...
Metrics (Analytics)
Experiment, capture, learn, improve both during development and production
Sharing
Share ideas, and metrics; allow devs more control over production systems using code.
Intelligent Definition and Assurance
Slide
Nice description of DevOps and CAMS here:
http://www.slideshare.net/geekle/devops-5348895/19-So_Whats_All_This_Changebr

The Tools Landscape

Periodic table of DevOps tools
https://xebialabs.com/periodic-table-of-devops-tools/
If only the world of DevOps were so simple (and static)
Intelligent Definition and Assurance
Slide

Environments
DevOps tools landscape
Dev
Test
CI
Production
Deployment
Config/Provisioning
Release and Pipeline Orchestration
Build
DevTest
Integ. Test
Service/API
UI
E2E
Logging
Monitoring
Collaboration/ChatOps
SCM
Repository
Platform
PaaS
IaaS
Virtualization
Containers
Load/Perf
Security
BI/Analytics
Cust. Experience
Static Analysis
© 2016 Paul Gerrard, Gerrard Consulting

How many tools do you use?
NewRelic - Application monitoring - gives us the eyes on our app and how it's being used / performing
PaperTrail - Log file collector - brings in log files from various servers to one single place - great for systems running across multiple servers
OpsView - Monitoring and alerting tool which we use to bring together monitoring from various systems
Nagios - Used for monitoring and alerting
PagerDuty - Used to alert (SMS and Email and Phone) when a service craps out
Elastic Search, Log Stash and Kibana - Data analysis and monitoring and trending - super powerful for analyzing what our product is actually doing
Chef - Auto build and deploy technology to allow us to rapidly build and destroy environments (with Chef Kitchen and Knife)
Vagrant - Create Virtual Environments
Real Time Board - Virtual Whiteboard - amazingly useful
Pivotal Tracker - Agile tracking tool
Fiddler - Proxy web tool
Firebug - Proxy web tool
Zed Attack Proxy - Security testing tool
Burpsuite - Security Testing Tool
HipChat - Real Time IM communication tool
Slack - Real Time IM Communication tool
Jira and Confluence - bug tracking and wiki
CloudFormations - Creates templates for Amazon instances
"No doubt we have some more hiding away but that's a pretty good list."
23
tools
Intelligent Definition and Assurance
Slide

The
Tools Knowledge Base
https://tkbase.com

Some Ancient History

CAST Report 4th edition (1999)
Intelligent Definition and Assurance
Slide

Remember
WinRunner?
This is a sample
tools page from the
CAST Report
WinRunner had a
two page entry in
the report

CAST Report –tools hierarchy
Intelligent Definition and Assurance
Slide

How many tools are there anyway?
I'm researching tools for tkbase.com
2424 of which 686 are programming languages
1658 tools for DevOps, SDET & Testers
Tool types and features
https://tkbase.com/tools
My guess is there are at least 2000.
Intelligent Definition and Assurance
Slide

Intelligent Definition and Assurance
Slide
314 bloggers, 33,034 blogposts indexed and searchable (~40 new posts daily)
2,424 tools, indexed and searchable
Work in progress: tools tagged with 317 features

Platform
IaaS, PaaS self-service
Virtualization: prepared images that can be deployed to provide full SaaS – if required
Containerization – this is the challenge to virtualization...
Intelligent Definition and Assurance
Slide

Containerization
Hottest topic in the DataCentre right now
Virtual machines replicate much of the host machine and kernel
Containers reuse the host's kernel so are much lighter and faster to boot
One host can support hundreds, even thousands of containers
Some concerns and limitations especially in the area of security and large-scale deployments (watch this space)
Docker™ gets most attention, but there are alternatives.
Intelligent Definition and Assurance
Slide

Configuration/Provisioning
"Infrastructure as code" is an emerging concept
Define your environments as code in text files
Use tools to follow those instructions
Manage your configurations using SCM – just code
Automated provisioning tools use the configuration to create environments
Some tools do everything e.g. Chef
Some tools like Vagrant control the provisioners (tools like Ansible, Puppet and Chef again).
Intelligent Definition and Assurance
Slide

Repository management
Infrastructure as code means...
Environments created accurately and reliably
But these still take time
Repositories hold environment 'components we made earlier' to save even more time
When you have a trusted recipe for an environment or platform it can be stored, managed and retrieved from the repository.
Intelligent Definition and Assurance
Slide

Development
Static analysis tools scan code and look for problems and provide code metrics that indicate 'code smells'
Build tools perform the source code downloads, compilations and integration to create testable apps or sub-systems
DevTest tools are unit-test frameworks to test components, classes and methods at a low level
Integration tools run API based transactions to demonstrate integration across components.
Intelligent Definition and Assurance
Slide

Test
Service/API tests call web services/APIs directly
GUI tools are the traditional user-oriented tests that simulate realistic use-cases
End-to-end tests harness both API and UI tests to create end to end transactions
Load/Performance tools simulate system loads to explore behaviour and performance
Security tools range from penetration tools to instrumentation that can signal threats/break-ins.
Intelligent Definition and Assurance
Slide

Production
Logging tools can be configured to
Record execution statistics at virtually any level
Trigger alarms for selected exceptions
Monitoring tools range from the server, database and network to application and security
Customer experience (CX) tools provide analytics on user journeys and behaviour to support customer service and change-decisions
Business Intelligence/Analytics tools provide the analyses and management information based on logs, instrumentation and monitoring.
Intelligent Definition and Assurance
Slide

Release and pipeline
orchestration
These tools define the deployment and release pipeline, tasks and dependencies
They can initiate the automated tasks in sequence and provide a dashboard with progress
Used collaboratively, everyone can see where the release is at and what needs to be done to resolve problems
The data they capture on tasks, problems and durations mean they can highlight bottlenecks and opportunities for improvement.
Intelligent Definition and Assurance
Slide

Collaboration/ChatOps
Tools so far enable technical folk to define and implement environments, build, deploy and test
Collaboration tools provide a focus for distributed teams to communicate/collaborate
ChatOps is collaboration using tools that are integrated with DevOps
ChatOps has been called 'conversations put to work'
'bots' report directly into chat rooms; bots can be commanded directly through chat messages
'Conversation-driven development', online decision-making and commitment.
Intelligent Definition and Assurance
Slide

ChatOps example
> bot: twitter says "we are down"
< user: @bot shut up twitter
> bot: twitter silenced for 15 min, get busy fixing stuff fast!
> bot: Nagios alert on web301: CRITICAL, high CPU over 95%
< user: @bot nagios ack web301
< user: @bot graph me io,net on web301
> bot: @user Here's your graph: https://mygraph.example.net/web301?show=io,net
> other_user: looks like it's just high load. Let's add couple of nodes!
< user: @bot autoscale add 2 nodes to cluster-3
> bot: @user On it, your execution is 5636fac02aa8856cc3f102ec
check the progress at https://st2.example.net/history/5636fac02aa8856cc3f102ec
Intelligent Definition and Assurance
Slide

ChatOps

Exercise: Create a Delivery Pipeline from these activities (env/activity)
Intelligent Definition and Assurance
Slide

Exercise: ChatOps
Let's look at a simulation of a Deployment Pipeline and use ChatOps to manage it
http://chatopsdemo.slack.com credentials:
[e-mail] password is 'p4ssw0rd‘
Monitor the pipeline here:
http://dev.sp.qa/workshop/pipeline
Intelligent Definition and Assurance
Slide

Test Automation - Ambition and Reality

"Automation is the future!"
Heard that before?
What exactly is possible and impossible with automation, right here, right now?
Where are the Digital, DevOps and Continuous Delivery crazes leading us?
Where do testers fit?
How do testers work in high-paced, automation-dominated environments?
Let's look into the near future and discuss how we survive or thrive.
Intelligent Definition and Assurance
Slide

What Goes Wrong with
Test Execution Automation?

Tools are sensitive instruments
You don't need tools and tools won't help, if your software is unstable.
Intelligent Definition and Assurance
Slide

The anti-regression goal
What are you trying to do with automation?
Anti-regression is the primary goal
Detect unwanted change
Insurance that developers can make changes safely
How do you achieve this?
The automation model must align with the developer model
You have to collaborate early to achieve this.
Intelligent Definition and Assurance
Slide

Testing and automation – a modelling problem not a tools problem
Need to shift testing thinking to the left.
We model to improve requirements,
to eliminate mistakes in code and automation.

Brian Marick's Original Agile Testing Quadrant
Intelligent Definition and Assurance
Slide

Crispin & Gregory (Agile Testing version)
Intelligent Definition and Assurance
Slide

Intelligent Definition and Assurance
Slide

Intelligent Definition and Assurance
Slide

What does this tell you?

Intelligent Definition and Assurance
Slide

Intelligent Definition and Assurance
Slide

How do we get better at test automation?

New Model Testing
The Left
The Right
where the checking happens
Intelligent Definition and Assurance
Slide

Left and Right
Automation won’t work if the thinking to the left is not in place
Thinking on the left contributes to the definition of features and how you test them
If you aren’t involved on the left, you aren’t in a good position to test (with or without tools)
Models on the left are the blueprints for the tests & automation on the right
Your opportunity to flush out muddled, loose, incomplete, flaky requirements is on the left
Applying automation to muddled, loose, incomplete, flaky features is doomed before you start
The Left
The Right
Intelligent Definition and Assurance
Slide

Shift-Left (thinking, not people)
If you are not contributing on the left, you are doomed to be a victim of muddled, incomplete thinking
Challenge requirements thru example and improve them
Share your models!
Usage models: (test through the UI) must be meaningful to users
Service models: (services/APIs) meaningful to developers and align with business rules
In case of test/automation ‘difficulty’ – go to 1.
Intelligent Definition and Assurance
Slide

Your homework for today
Are tools just drones that can only execute regression testing?
Could tools find functional bugs in untested software?
If I can think of a test, can I automate the application of it?
Think of tools as extensions
of your mind, not robots.
This is where the tools play their part
Intelligent Definition and Assurance
Slide

Digital Testing Strategy
Agile Test Strategy – an Oxymoron?

(Test Strategy as)
Agile Interventions
I’m using Scrum/Sprint terminology, but you don’t have to of course
https://www.youtube.com/embed/Ed6YkYEkCRM

Interventions (government client example)
On the following slides, we highlight 8 interventions
Some are test phases, but some aren’t
Intelligent Definition and Assurance
Slide

Integration into
Existing Code base
Automated testing
New Code
8. User Test
7. System Test
Sprint 1
Developed Stories
Developed Stories
Developed Stories
Sprint 3
Sprint 2
Sprint Backlog
Sprint Backlog
Sprint Backlog
Story Challenge
Suggest ‘what-ifs’ to challenge new stories and define story headlines
Increasing Scope of Sys. Test and UAT
Increasing Scope of Integration, System and Users Testing
2. Story Definition
Introduce scenarios to enhance the Acceptance Criteria
Complete Tests after Final Sprint
Project Level Test Activities
(This diagram shows three sprints, but there could be more or fewer)
6. Integration Test
6. Integration Test
6. Integration Test
Intelligent Definition and Assurance
Slide

Integration into
Existing Code base
Automated testing
New Code
8. User Test
7. System Test
Sprint 1
Developed Stories
Developed Stories
Developed Stories
Sprint 3
Sprint 2
Sprint Backlog
Sprint Backlog
Sprint Backlog
Story Challenge
Suggest ‘what-ifs’ to challenge new stories and define story headlines
Increasing Scope of Sys. Test and UAT
Increasing Scope of Integration, System and Users Testing
2. Story Definition
Introduce scenarios to enhance the Acceptance Criteria
Complete Tests after Final Sprint
Project Level Test Activities
(This diagram shows three sprints, but there could be more or fewer)
6. Integration Test
6. Integration Test
6. Integration Test
Intelligent Definition and Assurance
Slide

Integration into
Existing Code base
Automated testing
New Code
8. User Test
7. System Test
Sprint 1
Developed Stories
Developed Stories
Developed Stories
Sprint 3
Sprint 2
Sprint Backlog
Sprint Backlog
Sprint Backlog
Story Challenge
Suggest ‘what-ifs’ to challenge new stories and define story headlines
Increasing Scope of Sys. Test and UAT
Increasing Scope of Integration, System and Users Testing
2. Story Definition
Introduce scenarios to enhance the Acceptance Criteria
Complete Tests after Final Sprint
Project Level Test Activities
(This diagram shows three sprints, but there could be more or fewer)
6. Integration Test
6. Integration Test
6. Integration Test
Intelligent Definition and Assurance
Slide

Integration into
Existing Code base
Automated testing
New Code
8. User Test
7. System Test
Sprint 1
Developed Stories
Developed Stories
Developed Stories
Sprint 3
Sprint 2
Sprint Backlog
Sprint Backlog
Sprint Backlog
Story Challenge
Suggest ‘what-ifs’ to challenge new stories and define story headlines
Increasing Scope of Int. Sys. and UAT
Increasing Scope of Integration, System and Users Testing
2. Story Definition
Introduce scenarios to enhance the Acceptance Criteria
Complete Tests after Final Sprint
Project Level Test Activities
(This diagram shows three sprints, but there could be more or fewer)
6. Integration Test
6. Integration Test
6. Integration Test
Intelligent Definition and Assurance
Slide

Integration into
Existing Code base
Automated testing
New Code
8. User Test
7. System Test
Sprint 1
Developed Stories
Developed Stories
Developed Stories
Sprint 3
Sprint 2
Sprint Backlog
Sprint Backlog
Sprint Backlog
Story Challenge
Suggest ‘what-ifs’ to challenge new stories and define story headlines
Increasing Scope of Int. Sys. and UAT
Increasing Scope of Integration, System and Users Testing
2. Story Definition
Introduce scenarios to enhance the Acceptance Criteria
Complete Tests after Final Sprint
Project Level Test Activities
(This diagram shows three sprints, but there could be more or fewer)
6. Integration Test
6. Integration Test
6. Integration Test
Intelligent Definition and Assurance
Slide

Daily Scrum
Stand-Up
Meeting
24 Hours
2-4 Weeks
Backlog tasks
expanded
by team
Potentially Shippable
Product increment
Product backlog
As prioritised by Product Owner
Sprint Backlog
4. Story Refinement
Refine scenarios to enhance story definition, create system tests as stories, as required
6) Integration/System Testing
Incorporate automated unit tests into the CI regime.
On weekly basis and at end of Sprint, deploy to System test environment and tester runs system tests.
3. Daily Stand-Up
Report anomalies found, stories tested, amended, created
5) Developer Testing
Private ad-hoc tests and build/run automated unit tests
Test Activities in the Sprint
Intelligent Definition and Assurance
Slide

Daily Scrum
Stand-Up
Meeting
24 Hours
2-4 Weeks
Backlog tasks
expanded
by team
Potentially Shippable
Product increment
Product backlog
As prioritised by Product Owner
Sprint Backlog
4. Story Refinement
Refine scenarios to enhance story definition, create system tests as stories, as required
6) Integration/System Testing
Incorporate automated unit tests into the CI regime.
On weekly basis and at end of Sprint, deploy to System test environment and tester runs system tests.
3. Daily Stand-Up
Report anomalies found, stories tested, amended, created
5) Developer Testing
Private ad-hoc tests and build/run automated unit tests
Test Activities in the Sprint
Intelligent Definition and Assurance
Slide

Daily Scrum
Stand-Up
Meeting
24 Hours
2-4 Weeks
Backlog tasks
expanded
by team
Potentially Shippable
Product increment
Product backlog
As prioritised by Product Owner
Sprint Backlog
4. Story Refinement
Refine scenarios to enhance story definition, create system tests as stories, as required
6) Integration/System Testing
Incorporate automated unit tests into the CI regime.
On weekly basis and at end of Sprint, deploy to System test environment and tester runs system tests.
3. Daily Stand-Up
Report anomalies found, stories tested, amended, created
5) Developer Testing
Private ad-hoc tests and build/run automated unit tests
Test Activities in the Sprint
Intelligent Definition and Assurance
Slide

Daily Scrum
Stand-Up
Meeting
24 Hours
2-4 Weeks
Backlog tasks
expanded
by team
Potentially Shippable
Product increment
Product backlog
As prioritised by Product Owner
Sprint Backlog
4. Story Refinement
Refine scenarios to enhance story definition, create system tests as stories, as required
6) Integration/System Testing
Incorporate automated unit tests into the CI regime.
On weekly basis and at end of Sprint, deploy to System test environment and tester runs system tests.
3. Daily Stand-Up
Report anomalies found, stories tested, amended, created
5) Developer Testing
Private ad-hoc tests and build/run automated unit tests
Test Activities in the Sprint
Intelligent Definition and Assurance
Slide

The tester’s contribution
Think of testing as interventions, not stages
The testing role is redistributed and split
The tester doesn’t own testing – think TESTMASTER
Identify the intervention points and negotiate with your team
But you must get involved early to trust the test automation.
Intelligent Definition and Assurance
Slide

Surviving Continuous Delivery and DevOps
How do Testers fit into a DevOps regime?

Tests, Automation and Trust
Can we trust automation to do all our testing?
Consider four areas:
Checks that can be automated by developers
Checks that can be automated (typically by system testers) to exercise API-level, end-to-end
Compatibility checks across browsers, operating systems, platforms.
Tests that can only be performed by a human
Can you 'let go' of late, manual checking?
It requires proactive effort and trust
Intelligent Definition and Assurance
Slide

It's easier to trust if you're a visionary
Is your concern that you can't trust the developers?
Shifting test thinking to the left reduces doubt?
Testers as pathfinders: identify/assess risks, formulate tests; ensure they are included in automation etc.
You are not the gatekeeper of quality, the last defence, the only person who cares
Think more like a visionary, risk-identifier, risk manager, pathfinder, a facilitator and a coach/mentor.
Intelligent Definition and Assurance
Slide

The Deployment Pipeline
Consider your test and release processes as a linked chain of processes (very waterfall hey?)
Some processes are automated, some are manual at least initially (and perhaps always)
The transitions between them are decision points (go back, recover, proceed to next etc.)
Either the system or people make the decisions, but people are always accountable.
Intelligent Definition and Assurance
Slide

Test automation
The manual interventions between automated activities in DevOps are the pain points: bottlenecks, delays, and error
Manual testing is painful in this respect
You love exploratory testing
You think only you can find those gnarly bugs
You think you are the only person trustworthy to prevent disaster
It might be painful for you to trust developers and automation to do testing
If it hurts, you must do it more often.
Intelligent Definition and Assurance
Slide

Monitoring and Improvement
A key discipline of DevOps is a deeper level of monitoring in production
Alerts on failure before users experience their impact
Ambitious, but this is the ultimate goal
When problems are encountered, use the analytics
To trace the cause and resolve it
To refine the test process ("let's adapt so we don't let that bug through again")
Automated tests in the process are 'monitoring' too
DevOps process monitors enlarge the scope of testing.
Intelligent Definition and Assurance
Slide

As a Tester, What Should You Do?
There are 'critical moments' in all projects where you might gather data and present feedback
You need to identify your own critical moments
Identify the choices that you and your team can make
Offer your contribution and negotiate with your team
These are your 'interventions'
Offer leadership and guidance rather than to take on responsibility for testing work
It's easier to demonstrate your value to the team if you take this approach – and the team won't need as many testers.
Intelligent Definition and Assurance
Slide

I'm a Test Lead/Manager. What Should I Do?
It might be harder to justify your role if the intent of management is to reduce the cost of testing by shifting left
Provide test and assurance skills to business
Manage Requirements knowledge
Be a TestMaster
Be a DevOps- or ChatOps-Master
Manage outsourced/offshore teams.
Intelligent Definition and Assurance
Slide

When should DevOps not be attempted?
A good question but the answer is a simple challenge:
Why wouldn't you want developers and operations people talking to each other?
Why wouldn't you want more reliable builds and deployments into test and production?
Why wouldn't you want the best of technology to support more accurate, efficient and informative pipelines?
Testers need to embrace DevOps because it provides opportunities to be pro-active, gain more authority and respect in our project teams.
Intelligent Definition and Assurance
Slide

The phase after development is
Re-Work

Thank-You!
@paul_gerrard
Paul Gerrard
email: [e-mail]
Twitter: @paul_gerrard
LinkedIn: http://linkedin.com/in/paulgerrard
tkbase.com | testopera.com

Testing, Analytics and Decision-Making (Shift-Right)

Introduction
The purpose of testing is to gather information for someone to make a decision
Testing provides the most valuable information required by:
Developers (to fix defects)
Project managers (to understand and manage progress)
Stakeholders (to be updated and assured)
In this one respect, testing is all-powerful.
Intelligent Definition and Assurance
Slide

SMAC – Real-Time Analytics
Apps capture information about their users and the usage patterns of their apps themselves
Apps collect data in real time
Data is analyzed to detect trends, patterns of behaviour, user preferences and opportunities for improvement or new market initiatives
All apps are instrumented to collect information for decision making.
Intelligent Definition and Assurance
Slide

Introducing Test Analytics
“The capture, integration and analysis of test and production monitoring data to inform business and software development decision-making”
Testers treat output from test management tools as the main source of information for reporting
But this is far too limited a viewpoint.
Intelligent Definition and Assurance
Slide

Modern Practices – Opportunities for Testing
Shift-Left aims to reduce, if not eliminate, misunderstandings in requirements
Pervasive automation in DevOps generates much of the data we need automatically
Results capture and analyses are no longer manual; reporting is almost instant
Some companies don’t log defects or bugs; when defects are found – they are fixed
But how does testing support decision-making?
Intelligent Definition and Assurance
Slide

Testing and Decision-Making
Testing supports decision making at all levels
Testing Stakeholders can have many roles
The information they require from testing varies in line with their perspective and their need to make decisions
We must ask our Stakeholders what they would find most useful, valuable and economic
But how can we quantify testing?
Time for some physics ☺
Intelligent Definition and Assurance
Slide

The challenge of test prediction and planning
The Testing Uncertainty Principle:
We can predict test status, but not when it will be achieved;
We can predict when a test will end, but not its status
We must treat exit criteria as planning assumptions
If we don’t meet the criteria
Our assumptions are wrong
Our plan is wrong
We need to re-plan, or re-scope the project.
Intelligent Definition and Assurance
Slide

The Testing Theory of Relativity 1
The value of a test is affected by many things
A single test might cover five or five million lines of code
But who can say what that value is?
We must rely on our Stakeholders to judge value
They can’t put a value on any test – but they can say which test is more valuable
This sounds like a big disadvantage – but it isn’t really.
Intelligent Definition and Assurance
Slide

The Testing Theory of Relativity 2
Most of the challenge of test planning is scope
We cannot test everything so we must prioritise in some way
Knowing the relative value of tests means we should be able to select the more valuable ones and set the other less valuable tests aside – for now
Given a valuable test, is a second test that does the same thing as valuable? Not usually
To discuss that, we need some quantum theory.
Intelligent Definition and Assurance
Slide

Quantum Theory of Testing
When we perform a test, pass or fail - the test incrementally increases our knowledge and confidence
Each test provides a quantum of evidence (yes/no, true/false etc)
We aggregate these to take a rounded view
A test only has value if we learn something new
If we learn little or nothing – the test has little or no value
A test is significant if it increases our knowledge about the system under test
A repeat run of a test has no significance because we learn nothing more.
Intelligent Definition and Assurance
Slide