Testing and Automation and Cooperation, Oh My!

Tim Flink

tflink@fedoraproject.org

DevConf 2017 - January 29, 2017

Introduction

Questions for the Audience

Experience with Automation

  • Have you worked with automation systems?
  • Have you worked with test automation systems?

Tight Coupling

Have you been hoping that another project's automation system could work for your use case only to find out that it's so tightly tied to their process, a huge amount of effort would be needed to use it and you'd almost be better off writing your own?

Roll Your Own

Have you actually gone off and rolled your own?

What am I talking about?

  • Collaboration on test tooling between projects
  • OpenStack, Fedora to start with
  • Possibly Ansible, RHEL, others?

Where Are We?

Fedora

  • Jobs run in response to events
    • Builds, Updates, Composes
  • "Gating" is easily overridden and voluntary

OpenStack

  • Closer to "pure" CI
  • Zuul manages testing of and gating for all incoming changes
  • Not quite a linux distro but plenty complicated

Where are we going?

  • We want to see more "pure" CI in Fedora
  • Difficult to talk about cooperating when the desired destination is TBD

Determine the input(s)

  • For OpenStack, it's pending patches

What is the input for Fedora?

  • Dist-Git Changes
  • Koji Builds
  • Composes

Change Gating

  • In order to have more "pure" CI, we need to start doing some gating
  • OpenStack gates at the change/patch level

Possible points to gate at

  • Changes
  • Builds
  • Images

Which will we use?

Find a place to run

  • A stable place for all of the needed CI bits to run
  • Will eventually need to be more flexible

TLDR

  • Test all the things, all the time
  • … at least as much as we can before it's released

Cooperation/Collaboration

Reasons For Collaboration

Less Wheel re-invention

More time for cool new things if there's less duplication

More people using a similar system increases awareness and usage

Why is Collaboration Difficult?

Everyone Has More Work than Time

Test Automation Efforts Are Rarely Easy to Combine

  • Tend to be project-specific, regardless of execution method
  • This makes sense since testing is very closely tied to a project
  • Can we do better?

Risks

Workflows are not identical

Requirements are different

Possible Politics

Are we doomed?

The Edge of Doom

Image Source

No, we are not doomed!

  • From a high level, goals are similar

Tool Overlap

  1. Python is already the primary language in both groups
  2. Both groups are looking to use Ansible for the heavy lifting

Similar High Level Design

  • May not look like it but there are a lot of overlapping concepts

Cooperation Point is Flow and Gating, not Test Runners/Frameworks

  • Not talking about sharing an entire test setup
  • Focusing on change flow and gating

What is needed?

Agreement that it'll help us all

  • None of these separate projects are going to be drop-in compatible from the start
  • General agreement that it's worthwhile

Do The Work

  • Expecting someone else to do your work is not likely to work out
  • Potentially more work than something less generic
  • Likely a do-ocracy

Enable modules and allow flexibility

Start going in the same direction

  • Not at exactly in the same place today
  • Changing tack to get at a closer point will help future efforts

Conclusion

These are the first steps into a brave new world

  • For us, anyways :)

Stay tuned for how things turn out

  • and/or submit patches … good patches will help

Questions