Upstream First Testing

"Upstream First Testing"?

• Doing all testing downstream is possible
• Testing upstream can have benefits

Details

• The idea here is upstream first, not upstream everything
• There will always be some things which don't make sense to run upstream

Benefits

• Better relationship with upstream
• Catch important bugs earlier
• Improve quality upstream

Why Should I Care?

Source Image (Hyperbole and a Half)

As a Downstream

• More testing upstream means more issues are found and fixed upstream
• Those fixes benefit downstream faster
• Product quality improves

As an Upstream

• Increased testing and participation
• More testing means finding more bugs are found and fixed

What's Available to Help Me Help Test Fedora?

• openQA
• Beaker (stg for now)
• Taskotron

openQA

openQA is an automated test tool for operating systems and the engine at the heart of openSUSE's automated testing initiative.

Fedora QA has recently deployed an openQA instance which we are using to test Fedora composes.

What's openQA Good For?

• Graphical testing
• Tests where you can't go in through the back door

openQA's Execution Model

• Schedule Job Against Artifacts
• Delegate to worker
• Compare graphical output to known screenshots or serial console

Beaker

Beaker is a system for test automation and lab management maintained by Red Hat and used heavily to test RHEL.

Fedora's Beaker Instance has several clients and is available for all Fedora contributors

What's Beaker Good For?

• Tests that need bare metal
• Longer running tests which benefit from a fresh install

Beaker's Execution Model

• Submit a Job
• Client is allocated
• OS is installed
• Job is run
• Machine is returned

Taskotron

Taskotron is a system made up of several components which are written and maintained by Fedora QA.

The core of Taskotron is the libtaskotron which runs the actual tasks.

What's Taskotron Good For?

• Everything?
• Delegating specialized bits off to specialized systems

Taskotron's Execution Model

• Trigger on incoming fedmsg
• Create VM for task execution
• Clone git repo for task
• Execute described task
• Report results to ResultsDB

More Taskotron

• Designed to coordinate automated tasks
• Relatively language agnostic
• Doesn't require lots of setup

How Can I Help TestFedora?

• Try to get apropriate tests released
• Work with upstreams to get the tests running
• Come to my Automation Workshop @ 14:00 in A113

Examples

• Much of Fedora
• RTT's Beaker Fedora Tests

Conclusion

• Pushing sensible amounts of testing upstream helps everyone
• If you're an employee, you should ask to release tests
• Everyone should take advantage of the resources we have