libtaskotron API docs

This isn’t final API documentation, more of an example of how we can do API docs.

Once more of our code is using sphinx-style docs, we can start finalizing the format for this page.

Note that the docs here are for libtaskotron library functions and don’t include directives or task yaml details.

bodhi_utils

Utility functions for dealing with Bodhi

class libtaskotron.bodhi_utils.BodhiUtils(client=None)

Helper Bodhi methods.

Variables:client (fedora.client.BodhiClient) – Bodhi client instance

Create a new BodhiUtils instance.

Parameters:client – custom BodhiClient instance. If None, a default BodhiClient instance is used.

build2update(builds, strict=False)

Find matching Bodhi updates for provided builds.

Parameters:

• builds (iterable of str) – builds to search for in N(E)VR format (foo-1.2-3.fc20 or foo-4:1.2-3.fc20)
• strict (bool) – if False, incomplete Bodhi updates are allowed. If True, every Bodhi update will be compared with the set of provided builds. If there is an Bodhi update which contains builds not provided in builds, that update is marked as incomplete and removed from the result - i.e. all builds from builds that were part of this incomplete update are placed in the second dictionary of the result tuple.

Returns:

a tuple of two dictionaries:

• The first dict provides mapping between builds and their updates where no error occured.

  {build (string): Bodhi update (Bunch)}

• The second dict provides mapping between builds and their updates where some error occured. The value is None if the matching Bodhi update could not be found (the only possible cause of failure if strict=False). Or the value is a Bodhi update that was incomplete (happens only if strict=True).

  {build (string): Bodhi update (Bunch) or None}

Raises TaskotronValueError:  

if builds type is incorrect

query_update(package)

Get the last Bodhi update matching criteria.

Parameters:package (str) –

package NVR or package name or update title or update ID

Note

Only NVR allowed, not NEVR. See https://fedorahosted.org/bodhi/ticket/592.

Returns:most recent Bodhi update object matching criteria, or None when no update is found. Return type:bunch.Bunch

check

Helper tools for managing check status, outcome, and output.

class libtaskotron.check.CheckDetail(item, report_type=None, outcome=None, summary='', output=None, keyvals=None)

Class encompassing everything related to the outcome of a check run. It contains convenience methods for tracking, evaluating and reporting check results.

For some checks, it’s trivial to parse its output at the end of its execution and evaluate the results. For some, however, it’s much easier to do this continuously during check execution. This is especially true for multi-item checks (a single check providing results for many items - builds, updates, etc). That’s when this class comes very handy (it can be used even for the simple use cases, of course).

Variables:

• outcome_priority – a tuple of outcome keywords sorted by priority from the least important to the most important
• item (str) – a description the item being tested; for example a build NVR (vpnc-0.5-1.fc20), update ID (FEDORA-2014-3309) or a repository name (f20-updates)
• report_type (str) – a definition of the type of the object being checked; for example ‘a Koji build’, ‘a Bodhi update’ or ‘a yum repository name’. The allowed values are attributes in ReportType. You don’t have to fill this in (or you can provide a custom string value), but the reporting directives react only to the known types (you can always find it in ResultsDB, though).
• outcome (str) –

  a keyword specifying the final outcome of the check. Available outcome keywords:

  • PASSED - everything went well
  • INFO - everything went well, but there is some important information that needs to be pointed out
  • FAILED - the item in question fails the check
  • NEEDS_INSPECTION - the outcome can’t be determined and a manual inspection is needed
  • ABORTED - the check aborted itself because of some unexpected problems, i.e. a necessary network server is not reachable. Running this check with the same arguments later can help to mitigate this problem.
  • CRASHED - the check crashed and did not provide usable results

  If no outcome has been set, this attribute returns NEEDS_INSPECTION.

  Raises TaskotronValueError if you try to assign an unknown keyword.

• summary (str) – a few words or one-sentence summary about the result of the check run (e.g. 5 WARNINGS, 1 ERROR for an rpmlint result). Should not unnecessarily duplicate item or outcome, if possible.
• output – output from the check run (a list of strings). You can easily populate this by using @method store(), or you can modify it directly (for example filter out some messages you don’t want to see in the check output).
• keyvals (dict) – all key-value pairs in this dictionary are stored in ResultsDB as ‘extra data’. This can be used to store e.g. a compose id or a kernel version, if that information is important for querying the results. Keep it as short as possible.

broken()

Tells whether the check outcome is set to one of the broken states (i.e. ABORTED or CRASHED).

classmethod cmp_outcome(outcome1, outcome2)

Compare two outcomes according to outcome_priority and return -1/0/1 if outcome1 has lower/equal/higher priority than outcome2.

Parameters:

• outcome1 (str) – an outcome keyword to compare
• outcome2 (str) – an outcome keyword to compare

Raises TaskotronValueError:  

if any of the outcome keywords are not specified in outcome_priority

classmethod create_multi_item_summary(outcomes)

Create a string containing a sum of all outcomes, like this: 3 PASSED, 1 INFO, 2 FAILED

Parameters:outcomes – either one CheckDetail instance or an iterable of CheckDetails or an iterable of outcome strings

store(message, printout=True)

Add a string to the output, and print it optionally as well.

This is just a convenience method for most common use case, you can of course always access and modify output directly, and print or use logging facilities directly as well. This combines both into a single call.

Parameters:

• message (str) – a string to store in output
• printout (bool) – whether to print to standard output or not

update_outcome(outcome)

Update outcome with the provided value only and only if it has a higher priority than its current value (according to outcome_priority). Otherwise this call does nothing.

This is useful if your check performs a number of ‘sub-tasks’, each passing or failing, and you want to final outcome to be the worst/most serious one of those. You can just keep calling update_outcome() and the highest priority outcome type will be stored at outcome at the end.

Parameters:outcome (str) – the outcome value to assign to outcome if it has a higher priority than its current value. Handles None values gracefully (no action). Raises TaskotronValueError:  if any of the outcome keywords are not specified in outcome_priority

class libtaskotron.check.ReportType

Enum for different types of CheckDetail.report_type

BODHI_UPDATE = 'bodhi_update'

KOJI_BUILD = 'koji_build'

YUM_REPOSITORY = 'yum_repository'

libtaskotron.check.export_TAP(*check_details)

Generate TAP output used for reporting to ResultsDB.

Parameters:check_details – any number of CheckDetail instances Returns:TAP output with results for every CheckDetail instance provided Return type:str Raises TaskotronValueError:  if CheckDetail.item is empty for any parameter provided

Example output:

TAP Version 13
1..1
ok - $CHECKNAME for Koji build xchat-0.5-1.fc20
  ---
  details:
    output: |-
      xchat.x86_64: W: file-not-utf8 /usr/share/doc/xchat/ChangeLog
      xchat.x86_64: W: non-conffile-in-etc /etc/gconf/schemas/apps.schemas
      xchat.x86_64: W: no-manual-page-for-binary xchat
  item: xchat-0.5-1.fc20
  outcome: PASSED
  summary: 5 ERRORS, 10 WARNINGS
  type: koji_build
  ...

libtaskotron.check.import_TAP(source)

Parses TAP and returns a list of CheckDetails.

Parameters:source (str) – TAP-formatted text Raises TaskotronValueError:  if TAP syntax is incorrect

config

Global configuration for Taskotron and relevant helper methods.

libtaskotron.config.CONF_DIRS = ['/home/tflink/code/taskotron/libtaskotron/conf', '/etc/taskotron']

A list of directories where config files are stored. The config files are loaded from these locations in the specified order, and only the first config file found is used (further locations are ignored). The first location is dynamic, relative to the package location and it reflects the usual config location in a git checkout.

libtaskotron.config.CONF_FILE = 'taskotron.yaml'

the name of our configuration file

libtaskotron.config.PROFILE_VAR = 'TASKOTRON_PROFILE'

environment variable name for setting config profile

libtaskotron.config.get_config()

Get the Config instance. This method is implemented using the singleton pattern - you will always receive the same instance, which will get auto-initialized on the first method call.

Returns:either Config or its subclass, depending on taskotron profile used. Raises TaskotronConfigError:  if config file parsing and handling failed

config_defaults

This includes the default values for Taskotron configuration. This is automatically loaded by config.py and then overridden by values from config files available in system-wide location.

class libtaskotron.config_defaults.Config

Global configuration for Taskotron (development profile).

The documentation for individual options is available in the config files (unless they’re not present in the config files, then they’re documented here).

Implementation notes:

• If you want to add a new option, put it here and optionally into the config file as well.
• If you modify a default value for some option, don’t forget to modify it in both places - here and in the config file (if present).
• Don’t assign None as a default value. We need to know a value type in order to check for correct type of user-provided values.

bodhi_posting_comments_span = 4320

bodhi_server = 'https://admin.fedoraproject.org/updates/'

buildbot_task_step = 'runtask'

fas_password = ''

fas_username = 'taskotron'

koji_url = 'http://koji.fedoraproject.org/kojihub'

logdir = '/var/log/taskotron'

main_log_name = 'taskotron.log'

name of the main log file in logdir

pkg_url = 'http://kojipkgs.fedoraproject.org/packages'

profile = 'development'

report_to_bodhi = True

report_to_resultsdb = True

reporting_enabled = False

resultsdb_server = 'http://resultsdb.qa.fedoraproject.org/resultsdb/api/v1.0/'

taskotron_master = 'http://taskotron.qa.fedoraproject.org/taskmaster/'

tmpdir = '/var/tmp/taskotron'

class libtaskotron.config_defaults.ProductionConfig

Configuration for production profile. Inherits values from Config and overrides some. Read Config documentation.

profile = 'production'

reports_enabled = True

class libtaskotron.config_defaults.ProfileName

Enum of available profile names. These can be specified in the config file or as the environment variable.

DEVELOPMENT = 'development'

PRODUCTION = 'production'

TESTING = 'testing'

class libtaskotron.config_defaults.TestingConfig

Configuration for testing suite profile. Inherits values from Config and overrides some. Read Config documentation.

profile = 'testing'

exceptions

This module contains custom Taskotron exceptions

exception libtaskotron.exceptions.TaskotronConfigError

All errors related to Taskotron config files

exception libtaskotron.exceptions.TaskotronDirectiveError

All errors related to Taskotron directives

exception libtaskotron.exceptions.TaskotronError

Common ancestor for Taskotron related exceptions

exception libtaskotron.exceptions.TaskotronRemoteError

All network and remote-server related errors

exception libtaskotron.exceptions.TaskotronValueError

Taskotron-specific ValueError

exception libtaskotron.exceptions.TaskotronYamlError

Error in YAML config file of the executed check

file_utils

libtaskotron.file_utils.download(url, dest, overwrite=False, grabber=None)

Download a file.

Parameters:

• url (str) – file URL to download
• dest (str) – either an existing directory or a file path. In the latter case all parent directories will be created if necessary and the last fraction of the path will be used as the file name.
• overwrite (bool) – if the destination file already exists, whether to overwrite or not. If False, a simple check is performed whether the remote file is the same as the local file and re-downloads it anyway if they are not the same.
• grabber – custom urlgrabber.grabber.URLGrabber instance to be used for actual downloading

Returns:

file path of the downloaded file

Return type:

str

Raises TaskotronRemoteError:  

if download fails or directory can’t be created

libtaskotron.file_utils.get_urlgrabber(**kwargs)

Get a new instance of URLGrabber that is reasonably configured (progress bar when in terminal, automatic retries, connection timeout, etc)

Parameters:kwargs – additional kwargs to add to the URLGrabber constructor Returns:urlgrabber.grabber.URLGrabber instance

libtaskotron.file_utils.makedirs(fullpath)

This is the same as os.makedirs(), but does not raise an exception when the destination directory already exists.

Raises OSError:if directory doesn’t exist and can’t be created

koji_utils

Utility methods related to Koji

class libtaskotron.koji_utils.KojiClient(koji_session=None)

Helper Koji methods.

Variables:session (koji.ClientSession) – Koji client session

Create a new KojiClient

Parameters:koji_session (koji.ClientSession) – an existing Koji session instance or None if you want a new default session to be created

get_nvr_rpms(nvr, rpm_dir, arches=None, debuginfo=False, src=False)

Retrieve the RPMs associated with a build NVR into the specified

directory.

Parameters:

• nvr build (str) – NVR
• rpm_dir (str) – location where to store the RPMs

Returns:

list of local filenames of the grabbed RPMs

Return type:

list of str

Raises TaskotronRemoteError:  

if downloading failed

get_tagged_rpms(tag, dest, arches)

Downloads all RPMs of all NVRs tagged by a specific Koji tag.

Parameters:

• dest (str) – destination directory
• arches (list of str) – list of architectures

Raises TaskotronRemoteError:  

if downloading failed

nvr_to_urls(nvr, arches=None, debuginfo=False, src=True)

Get list of URLs for RPMs corresponding to a build.

Parameters:

• nvr (str) – build NVR
• arches – restrict the arches of builds to provide URLs for. Note: If basearch i386 is in the list, i686 arch is automatically added (since that’s the arch Koji API uses).

libtaskotron.koji_utils.getNEVR(build)

Extract RPM version identifier in NEVR format from Koji build object

Parameters:build (dict) – Koji buildinfo dictionary (as returned e.g. from koji.getBuild()) Returns:NEVR string; epoch is included when non-zero, otherwise omitted Raises TaskotronValueError:  if build is of incorrect type

logger

libtaskotron.logger.init(name='libtaskotron', level=20, stream=True, syslog=False, filelog=None)

Setup a logger

Parameters:

• name (str) – name of the logger, ‘libtaskotron’ is default
• level (level definitions from logging) – level of logging
• stream (bool) – enables logging to stderr
• syslog (bool) – enables logging to syslog
• filelog (str) – enables logging to a file, the value is the file name

python_utils

A collection of convenience methods related to Python base libraries.

libtaskotron.python_utils.iterable(obj, item_type=None)

Decide whether obj is an Iterable (you can traverse through its elements - e.g. list, tuple, set, even dict), but not basestring (which satisfies most collections’ requirements, but we don’t often want to consider as a collection). You can also verify the types of items in the collection.

Parameters:

• obj (any) – any object you want to check
• item_type (type) – all items in obj must be instances of this provided type. If None, no check is performed.

Returns:

whether obj is iterable but not a string, and whether obj contains only item_type items

Return type:

bool

Raises TaskotronValueError:  

if you provide invalid parameter value

libtaskotron.python_utils.sequence(obj, item_type=None, mutable=False)

This has the same functionality and basic arguments as iterable() (read its documentation), but decides whether obj is a Sequence (ordered and indexable collection - e.g. list or tuple, but not set or dict).

Parameters:mutable (bool) – if True, the obj must be a mutable sequence (e.g. list, but not tuple)

rpm_utils

Utility methods related to RPM

libtaskotron.rpm_utils.basearch(arch=None)

This returns the ‘base’ architecture identifier for a specified architecture (e.g. i386 for i[3-6]86), to be used by YUM etc.

Parameters:arch (str) – an architecture to be converted to a basearch. If None, then the arch of the current machine is used. Returns:basearch, or arch if no basearch was found for it Return type:str

libtaskotron.rpm_utils.cmpNEVR(nevr1, nevr2)

Compare two RPM version identifiers in NEVR format.

Parameters:

• nevr1 (str) – RPM identifier in N(E)VR format
• nevr2 (str) – RPM identifier in N(E)VR format

Returns:

-1/0/1 if nevr1 < nevr2 / nevr1 == nevr2 / nevr1 > nevr2

Return type:

int

Raises TaskotronValueError:  

if name in nevr1 doesn’t match name in nevr2

libtaskotron.rpm_utils.rpmformat(rpmstr, fmt='nvr', end_arch=False)

Parse and convert an RPM package version string into a different format. String identifiers: N - name, E - epoch, V - version, R - release, A - architecture.

Parameters:

• rpmstr (str) – string to be manipulated in a format of N(E)VR (foo-1.2-3.fc20 or bar-4:1.2-3.fc20) or N(E)VRA (foo-1.2-3.fc20.x86_64 or bar-4:1.2-3.fc20.i686)
• fmt (str) – desired format of the string to be returned. Allowed options are: nvr, nevr, nvra, nevra, n, e, v, r, a. If arch is not present in rpmstr but requested in fmt, noarch is used. Epoch is provided only when specifically requested (e.g. fmt='nevr') and being non-zero; otherwise it’s supressed (the only exception is fmt='e', where you receive 0 for zero epoch).
• end_arch (bool) – set this to True if rpmstr ends with an architecture identifier (foo-1.2-3.fc20.x86_64). It’s not possible to reliably distinguish that case automatically.

Returns:

string based on the specified format, or integer if fmt='e'

Raises TaskotronValueError:  

if fmt value is not supported

runner

libtaskotron.runner.process_args(args)

Processes raw input args and converts them into specific data types that can be used by tasks. This includes e.g. creating new args based on (item, item_type) pairs, or adjusting selected architecture.

Parameters:args (dict) – dictionary of raw arguments Returns:dict of args with appended specific data

yumrepoinfo

A wrapper object for yumrepoinfo.conf to access its information easily

class libtaskotron.yumrepoinfo.YumRepoInfo(arch=None, filelist=None)

This class is a wrapper for easily accessing repoinfo.conf file.

Parameters:

• arch (str) – architecture for which to adjust repo URLs. By default it refers to the architecture of the current machine. It’s always converted to basearch.
• filelist (iterable of str) – list of config files to read information from. The first available config file is used. If None, then the default list of locations is used.

Raises TaskotronConfigError:  

if no YUM repositories data is found (empty or non-existent config file). It’s not raised if you specifically request no data to load (filelist=[]).

arches(reponame)

Get the list of all supported architectures for a repo

Parameters:reponame (str) – repository name

get(reponame, key)

Get a specific key value from a repo

Parameters:

• reponame (str) – repository name
• key (str) – name of the key you want to retrieve from a repository (section)

Raises TaskotronConfigError:  

if the key can’t be retrieved (e.g. wrong key or repo name)

releases()

Get the list of stable (supported) Fedora releases.

repo(reponame)

Given a repo name, return the yumrepoinfo dict with keys: arches, parents, tag, url, path and name

Parameters:reponame (str) – repository name Return type:dict

repo_by_tag(tag)

Given a Koji tag, return the corresponding repo dict.

Parameters:tag (str) – a koji tag, e.g. f20-updates. Note: rawhide is not used as a Koji tag, use number identifier instead or use repo('rawhide')['tag'] to discover it first. Returns:repo dict as from repo(), or None if no such Koji tag is found

repos()

Get the list of all known repository names.

top_parent(reponame)

Go through the repo hiearchy and find the top parent for a repo

Parameters:reponame (str) – repository name Returns:the top parent reponame. If reponame doesn’t have any parent, its name is returned (it’s its own top parent) Return type:str Raises TaskotronConfigError:  if infinite parent loop detected

libtaskotron.yumrepoinfo.get_yumrepoinfo(arch=None)

Get YumRepoInfo instance. This method is implemented using the singleton pattern - you will always receive the same instance, which will get auto-initialized on the first method call.

Parameters:arch (str) – architecture to return the YumRepoInfo for. It’s always converted to basearch. If None, then local machine arch is used. Returns:shared YumRepoInfo instance Raises TaskotronConfigError:  if file config parsing and handling failed