atmo

These are the submodules of the atmo package that don’t quite fit “topics”, like the atmo.clusters, atmo.jobs and atmo.users packages.

atmo.celery

class atmo.celery.AtmoCelery(main=None, loader=None, backend=None, amqp=None, events=None, log=None, control=None, set_as_current=True, tasks=None, broker=None, include=None, changes=None, config_source=None, fixups=None, task_cls=None, autofinalize=True, namespace=None, strict_typing=True, **kwargs)

A custom Celery class to implement exponential backoff retries.

backoff(n, cap=3600)

Return a fully jittered backoff value for the given number.

send_task(*args, **kwargs)

Send task by name.

Supports the same arguments as @-Task.apply_async().

Arguments:

name (str): Name of task to call (e.g., “tasks.add”). result_cls (~@AsyncResult): Specify custom result class.

class atmo.celery.ExpoBackoffFullJitter(base, cap)

Implement fully jittered exponential retries.

See for more infos:

• https://www.awsarchitectureblog.com/2015/03/backoff.html
• https://github.com/awslabs/aws-arch-backoff-simulator

backoff(n)

Return the exponential backoff value for the given number.

expo(n)

Return the exponential value for the given number.

atmo.celery.celery = <AtmoCelery atmo>

The Celery app instance used by ATMO, which auto-detects Celery config values from Django settings prefixed with “CELERY_” and autodiscovers Celery tasks from tasks.py modules in Django apps.

atmo.context_processors

atmo.context_processors.alerts(request)

Here be dragons, for who are bold enough to break systems and lose data

This adds an alert to requests in stage and development environments.

atmo.context_processors.settings(request)

Adds the Django settings object to the template context.

atmo.context_processors.version(request)

Adds version-related context variables to the context.

atmo.decorators

atmo.decorators.add_permission_required(model, **params)

Checks add object permissions for the given model and parameters.

atmo.decorators.change_permission_required(model, **params)

Checks change object permissions for the given model and parameters.

atmo.decorators.delete_permission_required(model, **params)

Checks delete object permissions for the given model and parameters.

atmo.decorators.modified_date(view_func)

A decorator that when applied to a view using a TemplateResponse will look for a context variable (by default “modified_date”) to set the header (by default “X-ATMO-Modified-Date”) with the ISO formatted value.

This is useful to check for modification on the client side. The end result will be a header like this:

X-ATMO-Modified-Date: 2017-03-14T10:48:53+00:00

atmo.decorators.permission_required(perm, klass, **params)

A decorator that will raise a 404 if an object with the given view parameters isn’t found or if the request user does not have the given permission for the object.

E.g. for checking if the request user is allowed to change a user with the given username:

@permission_required('auth.change_user', User)
def change_user(request, username):
    # can use get() directly since get_object_or_404 was already called
    # in the decorator and would have raised a Http404 if not found
    user = User.objects.get(username=username)
    return render(request, 'change_user.html', context={'user': user})

atmo.decorators.view_permission_required(model, **params)

Checks view object permissions for the given model and parameters.

atmo.models

class atmo.models.CreatedByModel(*args, **kwargs)

An abstract data model that has a relation to the Django user model as configured by the AUTH_USER_MODEL setting. The reverse related name is created_<name of class>s, e.g. user.created_clusters.all() where user is a User instance that has created various Cluster objects before.

Parameters:created_by_id (ForeignKey to User) – User that created the instance.

assign_permission(user, perm)

Assign permission to the given user, e.g. ‘clusters.view_cluster’,

save(*args, **kwargs)

Saves the current instance. Override this in a subclass if you want to control the saving process.

The ‘force_insert’ and ‘force_update’ parameters can be used to insist that the “save” must be an SQL insert or update (or equivalent for non-SQL backends), respectively. Normally, they should not be set.

class atmo.models.EditedAtModel(*args, **kwargs)

An abstract data model used by various other data models throughout ATMO that store timestamps for the creation and modification.

Parameters:

• created_at (DateTimeField) – Created at
• modified_at (DateTimeField) – Modified at

save(*args, **kwargs)

Saves the current instance. Override this in a subclass if you want to control the saving process.

The ‘force_insert’ and ‘force_update’ parameters can be used to insist that the “save” must be an SQL insert or update (or equivalent for non-SQL backends), respectively. Normally, they should not be set.

class atmo.models.PermissionMigrator(apps, model, perm, user_field=None, group=None)

A custom django-guardian permission migration to be used when new model classes are added and users or groups require object permissions retroactively.

assign()

The primary method to assign a permission to the user or group.

remove()

The primary method to remove a permission to the user or group.

class atmo.models.URLActionModel(*args, **kwargs)

A model base class to be used with URL patterns that define actions for models, e.g. /foo/bar/1/edit, /foo/bar/1/delete etc.

url_actions = []

The list of actions to be used to reverse the URL patterns with

url_delimiter = '-'

The delimiter to be used for the URL pattern names.

url_field_name = 'id'

The field name to be used with the keyword argument in the URL pattern.

url_kwarg_name = 'id'

The keyword argument name to be used in the URL pattern.

url_prefix = None

The prefix to be used for the URL pattern names.

atmo.models.next_field_value(model_cls, field_name, field_value, start=2, separator='-', max_length=0, queryset=None)

For the given model class, field name and field value provide a “next” value, which basically means a counter appended to the value.

atmo.names

atmo.names.adjectives = ['admiring', 'adoring', 'affectionate', 'agitated', 'amazing', 'angry', 'awesome', 'blissful', 'boring', 'brave', 'clever', 'cocky', 'compassionate', 'competent', 'condescending', 'confident', 'cranky', 'dazzling', 'determined', 'distracted', 'dreamy', 'eager', 'ecstatic', 'elastic', 'elated', 'elegant', 'eloquent', 'epic', 'fervent', 'festive', 'flamboyant', 'focused', 'friendly', 'frosty', 'gallant', 'gifted', 'goofy', 'gracious', 'happy', 'hardcore', 'heuristic', 'hopeful', 'hungry', 'infallible', 'inspiring', 'jolly', 'jovial', 'keen', 'kind', 'laughing', 'loving', 'lucid', 'mystifying', 'modest', 'musing', 'naughty', 'nervous', 'nifty', 'nostalgic', 'objective', 'optimistic', 'peaceful', 'pedantic', 'pensive', 'practical', 'priceless', 'quirky', 'quizzical', 'relaxed', 'reverent', 'romantic', 'sad', 'serene', 'sharp', 'silly', 'sleepy', 'stoic', 'stupefied', 'suspicious', 'tender', 'thirsty', 'trusting', 'unruffled', 'upbeat', 'vibrant', 'vigilant', 'vigorous', 'wizardly', 'wonderful', 'xenodochial', 'youthful', 'zealous', 'zen']

The adjectives to be used to generate random names.

atmo.names.random_scientist(separator=None)

Generate a random scientist name using the given separator and a random 4-digit number, similar to Heroku’s random project names.

atmo.names.scientists = ['albattani', 'allen', 'almeida', 'agnesi', 'archimedes', 'ardinghelli', 'aryabhata', 'austin', 'babbage', 'banach', 'bardeen', 'bartik', 'bassi', 'beaver', 'bell', 'benz', 'bhabha', 'bhaskara', 'blackwell', 'bohr', 'booth', 'borg', 'bose', 'boyd', 'brahmagupta', 'brattain', 'brown', 'carson', 'chandrasekhar', 'shannon', 'clarke', 'colden', 'cori', 'cray', 'curran', 'curie', 'darwin', 'davinci', 'dijkstra', 'dubinsky', 'easley', 'edison', 'einstein', 'elion', 'engelbart', 'euclid', 'euler', 'fermat', 'fermi', 'feynman', 'franklin', 'galileo', 'gates', 'goldberg', 'goldstine', 'goldwasser', 'golick', 'goodall', 'haibt', 'hamilton', 'hawking', 'heisenberg', 'hermann', 'heyrovsky', 'hodgkin', 'hoover', 'hopper', 'hugle', 'hypatia', 'jackson', 'jang', 'jennings', 'jepsen', 'johnson', 'joliot', 'jones', 'kalam', 'kare', 'keller', 'kepler', 'khorana', 'kilby', 'kirch', 'knuth', 'kowalevski', 'lalande', 'lamarr', 'lamport', 'leakey', 'leavitt', 'lewin', 'lichterman', 'liskov', 'lovelace', 'lumiere', 'mahavira', 'mayer', 'mccarthy', 'mcclintock', 'mclean', 'mcnulty', 'meitner', 'meninsky', 'mestorf', 'minsky', 'mirzakhani', 'morse', 'murdock', 'neumann', 'newton', 'nightingale', 'nobel', 'noether', 'northcutt', 'noyce', 'panini', 'pare', 'pasteur', 'payne', 'perlman', 'pike', 'poincare', 'poitras', 'ptolemy', 'raman', 'ramanujan', 'ride', 'montalcini', 'ritchie', 'roentgen', 'rosalind', 'saha', 'sammet', 'shaw', 'shirley', 'shockley', 'sinoussi', 'snyder', 'spence', 'stallman', 'stonebraker', 'swanson', 'swartz', 'swirles', 'tesla', 'thompson', 'torvalds', 'turing', 'varahamihira', 'visvesvaraya', 'volhard', 'wescoff', 'wiles', 'williams', 'wilson', 'wing', 'wozniak', 'wright', 'yalow', 'yonath']

The scientists to be used to generate random names.

atmo.provisioners

class atmo.provisioners.Provisioner

A base provisioner to be used by specific cases of calling out to AWS EMR. This is currently storing some common code and simplifies testing.

Subclasses need to override there class attributes:

• log_dir
• name_component

job_flow_params(user_username, user_email, identifier, emr_release, size)

Given the parameters returns the basic parameters for EMR job flows, and handles for example the decision whether to use spot instances or not.

log_dir = None

The name of the log directory, e.g. ‘jobs’.

name_component = None

The name to be used in the identifier, e.g. ‘job’.

spark_emr_configuration()

Fetch the Spark EMR configuration data to be passed as the Configurations parameter to EMR API endpoints.

We store this in S3 to be able to share it between various Telemetry services.

atmo.tasks

Task: atmo.tasks.cleanup_permissions

A Celery task that cleans up old django-guardian object permissions.

atmo.templatetags

atmo.templatetags.full_url(url)

A Django template filter to prepend the given URL path with the full site URL.

atmo.templatetags.markdown(content)

A Django template filter to render the given content as Markdown.

atmo.templatetags.url_update(url, **kwargs)

A Django template tag to update the query parameters for the given URL.

atmo.views

class atmo.views.DashboardView(**kwargs)

The dashboard view that allows filtering clusters and jobs shown.

active_cluster_filter = 'active'

Active filter for clusters

clusters_filters = ['active', 'terminated', 'failed', 'all']

Allowed filters for clusters

default_cluster_filter = 'active'

Default cluster filter

http_method_names = ['get', 'head']

No need to accept POST or DELETE requests

maintainer_group_name = 'Spark job maintainers'

Name of auth group that is checked to display Spark jobs

template_name = 'atmo/dashboard.html'

Template name

atmo.views.permission_denied(request, exception, template_name='403.html')

Permission denied (403) handler.

Template:403.html

If the template does not exist, an Http403 response containing the text “403 Forbidden” (as per RFC 7231) will be returned.

atmo.views.server_error(request, template_name='500.html')

500 error handler.

Template:500.html