Update main for v0.1

.pre-commit-config.yaml

@@ -0,0 +1,38 @@
+# See https://pre-commit.com for more information
+repos:
+- repo: https://github.com/pre-commit/pre-commit-hooks
+  rev: v4.1.0  # Python 3.6 compatible
+  hooks:
+  # Python related checks
+  - id: check-ast
+  - id: check-builtin-literals
+  - id: check-docstring-first
+  - id: name-tests-test
+    name: Check unit tests start with 'test_'
+    args: ['--django']
+    files: 'test/.*'
+  # Other checks
+  - id: check-added-large-files
+  - id: check-case-conflict
+  - id: check-merge-conflict
+  - id: check-yaml
+  - id: debug-statements
+  - id: detect-private-key
+  - id: end-of-file-fixer
+  - id: mixed-line-ending
+    name: Force line endings to LF
+    args: ['--fix=lf']
+  - id: trailing-whitespace
+
+- repo: https://github.com/pre-commit/pygrep-hooks
+  rev: v1.10.0
+  hooks:
+  - id: python-check-mock-methods
+  - id: python-no-eval
+  - id: python-no-log-warn
+  - id: python-use-type-annotations
+
+# Pre-commit CI config, see https://pre-commit.ci/
+ci:
+  autofix_prs: false
+  autoupdate_schedule: quarterly

docs/setting_up_VM_with_app.md

@@ -0,0 +1,27 @@
+## Steps to follow to get a VM with monitoring app running on Apache
+
+To get a 'prototype' of the monitoring app running with Apache, follow these steps:
+- create a cloud VM of the type: scientific-linux-7-aq
+- continue by selecting sandbox 'testing_personality_2', archetype 'ral-tier1', personality 'apel-data-validation-test'
+
+Allow 15 minutes after the machine is created, then remember to edit security groups from OpenStack to allow Apache to work.
+Then follow these steps from within the machine:
+- quattor-fetch && quattor-configure --all
+- cd /usr/share/DJANGO_MONITORING_APP/monitoring
+- modify the file settings.py, specifically the dict called DATABASES, to include the right credentials and database names
+- cd ..
+- source venv/bin/activate
+- systemctl restart httpd
+- sudo chown apache .
+
+At this point the app should be working, so just get the ip address by writing "hostname -I" within the machine and the app should be already running at that address.
+
+
+## What to do if the app seems to stop working after closing the VM
+If the VM is shut down, next time we try to open the app, Apache might give the error message "Unable to open the database file".
+If this happens, just follow these steps on the machine:
+1. cd /usr/share/DJANGO_MONITORING_APP
+2. source venv/bin/activate
+3. sudo chown apache .
+
+Note that step 2 is necessary for step 3 to work and the error message to disappear.

docs/what_gets_installed.md

@@ -0,0 +1,23 @@
+For Django to work with apache, it is common to have a venv within the app, where django and djangorestframework get installed. Other packages needed for the app to work are installed by Aquilon outside the venv.
+
+## Packages installed by Aquilon outside the venv
+Following the config file that Aquilon uses, the following are the packages installed:
+- `httpd`
+- `python3-mod_wsgi` (for apache to work with django)
+- `python3-devel`
+- `gcc` (needed for dependencies)
+- `mariadb`
+- `tar`
+
+## Packages installed within the venv
+Within venv, the following are installed through pip:
+- `djangorestframework` (3.15.1)
+- `pymysql` (1.0.2) (needed for mariadb to work)
+- `pandas` (1.1.5) (needed by the app)
+- `django` (3.2.25)
+- `pytz` (2025.2)
+
+Note that when the version of the packages is specified, the app would not work with a different version (due to dependencies conflicts).
+
+Is is also important to note that different types of OS require different packages to be installed.
+The above are the packages that allow the app to work on a Rocky8.

docs/what_pages_can_be_accessed.md

@@ -0,0 +1,22 @@
+## Pages that can be accessed through the app
+
+The following urls are the ones that can be accessed without passing any parameter:
+- http://ip-address/publishing/cloud/
+- http://ip-address/publishing/gridsync/
+
+These pages show info for a number of sites, so do not require a site name to be specified within the url.
+
+The url http://ip-address/publishing/grid/ , instead, should be used together with the name of the site we are looking for.
+For example: http://ip-address/publishing/grid/BelGrid-UCL/
+It is not supposed to be used without passing the name of the site.
+
+The url http://ip-address/publishing/gridsync/ shows a sync table, and it's probably the most important bit of the personality 'apel-data-validation'.
+This table contains data related to many sites, specifically number of jobs being published vs in the db, and this number is shown for every (available) month of each site.
+
+Clicking on any name in the first column (containing site names) allows to access a similar table which only shows the data relative to that site.
+This more specific table is such that the first columns shows the months for which we have data (for that site).
+Clicking on the month allows to open another table that shows data for that month and site only, divided by submithost.
+
+The pages accessed through the links can of course be accessed by typing directly the url. For example, if we want data related to the site 'CSCS-LCG2' and month '2013-11', we would type :
+http://ip-address/publishing/gridsync/CSCS-LCG2/2013-11/
+However, in this case if there is no data for the month we are looking for, we would get an error.

monitoring/__init__.py

@@ -0,0 +1,2 @@
+import pymysql
+pymysql.install_as_MySQLdb()

monitoring/availability/apps.py

@@ -5,4 +5,4 @@
 
 
 class AvailabilityConfig(AppConfig):
-    name = 'availability'
+    name = 'monitoring.availability'

monitoring/availability/templates/status.html

@@ -0,0 +1 @@
+<html>{{ message }}</html>

monitoring/availability/urls.py

@@ -1,7 +1,7 @@
-from django.conf.urls import url
+from django.urls import path
 
-import views
+from monitoring.availability import views
 
 urlpatterns = [
-    url(r'^$', views.status),
+    path('', views.status, name='availability'),
 ]

monitoring/availability/views.py

@@ -1,15 +1,10 @@
 # -*- coding: utf-8 -*-
 from __future__ import unicode_literals
 
-import time
-
 from rest_framework.decorators import api_view
 from rest_framework.response import Response
 
 
 @api_view()
 def status(requst):
-    if int(time.time()) % 2:
-        return Response("Everything OK")
-    else:
-        return Response("Everything NOT ok.")
+    return Response({"message": "OK"}, status=200, template_name="status.html")

monitoring/benchmarks/admin.py

@@ -0,0 +1,3 @@
+from django.contrib import admin
+
+# Register your models here.

monitoring/benchmarks/apps.py

@@ -0,0 +1,5 @@
+from django.apps import AppConfig
+
+
+class BenchmarksConfig(AppConfig):
+    name = 'monitoring.benchmarks'

monitoring/benchmarks/models.py

@@ -0,0 +1,54 @@
+from django.db import models
+
+
+class BenchmarksBySubmithost(models.Model):
+    fetched = models.DateTimeField(auto_now=True)
+    SiteName = models.CharField(max_length=255)
+    SubmitHost = models.CharField(max_length=255)
+    BenchmarkType = models.CharField(max_length=50)
+    BenchmarkValue = models.DecimalField(max_digits=10, decimal_places=3)
+    RecordType = models.CharField(max_length=50)
+    UpdateTime = models.DateTimeField()
+
+    class Meta:
+        ordering = ('SiteName',)
+
+
+class VJobRecords(models.Model):
+    Site = models.CharField(max_length=255, primary_key=True)
+    SubmitHost = models.CharField(max_length=255)
+    ServiceLevelType = models.CharField(max_length=50)
+    ServiceLevel = models.DecimalField(max_digits=10, decimal_places=3)
+    UpdateTime = models.DateTimeField()
+    EndTime = models.DateTimeField()
+
+    class Meta:
+        managed = False
+        db_table = 'VJobRecords'
+        verbose_name = 'Job Record'
+
+
+class VSummaries(models.Model):
+    Site = models.CharField(max_length=255, primary_key=True)
+    SubmitHost = models.CharField(max_length=255)
+    ServiceLevelType = models.CharField(max_length=50)
+    ServiceLevel = models.DecimalField(max_digits=10, decimal_places=3)
+    UpdateTime = models.DateTimeField()
+
+    class Meta:
+        managed = False
+        db_table = 'VSummaries'
+        verbose_name = 'Summary'
+
+
+class VNormalisedSummaries(models.Model):
+    Site = models.CharField(max_length=255, primary_key=True)
+    SubmitHost = models.CharField(max_length=255)
+    ServiceLevelType = models.CharField(max_length=50)
+    ServiceLevel = models.DecimalField(max_digits=10, decimal_places=3)
+    UpdateTime = models.DateTimeField()
+
+    class Meta:
+        managed = False
+        db_table = 'VNormalisedSummaries'
+        verbose_name = 'Normalised Summary'

monitoring/benchmarks/serializers.py

@@ -0,0 +1,21 @@
+from rest_framework import serializers
+
+from monitoring.benchmarks.models import BenchmarksBySubmithost
+
+
+class BenchmarksBySubmithostSerializer(serializers.HyperlinkedModelSerializer):
+    # Override default format with None so that Python datetime is used as
+    # ouput format. Encoding will be determined by the renderer and can be
+    # formatted by a template filter.
+    UpdateTime = serializers.DateTimeField(format=None)
+
+    class Meta:
+        model = BenchmarksBySubmithost
+        fields = (
+            'SiteName',
+            'SubmitHost',
+            'BenchmarkType',
+            'BenchmarkValue',
+            'RecordType',
+            'UpdateTime',
+        )

monitoring/benchmarks/templates/benchmarks_by_submithost.html

@@ -0,0 +1,49 @@
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
+<html lang="en" xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
+
+<head>
+  <meta content="text/html;charset=utf-8" http-equiv="Content-Type" />
+  <title>Sites publishing benchmark records</title>
+</head>
+
+<body>
+  <h2>Sites publishing benchmark records in last 3 months</h2>
+  <p>Page last updated: {{ last_fetched|date:"Y-m-d H:i:s.u" }}</p>
+  <table border="1">
+    <tr>
+      <th>Record type</th>
+      <th>Number of Sites</th>
+    </tr>
+    {% for item in site_counts_by_record_type %}
+    <tr>
+        <td>{{ item.RecordType }}</td>
+        <td>{{ item.site_count }}</td>
+    </tr>
+     {% endfor %}
+  </table>
+
+  <br/>
+
+  <table border="1">
+    <tr>
+      <th>Site</th>
+      <th>Submit host</th>
+      <th>Benchmark type</th>
+      <th>Benchmark value</th>
+      <th>Record type</th>
+      <th>Last updated</th>
+    </tr>
+    {% for benchmark in benchmarks %}
+    <tr>
+      <td>{{ benchmark.SiteName }}</td>
+      <td>{{ benchmark.SubmitHost }}</td>
+      <td>{{ benchmark.BenchmarkType }}</td>
+      <td>{{ benchmark.BenchmarkValue }}</td>
+      <td>{{ benchmark.RecordType }}</td>
+      <td>{{ benchmark.UpdateTime|date:"Y-m-d H:i:s" }}</td>
+    </tr>
+    {% endfor %}
+  </table>
+</body>
+
+</html>

monitoring/benchmarks/tests.py

@@ -0,0 +1,3 @@
+from django.test import TestCase
+
+# Create your tests here.

monitoring/benchmarks/urls.py

@@ -0,0 +1,8 @@
+from rest_framework import routers
+
+from monitoring.benchmarks import views
+
+router = routers.SimpleRouter()
+router.register('', views.BenchmarksViewSet)
+
+urlpatterns = router.urls

monitoring/benchmarks/views.py

@@ -0,0 +1,45 @@
+from django.shortcuts import render
+from datetime import datetime, timedelta
+
+from django.db.models import Max, Count
+from django.db.models.functions import Lower
+
+from rest_framework import viewsets
+from rest_framework.renderers import TemplateHTMLRenderer
+
+
+from monitoring.benchmarks.models import BenchmarksBySubmithost
+
+from monitoring.benchmarks.serializers import BenchmarksBySubmithostSerializer
+
+class BenchmarksViewSet(viewsets.ReadOnlyModelViewSet):
+    # Lower('SiteName'): sorts sites alphabetically, case-insensitively.
+    # '-UpdateTime': sorts records within each site by UpdateTime in descending order (latest first).
+    queryset = BenchmarksBySubmithost.objects.all().order_by(Lower('SiteName'), '-UpdateTime')
+
+    serializer_class = BenchmarksBySubmithostSerializer
+    template_name = 'benchmarks_by_submithost.html'
+
+    def list(self, request):
+        last_fetched = BenchmarksBySubmithost.objects.aggregate(Max('fetched'))['fetched__max']
+        if last_fetched is not None:
+            print(last_fetched.replace(tzinfo=None), datetime.today() - timedelta(hours=1, seconds=20))
+
+        response = super(BenchmarksViewSet, self).list(request)
+
+        # Count number of distinct sites per RecordType
+        site_counts_by_record_type = (
+            BenchmarksBySubmithost.objects
+            .values('RecordType')
+            .annotate(site_count=Count('SiteName', distinct=True))
+            .order_by('RecordType')
+        )
+
+        if type(request.accepted_renderer) is TemplateHTMLRenderer:
+            response.data = {
+                'benchmarks': response.data,
+                'last_fetched': last_fetched,
+                'site_counts_by_record_type': site_counts_by_record_type
+            }
+
+        return response