fix: install vendored python 3.11 on ubuntu focal for analytics api

[muhammad-ammar]

Description: Changes in this PR have been copied from #269, #270 and #272

---

Make sure that the following steps are done before merging:

• Have a Site Reliability Engineer review the PR if you don't own all of the services impacted.
• If you are adding any new default values that need to be overridden when this change goes live, update internal repos and add an entry to the top of the CHANGELOG.
• Performed the appropriate testing.

[Copilot]

Pull Request Overview

This PR modifies the analytics API role to install Python 3.11 from vendored packages on Ubuntu Focal instead of using the standard repository installation. This is necessary because Ubuntu Focal (20.04) is out of support and Python 3.11 is not available in its default repositories.

Key changes:

• Switches from system Python 3.11 installation to vendored package installation controlled by a feature flag
• Adds tasks to download and install Python 3.11 packages from a vendored GitHub repository
• Disables the default Python 3.11 installation in the edx_django_service dependency

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File	Description
playbooks/roles/analytics_api/tasks/main.yml	Adds new tasks to conditionally install either standard or vendored Python 3.11 packages
playbooks/roles/analytics_api/meta/main.yml	Disables Python 3.11 installation in the edx_django_service role dependency
playbooks/roles/analytics_api/defaults/main.yml	Defines configuration variables for vendored Python 3.11 package installation including URLs, build versions, and dependencies

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[Copilot]

The condition logic is inverted. When use_vendored_py311 is true, the standard Python 3.11 installation task should be skipped, but this condition will execute the standard installation when vendored packages are being used. Change to when: not use_vendored_py311 | default(false) or verify the intended logic.

Suggested change

	when: not use_vendored_py311
	when: not use_vendored_py311 | default(false)

[Copilot]

Corrected spelling of 'dependant' to 'dependent'.

Suggested change

	# These must be kept topologically sorted, dependant packages last, as they
	# These must be kept topologically sorted, dependent packages last, as they

[Copilot]

Using command module with dpkg -i will always report as 'changed' even if packages are already installed. Consider using the apt module with deb parameter or add creates parameter to make the task idempotent and prevent unnecessary reinstallations on subsequent runs.

Suggested change

	command: dpkg -i "/tmp/vendored_python_{{ vendored_py311_build }}/{{ item }}"
	apt:
	deb: "/tmp/vendored_python_{{ vendored_py311_build }}/{{ item }}"

[muhammad-ammar]

@timmc-edx Please review now. I have removed the changes from the analytics_api role and moved them inside edx_django_service

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[muhammad-ammar]

@timmc-edx Please review the new changes.

[timmc-edx]

I think that should do it!

[muhammad-ammar]

@timmc-edx I am trying to build the Python3.12 for focal using the below steps mentioned in your commit but getting errors. Do I need to run the whole process on an ubuntu machine? I only have a Macbook. Could you please guide me how to build Python3.12 for focal?

I started with below steps.

Cloned https://github.com/deadsnakes/runbooks
Cloned https://github.com/deadsnakes/py3.12
Switch to https://github.com/deadsnakes/py3.12/commit/a580e16e9a46a5fb2c411933428953a682cb561f
export DEBFULLNAME="..."
export DEBEMAIL="..."
../runbooks/meta-gbp materialize focal
../runbooks/meta-gbp build

First error

muhammad.ammar@A006-01408 py3.12 % ../runbooks/meta-gbp build
+ git diff --quiet --no-ext-diff cpython
+ rm -rf dist
+ git -C cpython describe --abbrev=10 --tags
(work)+ /Users/muhammad.ammar/edx/deadsnakes/runbooks/build --git-build
*******************************************************************************
Parsing target distribution
*******************************************************************************
Unable to find image 'ghcr.io/deadsnakes/tools:latest' locally
latest: Pulling from deadsnakes/tools
docker: no matching manifest for linux/arm64/v8 in the manifest list entries.
See 'docker run --help'.
Traceback (most recent call last):
  File "/Users/muhammad.ammar/edx/deadsnakes/runbooks/build", line 78, in <module>
    raise SystemExit(main())
                     ~~~~^^
  File "/Users/muhammad.ammar/edx/deadsnakes/runbooks/build", line 41, in main
    dist = subprocess.check_output((
           ~~~~~~~~~~~~~~~~~~~~~~~^^
        TOOLS, 'dpkg-parsechangelog', '--show-field=distribution',
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )).decode().strip()
    ^^
  File "/opt/homebrew/Cellar/python@3.13/3.13.5/Frameworks/Python.framework/Versions/3.13/lib/python3.13/subprocess.py", line 472, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
           ~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
               **kwargs).stdout
               ^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.13/3.13.5/Frameworks/Python.framework/Versions/3.13/lib/python3.13/subprocess.py", line 577, in run
    raise CalledProcessError(retcode, process.args,
                             output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '('/Users/muhammad.ammar/edx/deadsnakes/runbooks/tools', 'dpkg-parsechangelog', '--show-field=distribution')' returned non-zero exit status 125.

Then I tried DOCKER_DEFAULT_PLATFORM=linux/amd64 ../runbooks/meta-gbp build and it gave me the below error

DOCKER_DEFAULT_PLATFORM=linux/amd64 ../runbooks/meta-gbp build
+ git diff --quiet --no-ext-diff cpython
+ rm -rf dist
+ git -C cpython describe --abbrev=10 --tags
(work)+ /Users/muhammad.ammar/edx/deadsnakes/runbooks/build --git-build
*******************************************************************************
Parsing target distribution
*******************************************************************************
Unable to find image 'ghcr.io/deadsnakes/tools:latest' locally
latest: Pulling from deadsnakes/tools
60d98d907669: Pull complete
2d1f0fdc97e2: Pull complete
a3a44222e8c4: Pull complete
8e9f89fda1fb: Pull complete
Digest: sha256:01ce780b82b6ee867224d9b88fe1d8febe0afc728d6f53f4ca484832d1764e9a
Status: Downloaded newer image for ghcr.io/deadsnakes/tools:latest
*******************************************************************************
Running build (results will be in ../dist)
*******************************************************************************
Traceback (most recent call last):
  File "/Users/muhammad.ammar/edx/deadsnakes/runbooks/build", line 78, in <module>
    raise SystemExit(main())
                     ~~~~^^
  File "/Users/muhammad.ammar/edx/deadsnakes/runbooks/build", line 69, in main
    *_gpg_volumes(),
     ~~~~~~~~~~~~^^
  File "/Users/muhammad.ammar/edx/deadsnakes/runbooks/build", line 29, in _gpg_volumes
    raise AssertionError('no gpg keys found?')
AssertionError: no gpg keys found?

I tried to create and add gpg keys but it didn't work so I commented out _gpg_volumes in runbooks/build but now I am getting the below error

DOCKER_DEFAULT_PLATFORM=linux/amd64 ../runbooks/meta-gbp build
+ git diff --quiet --no-ext-diff cpython
+ rm -rf dist
+ git -C cpython describe --abbrev=10 --tags
(work)+ /Users/muhammad.ammar/edx/deadsnakes/runbooks/build --git-build
*******************************************************************************
Parsing target distribution
*******************************************************************************
*******************************************************************************
Running build (results will be in ../dist)
*******************************************************************************
+ chmod 700 /root/.gnupg
chmod: cannot access '/root/.gnupg': No such file or directory

[timmc-edx]

Hmm! Interesting. I'll try building 3.12 today to see if it works on my machine.

[timmc-edx]

Yeah, works here, but I've used GPG on this machine and so it's seeing the .gnupg here. I wonder why it even checks, though... I don't see anything that's signing packages, and it would prompt for a passphrase if it were doing so.

I was able to build after commenting out a few GPG-related lines, but I don't know if it's actually safe to do so. I've filed an issue to ask about that: deadsnakes/issues#331

In any case, try keeping those lines but first run gpg --gen-key to generate a key. Does it work then?

[muhammad-ammar]

Yeah, works here, but I've used GPG on this machine and so it's seeing the .gnupg here. I wonder why it even checks, though... I don't see anything that's signing packages, and it would prompt for a passphrase if it were doing so.

I was able to build after commenting out a few GPG-related lines, but I don't know if it's actually safe to do so. I've filed an issue to ask about that: deadsnakes/issues#331

In any case, try keeping those lines but first run gpg --gen-key to generate a key. Does it work then?

Thank you so much for trying. Are you doing all of this on a MacBook? I will try this tomorrow.

[timmc-edx]

No, I have a Linux laptop -- Ubuntu Noble. So I can only build the amd64 images.

I went ahead and uploaded an amd64 Python 3.12 build here, if you could review: edx/vendored#1