[Bug]: Update Beam containers to numpy 2.x

[tvalentyn]

What happened?

Currently, users who pip install apache-beam[gcp] install a newer version of numpy at job submission than is installed in container images. This causes a misconfiguration for Dataframe api users. From: https://lists.apache.org/thread/3k3rpnoh1tjf7d9rhvl88lmrn04fr9cn.

One of the jobs I ran (Java multi-lang that uses Python Dataframe) failed with the following error.

ModuleNotFoundError: No module named 'numpy._core.numeric'

Indeed in

beam/sdks/python/container/py311/base_image_requirements.txt

Line 106 in 77f38cd

numpy==1.26.4

we have numpy 1.x. We should try to upgrade the containers to use numpy 2.x for Python versions that support it. We should investigate what dependency is preventing Python 3.10+ containers from picking up numpy 2.x

Issue Priority

Priority: 1 (data loss / total loss of function)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Infrastructure
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[tvalentyn]

Currently numpy is downgraded due to the following:

pandas==2.1.4
├── numpy [required: >=1.22.4,<2, installed: 1.26.4]

[tvalentyn]

We should support a newer version of Pandas.

[liferoad]

Couple of related PRs: #33627 #33638