Skip to content

DAAS-236 #103

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
CrimsonGlory wants to merge 6 commits into
base: master
Choose a base branch
from
Open

DAAS-236 #103

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
ef5d297
Implement immediate persistence of Sample metadata after subfile uplo…
CrimsonGlory Jan 26, 2026
495ae1c
Update CircleCI config to remove unnecessary Docker installation step
CrimsonGlory Jan 26, 2026
2f84e2a
Update certificates inside container
CrimsonGlory Jan 26, 2026
246f08e
test
CrimsonGlory Jan 27, 2026
d9756be
try with a newer version of winetricks
CrimsonGlory Jan 27, 2026
0429a37
install microsoft cert
CrimsonGlory Jan 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .circleci/config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ jobs:
working_directory: ~/daas
steps:
- checkout
- docker/install-docker
# Docker is pre-installed on machine executors, no need to install
- docker/install-docker-compose
- run: docker --version
- run: docker compose version
Expand Down
16 changes: 16 additions & 0 deletions meta_extractor/redis/task.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
# from .requeue import TaskRequeuer
from .queue import TaskQueue
from ..sample import Sample
from .. import api_connector


class Task:
Expand Down Expand Up @@ -60,4 +61,19 @@ def sample_found(self) -> bool:
def split_into_subtasks_per_subfile(self) -> None:
for subfile in self.sample.subfiles:
subfile_seaweedfs_file_id = seaweedfs.upload_file(stream=subfile.content, name=subfile.file_name)
# Set the seaweedfs_file_id so it's included in metadata
subfile.seaweedfs_file_id = subfile_seaweedfs_file_id
# Immediately persist the Sample to Django
try:
response_data = {
'force_reprocess': self.force_reprocess,
'callback': self.callback,
'sample': subfile.metadata
}
api_connector.send_result(self.api_url, response_data)
logging.info(f'Persisted Sample immediately after upload: sha1={subfile.sha1}, seaweedfs_file_id={subfile_seaweedfs_file_id}')
except Exception as e:
logging.error(f'Failed to persist Sample immediately after upload: {e}')
# Continue anyway - the task will be processed and Sample will be created later
# Then queue the task for processing
TaskQueue().add_subfile_to_queue(self.settings, subfile_seaweedfs_file_id)
18 changes: 14 additions & 4 deletions peWorkerDockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,14 +15,17 @@ ENV WINEARCH=win32
# Generic
RUN apt-get update && \
apt-get install --no-install-recommends -y build-essential apt-transport-https && \
apt-get install --no-install-recommends -y gnutls-bin \
apt-get install --no-install-recommends -y ca-certificates \
openssl \
gnutls-bin \
host \
unzip \
xauth \
xvfb \
zenity \
zlib1g \
zlib1g-dev && \
update-ca-certificates && \
rm -rf /var/lib/apt/lists/* && \
apt-get clean

Expand All @@ -42,12 +45,19 @@ RUN dpkg --add-architecture i386 && \
rm -rf /var/lib/apt/lists/* && \
apt-get clean

# Add Microsoft TLS G2 ECC CA OCSP 02.crt to ca-certificates
# Without this wget https://download.microsoft.com fails due to cert error.
RUN wget --user-agent="Mozilla/5.0 (X11; Linux x86_64)" https://www.microsoft.com/pkiops/certs/Microsoft%20TLS%20G2%20ECC%20CA%20OCSP%2002.crt && \
openssl x509 -inform DER -in "Microsoft TLS G2 ECC CA OCSP 02.crt" -out microsoft-tls-g2-ecc-ocsp-02.crt && \
cp microsoft-tls-g2-ecc-ocsp-02.crt /usr/local/share/ca-certificates/ && \
update-ca-certificates


# C#: Winetricks, dotnet45, vcrun2010
RUN echo "Installing winetricks" && \
wget -nc -nv https://github.com/Winetricks/winetricks/archive/20210825.zip -O /tmp/winetricks.zip && \
wget -nc -nv https://github.com/Winetricks/winetricks/archive/20260125.zip -O /tmp/winetricks.zip && \
unzip /tmp/winetricks.zip -d /tmp/winetricks/ && \
make -C /tmp/winetricks/winetricks-20210825 install && \
make -C /tmp/winetricks/winetricks-20260125 install && \
rm -rf /tmp/winetricks.zip && \
rm -rf /tmp/winetricks && \
mkdir -p /home/root/.cache/winetricks/ && \
Expand All @@ -58,7 +68,7 @@ COPY winetricks/corefonts/ /home/root/.cache/winetricks/corefonts/

# Dotnet45 should be installed on a different docker step. Otherwise, it will fail.
RUN echo "Installing Dotnet45" && \
winetricks -q dotnet45 corefonts && \
winetricks -q --force dotnet45 corefonts && \
echo "Dotnet45 installed"
RUN xvfb-run winetricks -q vcrun2010 && \
echo "vcrun2010 installed"
Expand Down