Skip to content

ArtUshak/wiki_tool_python

BranchesTags

Repository files navigation

wiki_tool_python

Script to perform various tasks with websites (wikiprojects) on MediaWiki engine using MediaWiki API.

Installation

Install Python 3.12 or higher, install uv, run uv sync --no-dev.

Then you can just run uv run COMMAND to run specific commands under python virtual environment created by uv.

Installation example

Assuming Python 3.12 or higher and uv are installed.

Initialize and update virtual environment (assuming you are in the folder with this README file):

uv sync --no-dev

Run script:

uv run python -m wiki_tool_python.wikitool --help

Usage

Getting help

--help Show help message and exit. If this option is used with command, then help message for that specific command will be printed.

Setting login and password

Some commands require authentication. To tell login and password, use option --credentials LOGIN:PASSWORD or set environment variable MEDIAWIKI_CREDENTIALS to LOGIN:PASSWORD.

Throttling requests

To set interval between requests in seconds, use option --requests-interval SECONDS.

Selecting MediaWiki version

To select MediaWiki version, use option --mediawiki-version VERSION. Versions 1.31 and 1.19 are allowed, but 1.19 is not fully supported. Version 1.31 is default.

Overwriting User-Agent

To set User-Agent header, use option --user-agent STRING. User-agent is set to WikiToolPython by default.

Force login

To log in if it is not mandatory for API method, use option --login.

MediaWiki API URL

Parameter API_URL should be URL of MediaWiki API without /api.php, for example https://wow.gamepedia.com or https://absurdopedia.wiki/w.

Command list-images

python -m wiki_tool_python.wikitool list-images [OPTIONS] API_URL

List wikiproject images (titles and URLs).

For each image four lines are printed:

  1. Word FILE2.
  2. Image name on wikiproject (including namespace, for example File:Ogyglo.png).
  3. Image URL on wikiproject.
  4. Processed file name to save image.

The same format is used by download-images command.

Note: deleted images may be listed in command output.

Note: in earlier versions output format was different.

Command list-images: options

--output-file FILENAME Text file to write image list (standard output is used by default)

--api-limit INTEGER Maximum number of entries per API request (default value is 500).

--confine-encoding TEXT Encoding to confine file name to (drop characters outside that encoding)

Command list-images: example

python -m wiki_tool_python.wikitool list-images https://wow.gamepedia.com

Output:

...

FILE2
Achievement guild forgreatjusticerank2.png
https://gamepedia.cursecdn.com/wowpedia/d/d6/Achievement_guild_forgreatjusticerank2.png
02986-Achievement guild forgreatjusticerank2.png

...

Command list-category-images

python -m wiki_tool_python.wikitool list-category-images [OPTIONS] API_URL CATEGORY

List wikiproject images (titles and URLs) from specific category in the same format as list-images.

Command list-category-images: options

--output-file FILENAME Text file to write image list (standard output is used by default)

--api-limit INTEGER Maximum number of entries per API request (default value is 500).

--api-image-ids-limit INTEGER Maximum number of image IDs per API request (default value is 50).

--confine-encoding TEXT Encoding to confine file name to (drop characters outside that encoding)

Command list-category-images: example

python -m wiki_tool_python.wikitool list-category-images https://wow.gamepedia.com "Category:Images with watermarks"

Output:

...

FILE2
Quartermaster Alcorn - Beta.jpg
https://gamepedia.cursecdn.com/wowpedia/2/2a/Quartermaster_Alcorn_-_Beta.jpg
00897-Quartermaster Alcorn - Beta.jpg

...

Command list-pages

python -m wiki_tool_python.wikitool list-pages [OPTIONS] API_URL

List wikiproject page names of all wikiproject namespaces.

Each page name is written on separate line.

Note: deleted pages may be listed in command output (we have not checked for this yet).

Command list-pages: options

--output-file FILENAME Text file to write page names (standard output is used by default)

--api-limit INTEGER Maximum number of entries per API request (default value is 500).

Command list-pages: example

python -m wiki_tool_python.wikitool list-pages https://wow.gamepedia.com

Output:

...

Talk:Alchemist (Warcraft III)
Talk:Alchemist Finklestein
Talk:Alchemist Pestlezugg
Talk:Alchemist Stone
Talk:Alchemy
Talk:Alchemy Lab
Talk:Alchemy proficiencies
Talk:Alchemy recipes
Talk:Alchemy recipes/Grand Master
Talk:Alchemy trainers)

...

Command list-namespace-pages

python -m wiki_tool_python.wikitool list-namespace-pages [OPTIONS] API_URL NAMESPACE

List wikiproject page names of namespace (NAMESPACE should be integer ID of namespace).

Each page name is written on separate line.

Note: deleted pages may be listed in command output (we have not checked for this yet).

Command list-namespace-pages: options

--output-file FILENAME Text file to write page names (standard output is used by default)

--api-limit INTEGER Maximum number of entries per API request (default value is 500).

Command list-namespace-pages: example

python -m wiki_tool_python.wikitool list-namespace-pages https://wow.gamepedia.com 1

Output:

...

Talk:"Cookie" McWeaksauce
Talk:"Dirty" Larry
Talk:"Dirty" Michael Crowe
Talk:"Let the Fires Come!"/Notes
Talk:"Let the Fires Come!" (Alliance)
Talk:"Let the Fires Come!" (Horde)
Talk:"Red" Jack Findle
Talk:'Tis the Season
Talk:1.11.0 API changes (Iriel)
Talk:1.8.4
Talk:10-day Free Trial
Talk:19 Pound Catfish
Talk:1st Legion

...

Command list-deletedrevs

python -m wiki_tool_python.wikitool list-deletedrevs [OPTIONS] OUTPUT_DIRECTORY API_URL

List deleted revisions from wikiproject in JSON format, output may be splitted to many JSON files, files are saved to OUTPUT_DIRECTORY.

Note: this command requires authentication.

Note: files will be named: entry-N.json, where N is number of file, starting from 0.

Command list-deletedrevs: options

--all-namespaces BOOLEAN Boolean value, TRUE to list for all namespaces, FALSE for main namespace only.

--file-entry-num INTEGER Number of entries per JSON file.

--api-limit INTEGER Maximum number of entries per API request (default value is 500).

Command delete-pages

python -m wiki_tool_python.wikitool delete-pages [OPTIONS] FILTER_EXPRESSION API_URL

Delete pages matching regular expression FILTER_EXPRESSION.

Note: this command requires authentication.

Note: this command deletes pages, use it with caution!

Note: regular expressions are parsed with module re of python standard library. See re documentation for details.

Command delete-pages: options

--reason TEXT Deletion reason.

--exclude-expression TEXT Additional expression to exclude pages matching it from deletion.

--first-page TEXT First page to delete (start deletion from it).

--first-page-namespace INTEGER Namespace of first page to delete.

--api-limit INTEGER Maximum number of entries per API request (default value is 500).

--namespace INTEGER Namespace to search pages for deletion (this option can be used multiple times to add multiple namespaces).

Command edit-pages

python -m wiki_tool_python.wikitool edit-pages [OPTIONS] FILTER_EXPRESSION NEW_TEXT API_URL

Edit pages matching regular expression FILTER_EXPRESSION, replacing their content with new text NEW_TEXT.

Note: this command requires authentication.

Note: this command edits pages, use it with caution!

Note: regular expressions are parsed with module re of python standard library. See re documentation for details.

Command edit-pages: options

--reason TEXT Edit reason.

--exclude-expression TEXT Additional expression to exclude pages matching it from editing.

--first-page TEXT First page to edit (start editing from it).

--first-page-namespace INTEGER Namespace of first page to edit.

--api-limit INTEGER Maximum number of entries per API request (default value is 500).

--namespace INTEGER Namespace to search pages for deletion (this option can be used multiple times to add multiple namespaces).

Command edit-pages-clone-interwikis

python -m wiki_tool_python.wikitool edit-pages-clone-interwikis [OPTIONS] API_URL OLD NEW

Add interwiki NEW to pages that contain interwiki OLD, but do not contain interwiki NEW yet.

For example, if NEW is en-gb, and OLD is en, than:

[[en:Jaina Proudmoore]]

will be replaced with:

[[en:Jaina Proudmoore]]
[[en-gb:Jaina Proudmoore]]

This command may be used, for example, if your wiki contains another language version, and this version has forked, and you need to add interwikis to fork to pages that already contain old version.

Note: this command requires authentication.

Note: this command edits pages, use it with caution!

Command edit-pages-clone-interwikis: options

--reason TEXT Edit reason.

--exclude-expression TEXT Additional expression to exclude pages matching it from editing.

--first-page TEXT First page to edit (start editing from it).

--first-page-namespace INTEGER Namespace of first page to edit.

--api-limit INTEGER Maximum number of entries per API request (default value is 500).

Command replace-links

python -m wiki_tool_python.wikitool replace-links [OPTIONS] API_URL OLD NEW

Replace links to page OLD by links to page NEW.

For example, if NEW is Wikiproject administrator, and OLD is Administrator, than:

Arthur Brahmaguptov was [[administrator]] of [[Encyclopedia Dramatica]].

will be replaced with:

Arthur Brahmaguptov was [[Wikiproject administrator|administrator]] of [[Encyclopedia Dramatica]].

Note: this command requires authentication.

Note: this command edits pages, use it with caution!

Command replace-links: options

--reason TEXT Edit reason.

--api-limit INTEGER Maximum number of entries per API request (default value is 500).

Command: replace-links: example

python -m wiki_tool_python.wikitool replace-links http://wikireality.ru/w Администратор "Администратор википроекта"

Command download-images

python -m wiki_tool_python.wikitool download-images [OPTIONS] LIST_FILE DOWNLOAD_DIR

Download images listed in file LIST_FILE (file should be in the same format as output of list-images command) to directory DOWNLOAD_DIR.

Each image is downloaded from URL given in file and saved to file with name given in file (for example, File:Ogyglo.png may be saved as 00500-Ogyglo.png).

Note: progress bar is displayed.

Note: if image can not be downloaded, warning is printed, but process is not stopped.

Command download-images: options

No specific options.

Command download-images: example

python -m wiki_tool_python.wikitool download-images images.txt wiki_images

Where images.txt is list of images.

Output:

Progress bar is displayed.

Images are downloaded to directory wiki_images.

Command upload-image

python -m wiki_tool_python.wikitool upload-image [OPTIONS] FILE_NAME FILE API_URL

Upload image FILE to wiki with new name FILE_NAME.

Note: this command requires authentication.

Command upload-image: options

No specific options.

Command upload-image: example

python -m wiki_tool_python.wikitool upload-image NVGogol.png Pic9231i9149i.png https://absurdopedia.wiki/w

Command upload-images

python -m wiki_tool_python.wikitool upload-images [OPTIONS] LIST_FILE DOWNLOAD_DIR API_URL

Upload images listed in file LIST_FILE (file should be in the same format as output of list-images command) from directory DOWNLOAD_DIR to wiki.

Note: this command requires authentication.

Note: progress bar is displayed.

Note: if image can not be upload, warning is printed, but process is not stopped.

Command upload-images: options

--skip-nonexistent/--no-skip-nonexistent Do not fail on non-existent files, skip them instead

Command upload-images: example

python -m wiki_tool_python.wikitool upload-images images.txt wiki_images https://absurdopedia.wiki/w

Where images.txt is list of images.

Output:

Progress bar is displayed.

Command votecount

python -m wiki_tool_python.wikitool votecount [OPTIONS] API_URL USER_LIST_FILE

Get user edit count for users from file USER_LIST_FILE and calculate vote power (according to Wikireality rules).

Command votecount: options

--api-limit INTEGER Maximum number of entries per API request (default value is 500).

--namespacefile FILENAME JSON file to read namespaces data from.

--start [%Y-%m-%d|%Y-%m-%dT%H:%M:%S|%Y-%m-%d %H:%M:%S] Start date for counting edits.

--end [%Y-%m-%d|%Y-%m-%dT%H:%M:%S|%Y-%m-%d %H:%M:%S] End date for counting edits.

--output-format [txt|mediawiki|json] Output data format (text, MediaWiki table or JSON).

--redirect-regex-text TEXT Regular expression to detect redirect creation.

Command votecount: example

Namespace file namespaces.json is:

{
    "edit_weights": {
        "0": 0.04,
        "1": 0.003,
        "4": 0.015,
        "5": 0.003,
        "6": 0.03,
        "7": 0.003,
        "8": 0.015,
        "9": 0.003,
        "10": 0.015,
        "11": 0.003,
        "14": 0.015,
        "15": 0.003
    },
    "page_weights": {
        "0": 0.6
    }
}

User list file ../data/votecount/voters.txt is:

Arbnos
Arsenal
Cat1987
python -m wiki_tool_python.wikitool --mediawiki-version 1.19 votecount --start 2018-07-12 --end 2018-10-12 --namespacefile ../data/votecount/namespaces.json --output-format txt http://wikireality.ru/w ../data/votecount/voters.txt

Output:

Processing user Arbnos...
Processing user Arsenal...
Processing user Cat1987...
User Arbnos
N0: 0
N1: 0
N4: 0
N5: 0
N6: 0
N7: 0
N8: 0
N9: 0
N10: 0
N11: 0
N14: 0
N15: 0
NewPages: 0
VotePower: 0.0

User Arsenal
N0: 20
N1: 0
N4: 9
N5: 0
N6: 0
N7: 0
N8: 0
N9: 0
N10: 2
N11: 0
N14: 0
N15: 0
NewPages: 2
VotePower: 2.165

User Cat1987
N0: 30
N1: 0
N4: 1
N5: 0
N6: 3
N7: 0
N8: 1
N9: 0
N10: 0
N11: 0
N14: 0
N15: 0
NewPages: 0
VotePower: 1.32

...

Command list-directory-pages

python -m wiki_tool_python.wikitool list-directory-pages [OPTIONS] INPUT_DIRECTORY OUTPUT_FILE

Write list of .txt file paths in directory to JSON file.

Directories are handled recursively.

Command list-directory-pages: example

There are files ../data/wiki-pages/Test.txt and ../data/wiki-pages/Test/Test1.txt.

python -m wiki_tool_python.wikitool list-directory-pages ../data/wiki-pages/ ../data/wiki-pages-list.json

File ../data/wiki-pages-list.json will contain the following text:

[
    "Test.txt",
    "Test/Test1.txt"
]

Command list-directory-pages: options

No specific options.

Command upload-pages

python -m wiki_tool_python.wikitool upload-pages [OPTIONS] API_URL INPUT_DIRECTORY LIST_FILE

Create or overwrite pages from files in input directory listed in list file.

Files with extension are uploaded as pages, page title for file is prefix concatenated with file path (as written in list file) without extension.

Note: this command requires authentication.

Note: list file format is either JSON list (same as of file generated by list-directory-pages command) or JSON dictionary (keys are page pathes, values are page titles).

Command upload-pages: example 1

There are files ../data/wiki-pages/Test.txt and ../data/wiki-pages/Test/Test1.txt.

File ../data/wiki-pages-list.json contains the following text:

[
    "Test.txt",
    "Test/Test1.txt"
]

Command is:

python -m wiki_tool_python.wikitool upload-pages --prefix="User:ArtushakBot/" --summary="Test page upload" https://genepedia.ru ../data/wiki-pages/ ../data/wiki-pages/wiki-pages-list.json

The following pages will be created:

  • User:ArtushakBot/Test with text from ../data/wiki-pages/Test.txt
  • User:ArtushakBot/Test/Test1 with text from ../data/wiki-pages/Test/Test1.txt

Command upload-pages: example 2

There are files ../data/wiki-pages/1.txt and ../data/wiki-pages/2.txt.

File ../data/wiki-pages-list.json contains the following text:

{
    "1.txt": "Page 1",
    "2.txt": "Page 2"
}

Command is:

python -m wiki_tool_python.wikitool upload-pages --dictionary --prefix="User:ArtushakBot/" --summary="Test page upload" https://genepedia.ru ../data/wiki-pages/ ../data/wiki-pages/wiki-pages-list.json

The following pages will be created:

  • User:ArtushakBot/Page 1 with text from ../data/wiki-pages/Page 1
  • User:ArtushakBot/Page 2 with text from ../data/wiki-pages/Page 2

Command upload-pages: options

--dictionary / --no-dictionary Use dictionary file of pathes and titles instead of file list.

--extended-dictionary / --no-extended-dictionary Read dictionary values as dictionaries with title and path keys.

--prefix TEXT Prefix for page titles.

--summary TEXT Edit summary.

--first-page INTEGER Page number to start with.

--mode [append|overwrite] Whether to append text from file to old text on existing pages, or overwrite old text with text from file.

Command generate-import-script

python -m wiki_tool_python.wikitool list-directory-pages [OPTIONS] LIST_FILE INPUT_DIRECTORY OUTPUT_SCRIPT_FILE LOG_FILE

Write bash script to import pages from list file using MediaWiki maintenance script importTextFiles.php (see MediaWiki help). It can be uploaded to server along with page files and then run to import to them to website to avoid API overhead.

LIST_FILE is path to page list file on local machine. INPUT_DIRECTORY is parent path for page text files on server, path should be absolute or relative to script working directory.

LOG_FILE is path to output text log file on server, path should be absolute or relative to script working directory.

Note: server access (for example, SSH) is required for this method of page uploading.

Command generate-import-script: options

--prefix TEXT Prefix for page titles.

--rc / --no-rc Add --rc option to output script.

--bot / --no-bot Add --bot option to output script.

--user TEXT Add -u option to output script.

--summary TEXT Add -s option to output script.

--maintenance-directory TEXT Directory with importTextFiles.php script on server, path should be absolute or relative to script working directory.

--show-progress-bar / --no-show-progress-bar Add progress bar display to output script

--first-page INTEGER First page number to add

Command generate-import-script: example

There are files ../data/wiki-pages/Test.txt and ../data/wiki-pages/Test/Test1.txt.

File ../data/wiki-pages-list.json contains the following text:

[
    "Test.txt",
    "Test/Test1.txt"
]

Command is:

python -m wiki_tool_python.wikitool generate-import-script --prefix="User:ArtushakBot/" --summary="Test page upload" --bot --maintenance-directory="/srv/http/mediawiki/w/maintenance" ../data/wiki-pages/wiki-pages-list.json ./data/wiki-pages ../data/importPages.sh ./data/log.txt

File ../data/importPages.sh will contain following text:

#!/bin/bash

php /srv/http/mediawiki/w/maintenance/importTextFiles.php --rc --bot -s 'Test page upload' --prefix User:ArtushakBot/ data/wiki-pages/Test.txt >> ./data/log.txt 2>&1 
echo -ne '[##########          ]         1 /         2\r'
php /srv/http/mediawiki/w/maintenance/importTextFiles.php --rc --bot -s 'Test page upload' --prefix User:ArtushakBot/Test/ data/wiki-pages/Test/Test1.txt >> ./data/log.txt 2>&1 
echo -ne '[####################]         2 /         2\r'

Main example

uv sync --no-dev
mkdir ../absurdopedia
mkdir ../absurdopedia/download
uv run python -m wiki_tool_python.wikitool list-images --output-file ../absurdopedia/image.txt https://absurdopedia.wiki/w
uv run python -m wiki_tool_python.wikitool download-images ../absurdopedia/image.txt ../absurdopedia/download

Files

  • mediawiki.py contains exceptions and classes to interact with MediaWiki API. Abstract base class is named MediaWikiAPI, implementations for specific MediaWiki versions are subclassed from it.
  • wikitool.py contains commands described in this file. To parse them, Click is used.

Special thanks

  • MediaWiki developers.
  • Python developers.
  • uv developers.
  • requests developers.
  • ruff developers.
  • Click developers.
  • Рыцарь (Knight) aka Riddari aka Тэйтанка-птекила (Teitanka-ptekila).
  • Wikireality users, especially members of Dimetr's Telegram chat.
  • Other folks from Absurdopedia, Russian version of Uncyclopedia (absurdopedia.wiki and absurdopedia.net)
  • Other cool folks.

About

Script for exporting data from MediaWiki API

Topics

python api-client mediawiki-api wikiproject

Resources

Readme

License

MIT license
Activity

Stars

9 stars

Watchers

0 watching

Forks

3 forks
Report repository

Releases 8

v0.3.0 Latest
Dec 7, 2025
+ 7 releases

Packages

No packages published

Contributors 2

  •  
  •  

Languages