Mediawiki installation/create or upgrade shared-wiki-extensions.sh
From Biowikifarm Metawiki
< Mediawiki installation
Revision as of 13:52, 26 February 2016 by Andreas Plank (Talk | contribs) (→Bash script: samefile on $usr_share_extension_path not on samefile $usr_share_wiki_path/index.php)
Run this as a script in normal mode not with sudo and not via copy and paste!
./create_or_upgrade_shared-wiki-extensions.sh
The script does:
- extract listed extensions in the script from /usr/share/mw-wmf-clone/extensions to a shared resource /usr/share/mediawikiXXX
- list wikis depending on /usr/share/mediawikiXXX
- if the requested git_tag_or_branch is not available whatsoever the script extracts
origin/master
then - the script needs running
sudo
without login prompt
The script does not:
- manage composer extension. Read Composer (PHP) or mediawikiwiki:Composer and manage extensions in
composer.local.json
not incomposer.json
. Note that composer managed extensions are immediately active and some also need a maintenance update, e.g.cd /path/to/v-hostwiki && sudo -u www-data php ./maintenance/update.php --dbuser wikiadmin --quick --conf ./LocalSettings.php
Requirements:
- it needs an existing shared wiki resource ready to use like /usr/share/mediawikiXXX (see
create_or_upgrade_shared_wiki-core-resource-structure.sh
) - see also
write_wmf-shared_git-branches-tags2home_ext-tags-and-branches.sh
to list all available tags and branches of git-maintained extensions.
Notes:
- take care of subversion’s automatic checkout upon commit as well in
/var/lib/svn/hooks/post-commit
Extension:VisualEditor
Required an extra git clone (according to VisualEditor documentation), using the normal git archive
caused missing resources and files from it's submodule. Untested: possibly https://github.com/Kentzo/git-archive-all would be worth a try to get also submodules, but getting a shallow git clone did work:
cd /usr/share/mediawiki27wmf/extensions-simple-features
# cd /usr/share/mediawiki27wmf/extensions-rich-features
if [[ -d VisualEditor ]];then sudo mv VisualEditor VisualEditor_backup`date +%Y%m%d%H%M`; fi
sudo git clone --depth 1 --branch origin/wmf/1.27.0-wmf.9 https://gerrit.wikimedia.org/r/p/mediawiki/extensions/VisualEditor.git
cd VisualEditor && sudo git submodule update --init --recursive
cd lib/ve && sudo git checkout origin/wmf/1.27.0-wmf.9
Not sure how to handle submodules or if /usr/share/mw-wmf-clone/extensions/VisualEditor contains a valid directory structure that cause no missing resource files.
Bash script
You can make the script executable by
# u o g means: user who owns it , group users in the file's group, other users # add executable mode for the owned user and groups chmod ug+x create_or_upgrade_shared-wiki-extensions.sh
Bash script:./create_or_upgrade_shared-wiki-extensions.sh
#!/bin/bash
# Warning: run this in normal mode not as sudo and not as copy and paste in console!!
# @description: create or upgrade extensions in a shared MediaWiki resource
# @requires: running sudo without prompt
# @requires: a ready to use shared-wiki-resource e.g. in /usr/share/mediawiki26
# @requires: depending on the desired git_branch you need an
# up to date git clone of /usr/share/mw-wmf-clone/extensions
######################################################
##↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓
# # Update the git clone on biowikifarm:
# cd /usr/share/mw-wmf-clone/core; sudo git fetch;
# cd /usr/share/mw-wmf-clone/extensions && sudo git pull; sudo git submodule update --init --recursive; sudo /usr/share/mw-wmf-clone/extensions/quick-update
# cd /usr/share/mw-wmf-clone/skins; sudo git pull; sudo git submodule update --init --recursive;
#
# # OPTIONAL: list information on tags etc:
# cd /usr/share/mw-wmf-clone/core;
# git branch --remotes | sort --version-sort # list branches
# git tag --list | sort --version-sort # list tags
# cd /usr/share/mw-wmf-clone/extensions;
# git branch --remotes | sort --version-sort # list branches
# git tag --list | sort --version-sort # list tags
##↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑
##↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓
# Settings: edit the following settings at your convenience
usr_share_wiki_path="/usr/share/mediawiki27wmf"
usr_share_wiki_path="/usr/share/mediawiki26"
git_tag_or_branch="origin/REL1_26" ## OR "origin/wmf/1.27.0-wmf.7" for wmf version, OR REL1_25
# extension folder
usr_share_extension_path="${usr_share_wiki_path}/extensions-simple-features"
# usr_share_extension_path="${usr_share_wiki_path}/extensions-rich-features"
# Note extensions managed by composer get overwritten and are not managed in this script
# GraphViz
# ImageMap
# Maps
# SemanticForms
# SemanticMediaWiki
# SemanticMaps
# SemanticResultFormats
# Translate mediawiki/translate
# Translate suggest: mediawiki/babel
# UniversalLanguageSelector - mediawiki/universal-language-selector
# Translate suggest: mediawiki/translation-notifications
# Validator: mediawiki/validator
# List of EXTENSIONS can manage as follows
# ExtensionName1 (and no specific tag_or_branch -> global git_tag_or_branch is used)
# ExtensionName2, comma and tag_or_branch (this specific given tag_or_branch is used instead of global)
EXTENSIONS="
Arrays
BetaFeatures
Campaigns
CategoryTree
CentralAuth
CharInsert
CheckUser
CirrusSearch
Cite
CleanChanges
CodeReview
Collection
CollapsibleVector
CommonsMetadata
ConfirmAccount
ConfirmEdit
ContactPage
DataTransfer
DismissableSiteNotice
DynamicSidebar
Echo
EducationProgram
Elastica
ExternalData, 1.8.2
FlaggedRevs
Flow
Gadgets
GeoData
GlobalUsage
HeaderTabs, 1.0.2
InputBox
Interwiki
LiquidThreads
LocalisationUpdate
Lockdown
Loops
MassEditRegex
Math
MediaFunctions
MwEmbedSupport, origin/master
MultimediaViewer
Nuke
OAuth
OpenID
PageTools
PagedTiffHandler
ParserFunctions
PdfExport
PdfHandler
Poem
ProofreadPage
Quiz
Renameuser
ReplaceText
SemanticCompoundQueries, 0.4
SemanticDrilldown, 2.0.1
SemanticFormsInputs
SemanticInternalObjects, 0.8.1
SpamBlacklist
SyntaxHighlight_GeSHi
Thanks
TimedMediaHandler, 0.5
TitleBlacklist
TranslationNotifications
TitleKey
UploadWizard
UserMerge
Variables
VisualEditor
Widgets
WikiEditor
Wikibase
WikimediaMessages
VisualEditor
"
# just some extensions:"
EXTENSIONS="
MultimediaViewer
"
# End of settings
##↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑
# Fix
EXTENSIONS=`echo "$EXTENSIONS" | sed '/^$/d'` # remove blank lines
# Check shared MediaWiki core
if [[ ! -d ${usr_share_wiki_path} ]]; then
echo "### Shared resource does not exist: '${usr_share_wiki_path}' (stop)"
echo "### Run script create_or_upgrade_shared_wiki-core-resource-structure.sh (see biowikifarm.net)"
exit 1;
fi
# prompt User
echo "###############################################################"
[[ -d $usr_share_extension_path ]] && \
echo -e "# Upgrade shared Wiki extensions to branch \e[31m${git_tag_or_branch}\e[0m" || \
echo -e "# Create shared Wiki extensions to branch \e[31m${git_tag_or_branch}\e[0m"
echo "# /usr/share/mw-wmf-clone/extensions → $usr_share_extension_path"
echo "# Check before: up-to-date?"
echo "# Did you run sudo git pull and script quick-update in /usr/share/mw-wmf-clone/extensions?"
echo "# Check before: information on branches, tags"
echo "# Run perhaps script write_wmf-shared_git-branches-tags2home_ext-tags-and-branches.sh"
echo "# to get all available branches and tags available in /usr/share/mw-wmf-clone/extensions"
echo "# Extensions:"
echo -e "# If an extension is not available as ${git_tag_or_branch} the script uses \e[1morigin/master\e[0m then."
echo -e "# The following extensions are going to be updated (${git_tag_or_branch}):"
echo "$EXTENSIONS" | sed 's@^ *\(.\+\)[ ]*$@\1@; s@\(.\+\)[,][ ]*\(.\+\)@\1 (\2)@g; $!{ s@$@;@;}' | fmt - | sed 's@^\(.\+\)@# \1@;'
echo "# "
if [[ -d $usr_share_wiki_path ]]; then
echo "# Information: the following wikis using the same setup. So take really care of what you do ;-) ..."
echo "# (May be you need a maintenace update hereinafter as well)"
echo "# $usr_share_wiki_path"
# avoid error messages 2>/dev/null
find -L /var/www/ -maxdepth 3 -samefile $usr_share_extension_path -exec dirname '{}' ';' 2>/dev/null | sort | awk '{print "# " NR " -> " $0}'
fi
CAN_I_RUN_SUDO=$(sudo -n uptime 2>&1|grep "load"|wc -l)
if [[ ! $CAN_I_RUN_SUDO -gt 0 ]]; then
echo "# "
echo -e "# The script was stopped because it \e[1mrequires sudo\e[0m running without promt. It will fail to work now. Please run any sudo"
echo "# action before to have sudo action available without password prompt, e.g. list directory as sudo"
echo -e "# \e[1msudo ls\e[0m"
echo "# "
exit 1
fi
[[ -d $usr_share_extension_path ]] && \
echo -ne "# Do you want to upgrade (\e[32m$usr_share_extension_path\e[0m)?\n[yes or no (default: no)]: " || \
echo -ne "# Do you want to create (\e[32m$usr_share_extension_path\e[0m)?\n[yes or no (default: no)]: "
read yno
case $yno in
[yY]|[yY][Ee][Ss])
echo "Continue ..."
;;
[nN]|[nN][oO])
echo "Stop";
exit 1
;;
*) echo "Invalid or no input (stop)"
exit 1
;;
esac
#######################################
# 2. Create MediaWiki extensions in usr/share
# IMPORTANT: extensions managed by PHP script composer.phar and
# their dependencies must not be included here, presently these are:
# "mediawiki/graph-viz"
# "mediawiki/image-map"
# "mediawiki/semantic-media-wiki"
# "mediawiki/semantic-breadcrumb-links"
# "mediawiki/semantic-result-formats"
# "mediawiki/semantic-forms"
# "mediawiki/semantic-maps"
##↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓
echo "# Set ${usr_share_extension_path} to be owned by www-data (subversion post-commit demands these rights)"
if [ -d "${usr_share_extension_path}" ];then
sudo chown www-data:www-data "${usr_share_extension_path}" -R
else
echo "# Create ${usr_share_extension_path} ..."
sudo -u root -g mwadmin mkdir --parents "${usr_share_extension_path}"
sudo chown www-data:www-data "${usr_share_extension_path}" -R
fi
echo "# Update extensions in ${usr_share_extension_path} ..."
# --
# split line at ", ". If only one value, export global git_tag_or_branch, else first value is branch/tag. Example:
# SemanticMediaWiki, 1.8
# ↓ ↓
# $extension, $this_clone_version
# will export tag 1.8
# NOTE: check the line (similar to ternary operator: if condition ? dothis : doOtherThings)
# $( ... ) substitutes/returns the command's output
# ${ext%,*} check from end of $ext → delete shortest (%) or longest (%%) match
# ${ext#*,* } check from beginning of $ext → delete shortest (#) or longest (##) match
# --------------------------------------
this_ext_update_status=""
IFS=$'\n'; # IFS=script (I)nternal (F)ield (S)eparator -> set to newline
i_ext=1; n_ext=` echo "$EXTENSIONS" | wc --lines`
do_substitute_with_originMaster="" # empty or any value
for ext in $EXTENSIONS; do
# test zero string length
if [[ ! -z ${ext// /} ]];then
this_extension=$([ "${ext%,*}" == "$ext" ] && echo "$ext" || echo "${ext%%,*}");
this_clone_version=$([ "${ext%,*}" == "$ext" ] && echo "$git_tag_or_branch" || echo "${ext#*, *}");
if ! [[ -d /usr/share/mw-wmf-clone/extensions/${this_extension} ]]; then
echo -e "# Shared extension resource: \e[31m/usr/share/mw-wmf-clone/extensions/${this_extension}\e[0m \e[1mDOES NOT EXIST AS GIT SOURCE\e[0m"
echo "# Read documentation https://www.mediawiki.org/wiki/Extension:$this_extension";
echo "# check the extension's name or path or it may be an external extension (skipped)";
else
if [[ ! -d ${usr_share_extension_path}/${this_extension} ]]; then
this_ext_update_status="+";
sudo mkdir --parents "${usr_share_extension_path}/${this_extension}";
# extension folder must be owned by www-data because of the subversion post-commit update
sudo chown www-data:www-data "${usr_share_extension_path}/${this_extension}";
else
this_ext_update_status="u";
fi;
echo "# "
printf "# Shared extension resource: %03s of %03s: %1s %-30s (tag or branch: %s)\n" $i_ext $n_ext $this_ext_update_status $this_extension $this_clone_version;
if [[ -d /usr/share/mw-wmf-clone/extensions/${this_extension} ]]; then
cd "/usr/share/mw-wmf-clone/extensions/${this_extension}";
AVAILABLE_remoteBRANCH_VERSIONS=`git branch --remote | sort --version-sort`;
AVAILABLE_TAGS_VERSIONS=`git tag --list | sort --version-sort`;
##### FOLLOWING LINES CAUSE ERROR -> CHECKING code temporarily DISABLED
if [[ `echo "$AVAILABLE_remoteBRANCH_VERSIONS" | grep --count "$this_clone_version"` -eq 0 ]]; then
if [[ `echo "$AVAILABLE_TAGS_VERSIONS" | grep --count "$this_clone_version"` -eq 0 ]]; then
echo -e "# \e[31mClone version of tag or branch $this_clone_version of $this_extension is not available!!\e[0m";
echo "# Check available branches:";
echo "# cd /usr/share/mw-wmf-clone/extensions/${this_extension} && git branch --remotes | sort --version-sort";
echo "$AVAILABLE_remoteBRANCH_VERSIONS" | column -c 100;
echo "# Check available tags:";
echo "# cd /usr/share/mw-wmf-clone/extensions/${this_extension} && git tag --list | sort --version-sort";
echo "$AVAILABLE_TAGS_VERSIONS" | column -c 100;
do_substitute_with_originMaster="1"
else
do_substitute_with_originMaster=""
echo "# OK $this_clone_version is available";
fi
else
do_substitute_with_originMaster=""
echo "# OK $this_clone_version is available";
fi;
# test zero string length
if [[ -z ${do_substitute_with_originMaster// /} ]];then
# EXAMPLE: sudo git archive --prefix=AdminLinks/ HEAD | sudo -u www-data tar --extract --overwrite --directory=/usr/share/mediawiki25/extensions
# TODO not sure if VisualEditor can be used from /usr/share/mw-wmf-clone/extensions/VisualEditor
# or whether it does need an addition al git fetch in /usr/share/mediawiki27wmf/extensions
[[ "VisualEditor" == ${this_extension} ]] && \
echo "# Shared extension resource: WARNING YOU HAVE TO DO a git clone into ./extensions/VisualEditor MANUALLY !! (see web documentation)"
if [[ -d ${usr_share_extension_path}/${this_extension} ]];then
echo "# clean ${usr_share_extension_path}/${this_extension}"
sudo rm --recursive -- "${usr_share_extension_path}/${this_extension}"/*
fi
echo "# sudo git archive --prefix=\"${this_extension}/\" \"$this_clone_version\""
sudo git archive --prefix="${this_extension}/" "$this_clone_version" | sudo -u www-data tar --extract --overwrite --directory="${usr_share_extension_path}";
# VE-check fi
else
echo -e "# $this_clone_version of ${this_extension} not available, \e[31mSUBSTITUTE with origin/master\e[0m"
sudo git archive --prefix="${this_extension}/" "origin/master" | sudo -u www-data tar --extract --overwrite --directory="${usr_share_extension_path}";
fi
else
echo -e "# Warning '\e[31m/usr/share/mw-wmf-clone/extensions/${this_extension}\e[0m' not found!! Check extension name or git pull (skipped this step)";
fi;
fi;
fi;
i_ext=$((i_ext + 1))
done;
cd ..
IFS=$' \t\n'; # reset IFS=script (I)nternal (F)ield (S)eparator
##↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑
#######################################
# subversion step 2: biowikifarm extensions
cd "${usr_share_extension_path}"
echo "# Update: subversion extensions of biowikifarm to ${usr_share_extension_path}"
sudo -u www-data -g www-data svn checkout file:///var/lib/svn/LocalSVNextensions "${usr_share_extension_path}"
#######################################
echo "# NOTICE: Take care of subversion’s checkout as well in /var/lib/svn/hooks/post-commit"
echo "# to get ${usr_share_extension_path} updated upon every subversion commit"
echo "# "
echo "# Extensions managed by composer get overwritten and are active immediately on all depending wikis."
echo "# Manage those extensions in composer.local.json not in composer.json. After you have set up a correct composer file you can execute:"
echo -e "# \e[1mcd /path/to/v-hostwiki && sudo php /usr/local/bin/composer.phar update --no-dev\e[0m"
echo "# Note that some extensions need also a maintenace update (consult extension’s docu before):"
echo -e "# \e[1mcd /path/to/v-hostwiki && sudo -u www-data php ./maintenance/update.php --dbuser wikiadmin --quick --conf ./LocalSettings.php\e[0m"