Mediawiki installation/create or upgrade shared-wiki-extensions.sh

From Biowikifarm Metawiki
< Mediawiki installation
Revision as of 15:58, 25 October 2017 by Andreas Plank (Talk | contribs) (Widgets warning; +SemanticImageInput)

Jump to: navigation, search

Run this as a script in normal mode not with sudo and not via copy and paste! You can adjust the extension list to your needs and run it from your home directory like this: 

./create_or_upgrade_shared-wiki-extensions.sh

The script does:

  • extract specified extensions from the script header from /usr/share/mw-wmf-clone/extensions to a shared resource /usr/share/mediawikiXXX
  • informs about wikis depending on /usr/share/mediawikiXXX before running
  • if the requested git_tag_or_branch is not available whatsoever the script switches to origin/master then
  • the script attempts to make a bakup of an extension, in case git archive fails to extract it restores the backup. On successful extraction the backup is removed.
  • the script needs running sudo without login prompt

The script does not:

  • manage composer extension. Read Composer (PHP) or mediawikiwiki:Composer and manage extensions in composer.local.json not in composer.json. Note that composer managed extensions are immediately active and some also need a maintenance update, e.g.
    cd /path/to/v-hostwiki && sudo -u www-data php ./maintenance/update.php --dbuser wikiadmin --quick --conf ./LocalSettings.php

Requirements:

Issues:

  • take care of subversion’s automatic checkout upon commit as well in /var/lib/svn/hooks/post-commit
  • check out category:Patch if there are patches needed for some extensions

Extension:ConfirmAccount

Extension:OpenID

Needs extra OpenID PHP software classes 

cd /usr/share/mediawiki26/extensions-rich-features
sudo git clone git://github.com/openid/php-openid.git
sudo mv php-openid/Auth/ Auth
sudo rm -r php-openid

Extension:SemanticMediaWiki

It is actually managed by composer and not this script. After install

Extension:VisualEditor

Required an extra git clone (according to VisualEditor documentation), using the normal git archive caused missing resources and files from it's submodule. Untested: possibly https://github.com/Kentzo/git-archive-all would be worth a try to get also submodules. Getting git clone according to VisualEditor documentation, but a shallow one, did work: 

cd /usr/share/mediawiki27wmf/extensions-simple-features
# cd /usr/share/mediawiki27wmf/extensions-rich-features
if [[ -d VisualEditor ]];then sudo mv VisualEditor VisualEditor_backup`date +%Y%m%d%H%M`; fi
sudo git clone --depth 1 --branch origin/wmf/1.27.0-wmf.9 https://gerrit.wikimedia.org/r/p/mediawiki/extensions/VisualEditor.git
cd VisualEditor && sudo git submodule update --init --recursive
cd lib/ve && sudo git checkout origin/wmf/1.27.0-wmf.9

Not sure how to handle submodules or if /usr/share/mw-wmf-clone/extensions/VisualEditor contains a valid directory structure that cause no missing resource files.

Extension:Widgets

See Extension:Widgets on Mediawiki_installation/REL1_26.

Bash script

Copy the script to your home directory. Then you make the script executable by 

# u o g a means: user who owns it , group users in the file's group, other users, all users
# add executable mode for the owned user and groups
# executable, writeable only for the user who owns it and all let all others just read
chmod u=rwx,g=r,o=r create_or_upgrade_shared-wiki-extensions.sh

Steps to consider before running this script:

  1. check the git tag or branch of the extensions you need
  2. modify the script in the settings section accordingly
    • the global git_tag_or_branch of the script
    • the full list in EXTENSIONS or just a small one (uncomment below the full EXTENSIONS list)
    • the usr_share_extension_path (shared extension resources)
  3. in case of update:
    • consider that some wikis may already be dependent on the extension you want to update
    • depending on the update changes, you may need to run a v-host-wiki update as well 
      cd /var/www/path/to/v-host/wiki && sudo -u www-data php ./maintenance/update.php --dbuser wikiadmin --quick --conf ./LocalSettings.php

Run the bash script:
./create_or_upgrade_shared-wiki-extensions.sh 

#!/bin/bash
# Warning: run this in normal mode not as sudo and not as copy and paste in console!!
# @description: create or upgrade extensions in a shared MediaWiki resource
# @requires: running sudo without prompt
# @requires: a ready to use shared-wiki-resource e.g. in /usr/share/mediawiki26
# @requires: depending on the desired git_branch you need an
#   up to date git clone of /usr/share/mw-wmf-clone/extensions
######################################################

##↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓
#  # Update the git clone on biowikifarm:
#  cd /usr/share/mw-wmf-clone/core; sudo git fetch;
#  cd /usr/share/mw-wmf-clone/extensions && sudo git pull; sudo git submodule update --init --recursive; sudo /usr/share/mw-wmf-clone/extensions/quick-update
#  cd /usr/share/mw-wmf-clone/skins; sudo git pull; sudo git submodule update --init --recursive;
#
#  # OPTIONAL: list information on tags etc:
#  cd /usr/share/mw-wmf-clone/core;
#    git branch --remotes | sort --version-sort # list branches
#    git tag --list | sort --version-sort # list tags
#  cd /usr/share/mw-wmf-clone/extensions;
#    git branch --remotes | sort --version-sort # list branches
#    git tag --list | sort --version-sort # list tags
##↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑

##↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓
# Settings: edit the following settings at your convenience
usr_share_wiki_path="/usr/share/mediawiki27wmf"
usr_share_wiki_path="/usr/share/mediawiki26"
git_tag_or_branch="origin/REL1_26" ## OR "origin/wmf/1.27.0-wmf.7" for wmf version, OR REL1_25
# extension folder
usr_share_extension_path="${usr_share_wiki_path}/extensions-simple-features"
# usr_share_extension_path="${usr_share_wiki_path}/extensions-rich-features"

# Note extensions managed by composer get overwritten and are not managed in this script
#   GraphViz
#   ImageMap
#   Maps
#   SemanticForms
#   SemanticMediaWiki
#   SemanticMaps
#   SemanticResultFormats
#   Translate mediawiki/translate
#   Translate suggest: mediawiki/babel
#   UniversalLanguageSelector - mediawiki/universal-language-selector
#   Translate suggest: mediawiki/translation-notifications
#   Validator: mediawiki/validator

# List of EXTENSIONS can manage as follows
# ExtensionName1 (and no specific tag_or_branch -> global git_tag_or_branch is used)
# ExtensionName2, comma and tag_or_branch (this specific given tag_or_branch is used instead of global)
EXTENSIONS="
Arrays
BetaFeatures
Campaigns
CategoryTree
CentralAuth
CharInsert
CheckUser
CirrusSearch
Cite
CleanChanges
CodeReview
Collection
CollapsibleVector
CommonsMetadata
ConfirmAccount
ConfirmEdit
ContactPage
DataTransfer
DeleteBatch
DismissableSiteNotice
DynamicSidebar
Echo
EducationProgram
Elastica
ExternalData, 1.8.2
FlaggedRevs
Flow
Gadgets
GeoData
GlobalUsage
HeaderTabs, 1.0.2
HTMLets
InputBox
Interwiki
LiquidThreads
LocalisationUpdate
Lockdown
Loops
MassEditRegex
Math
MediaFunctions
MobileFrontend
MwEmbedSupport, origin/master
MultimediaViewer
Nuke
OAuth
OpenID
PageTools
PagedTiffHandler
ParserFunctions
PdfExport
PdfHandler
Poem
ProofreadPage
Quiz
Renameuser
ReplaceText
SemanticCompoundQueries, 0.4
SemanticDrilldown, 2.0.1
SemanticFormsInputs
SemanticImageInput
SemanticInternalObjects, 0.8.1
SpamBlacklist
SyntaxHighlight_GeSHi
Thanks
TimedMediaHandler, 0.5
TitleBlacklist
TranslationNotifications
TitleKey
UploadWizard
UserMerge
Variables
VisualEditor
Widgets
WikiEditor
Wikibase
WikimediaMessages
VisualEditor
"
# just some extensions:"
# EXTENSIONS="
# DeleteBatch
# "
# End of settings
##↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑

# Fix
EXTENSIONS=`echo "$EXTENSIONS" | sed '/^$/d'` # remove blank lines

# Check shared MediaWiki core
if [[ ! -d ${usr_share_wiki_path} ]]; then
  echo "### Shared resource does not exist: '${usr_share_wiki_path}' (stop)"
  echo "### Run script create_or_upgrade_shared_wiki-core-resource-structure.sh (see biowikifarm.net)"
  exit 1;
fi

# prompt User
echo    "###############################################################"

[[ -d $usr_share_extension_path ]] && \
echo -e "# Upgrade shared Wiki extensions to branch \e[31m${git_tag_or_branch}\e[0m" || \
echo -e "# Create shared Wiki extensions to branch \e[31m${git_tag_or_branch}\e[0m"

echo    "#   /usr/share/mw-wmf-clone/extensions → $usr_share_extension_path"
echo    "# Check before: up-to-date?"
echo    "#   Did you run sudo git pull and script quick-update in /usr/share/mw-wmf-clone/extensions?"
echo    "# Check before: information on branches, tags"
echo    "#   Run perhaps script write_wmf-shared_git-branches-tags2home_ext-tags-and-branches.sh"
echo    "#   to get all available branches and tags available in /usr/share/mw-wmf-clone/extensions"
echo    "# Extensions:"
echo -e "#   If an extension is not available as ${git_tag_or_branch} the script uses \e[1morigin/master\e[0m then."
echo -e "#   The following extensions are going to be updated (${git_tag_or_branch}):"
echo "$EXTENSIONS"  | sed 's@^ *\(.\+\)[ ]*$@\1@; s@\(.\+\)[,][ ]*\(.\+\)@\1 (\2)@g; $!{ s@$@;@;}' | fmt - | sed 's@^\(.\+\)@#     \1@;'
echo     "# "
if [[ -d $usr_share_wiki_path ]]; then
echo     "# Information: the following wikis using the same setup. So take really care of what you do ;-) ..."
echo     "# (May be you need a maintenace update hereinafter as well)"
echo     "#   $usr_share_wiki_path"
  # avoid error messages 2>/dev/null
  find -L /var/www/ -maxdepth 3 -samefile $usr_share_extension_path -exec dirname '{}' ';' 2>/dev/null | sort | awk '{print "#   " NR " -> " $0}'
fi

CAN_I_RUN_SUDO=$(sudo -n uptime 2>&1|grep "load"|wc -l)
if [[ ! $CAN_I_RUN_SUDO -gt 0 ]]; then
echo     "# "
echo -e  "# The script was stopped because it \e[1mrequires sudo\e[0m running without promt. It will fail to work now. Please run any sudo"
echo     "# action before to have sudo action available without password prompt, e.g. list directory as sudo"
echo -e  "#   \e[1msudo ls\e[0m"
echo     "# "
exit 1
fi

[[ -d $usr_share_extension_path ]] && \
echo -ne "# Do you want to upgrade (\e[32m$usr_share_extension_path\e[0m)?\n[yes or no (default: no)]: " || \
echo -ne "# Do you want to create (\e[32m$usr_share_extension_path\e[0m)?\n[yes or no (default: no)]: "
read yno
case $yno in
  [yY]|[yY][Ee][Ss])
    echo "Continue ..."
  ;;
  [nN]|[nN][oO])
    echo "Stop";
    exit 1
  ;;
  *) echo "Invalid or no input (stop)"
    exit 1
  ;;
esac

#######################################
# 2. Create MediaWiki extensions in usr/share
# IMPORTANT: extensions managed by PHP script composer.phar and
# their dependencies must not be included here, presently these are:
# "mediawiki/graph-viz"
# "mediawiki/image-map"
# "mediawiki/semantic-media-wiki"
# "mediawiki/semantic-breadcrumb-links"
# "mediawiki/semantic-result-formats"
# "mediawiki/semantic-forms"
# "mediawiki/semantic-maps"

##↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓
echo "# Set ${usr_share_extension_path} to be owned by www-data (subversion post-commit demands these rights)"
if [ -d "${usr_share_extension_path}" ];then
  sudo chown www-data:www-data "${usr_share_extension_path}" -R
else
  echo "# Create ${usr_share_extension_path} ..."
  sudo -u root -g mwadmin mkdir --parents "${usr_share_extension_path}"
  sudo chown www-data:www-data "${usr_share_extension_path}" -R
fi
echo "# Update extensions in ${usr_share_extension_path} ..."

# --
# split line at ", ". If only one value, export global git_tag_or_branch, else first value is branch/tag. Example:
#   SemanticMediaWiki, 1.8
#      ↓                   ↓
#   $extension, $this_clone_version
#   will export tag 1.8
# NOTE: check the line (similar to ternary operator: if condition ? dothis : doOtherThings)
# $( ... ) substitutes/returns the command's output
# ${ext%,*}    check from end       of $ext → delete shortest (%) or longest (%%) match
# ${ext#*,* }  check from beginning of $ext → delete shortest (#) or longest (##) match
# --------------------------------------

this_ext_update_status=""
IFS=$'\n'; # IFS=script (I)nternal (F)ield (S)eparator -> set to newline
i_ext=1; n_ext=` echo "$EXTENSIONS" | wc --lines`
do_substitute_with_originMaster="" # empty or any value

for ext in $EXTENSIONS; do
  # test zero string length
  if [[ ! -z ${ext// /} ]];then
    this_extension=$([ "${ext%,*}" == "$ext" ] && echo "$ext" || echo "${ext%%,*}");
    this_clone_version=$([ "${ext%,*}" == "$ext" ] && echo "$git_tag_or_branch" || echo "${ext#*, *}");
    if ! [[ -d /usr/share/mw-wmf-clone/extensions/${this_extension} ]]; then
      echo -e "#   Shared extension resource: \e[31m/usr/share/mw-wmf-clone/extensions/${this_extension}\e[0m \e[1mDOES NOT EXIST AS GIT SOURCE\e[0m"
      echo    "#     Read documentation https://www.mediawiki.org/wiki/Extension:$this_extension";
      echo    "#     check the extension's name or path or it may be an external extension (skipped)";
    else
      if [[ ! -d ${usr_share_extension_path}/${this_extension} ]]; then
        this_ext_update_status="+";
        sudo mkdir --parents "${usr_share_extension_path}/${this_extension}";
      # extension folder must be owned by www-data because of the subversion post-commit update
        sudo chown www-data:www-data "${usr_share_extension_path}/${this_extension}";
      else
        this_ext_update_status="u";
      fi;
      echo "# "
      printf "#   Shared extension resource: %03s of %03s: %1s %-30s (tag or branch: %s)\n" $i_ext $n_ext $this_ext_update_status $this_extension $this_clone_version;
      if [[ -d /usr/share/mw-wmf-clone/extensions/${this_extension} ]]; then
        cd "/usr/share/mw-wmf-clone/extensions/${this_extension}";
        AVAILABLE_remoteBRANCH_VERSIONS=`git branch --remote | sort --version-sort`;
        AVAILABLE_TAGS_VERSIONS=`git tag --list | sort --version-sort`;
        ##### FOLLOWING LINES CAUSE ERROR -> CHECKING code temporarily DISABLED
        if [[ `echo "$AVAILABLE_remoteBRANCH_VERSIONS" | grep --count "$this_clone_version"` -eq 0 ]]; then
          if  [[ `echo "$AVAILABLE_TAGS_VERSIONS" | grep --count "$this_clone_version"` -eq 0 ]]; then
            echo -e "#   \e[31mClone version of tag or branch $this_clone_version of $this_extension is not available!!\e[0m";
            echo    "#   Check available branches:";
            echo    "#     cd /usr/share/mw-wmf-clone/extensions/${this_extension} && git branch --remotes  | sort --version-sort";
            echo    "$AVAILABLE_remoteBRANCH_VERSIONS" | column -c 100;
            echo    "#   Check available tags:";
            echo    "#     cd /usr/share/mw-wmf-clone/extensions/${this_extension} &&  git tag --list  | sort --version-sort";
            echo    "$AVAILABLE_TAGS_VERSIONS" | column -c 100;
            do_substitute_with_originMaster="1"
          else
            do_substitute_with_originMaster=""
            echo "#     OK $this_clone_version is available";
          fi
        else
          do_substitute_with_originMaster=""
          echo "#     OK $this_clone_version is available";
        fi;
        # backup instead of delete old extension directory
        if [[ -d ${usr_share_extension_path}/${this_extension} ]];then
          echo "#     bakup ${this_extension} to ${this_extension}_bak"
          # sudo rm --recursive -- "${usr_share_extension_path}/${this_extension}"/*
          sudo mv --force "${usr_share_extension_path}/${this_extension}" "${usr_share_extension_path}/${this_extension}_bak"
        fi
        # master or git_tag_or_branch: test zero string length
        if [[ -z ${do_substitute_with_originMaster// /} ]];then
          # EXAMPLE: sudo git archive --prefix=AdminLinks/ HEAD | sudo -u www-data tar --extract --overwrite --directory=/usr/share/mediawiki25/extensions
          # TODO not sure if VisualEditor can be used from /usr/share/mw-wmf-clone/extensions/VisualEditor
          # or whether it does need an addition al git fetch in /usr/share/mediawiki27wmf/extensions
          echo "#     sudo git archive --prefix=\"${this_extension}/\" \"$this_clone_version\""
          sudo git archive --prefix="${this_extension}/" "$this_clone_version" | sudo -u www-data tar --extract --overwrite --directory="${usr_share_extension_path}";
          # VE-check fi
        else
          echo -e "#     $this_clone_version of ${this_extension} not available, \e[31mSUBSTITUTE with origin/master\e[0m"
          echo    "#     sudo git archive --prefix=\"${this_extension}/\" \"origin/master\""
          sudo git archive --prefix="${this_extension}/" "origin/master" | sudo -u www-data tar --extract --overwrite --directory="${usr_share_extension_path}";
        fi
        # bakup cleaning or restoration
        if [[ `ls "${usr_share_extension_path}/${this_extension}" | wc -l` -gt 0 ]];then
          echo "#     remove unnecessary ${this_extension}_bak"
          sudo rm --recursive -- "${usr_share_extension_path}/${this_extension}_bak"
        else
          echo "#     FATAL ERROR no files were extracted from git clone. Check specification and commands manually:"
          echo "#       cd \"/usr/share/mw-wmf-clone/extensions/${this_extension}\""
          echo "#       sudo git archive --prefix=\"${this_extension}/ \" \"$this_clone_version\" \ "
          echo "#       | sudo -u www-data tar --extract --overwrite --directory=\"${usr_share_extension_path}\";"
          if [[ -d ${usr_share_extension_path}/${this_extension}_bak ]];then
            echo "#     Restore backup: ${this_extension}_bak"
            sudo mv --force "${usr_share_extension_path}/${this_extension}_bak" "${usr_share_extension_path}/${this_extension}"
          fi
        fi
        [[ "ConfirmAccount" == ${this_extension} ]] && \
          echo "#   Shared extension resource: WARNING apply patch File:ConfirmAccount_body.patch to ConfirmAccount"

        [[ "OpenID" == ${this_extension} ]] && \
          echo "#   Shared extension resource: WARNING needs extra PHP classes see README or web documentation"

        [[ "SemanticMediaWiki" == ${this_extension} ]] && \
          echo "#   Shared extension resource: usually managed by composer. Apply perhaps patch File:Resources-php.SMW_2.3.1_remoteExtPath.patch"

        [[ "VisualEditor" == ${this_extension} ]] && \
          echo "#   Shared extension resource: WARNING YOU HAVE TO DO a git clone into ./extensions/VisualEditor MANUALLY !! (see web documentation)"

        [[ "Widgets" == ${this_extension} ]] && \
          echo "#   Shared extension resource: WARNING YOU HAVE TO DO a git clone into ./extensions/Widgets MANUALLY !! (see web documentation)"

      else
        echo -e "#     Warning '\e[31m/usr/share/mw-wmf-clone/extensions/${this_extension}\e[0m' not found!! Check extension name or git pull (step was skipped)";
      fi;
    fi;
  fi;
  i_ext=$((i_ext + 1))
done;
cd ..
IFS=$' \t\n'; # reset IFS=script (I)nternal (F)ield (S)eparator
##↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑


#######################################
# subversion step 2: biowikifarm extensions
cd "${usr_share_extension_path}"
echo "#   Update: subversion extensions of biowikifarm to ${usr_share_extension_path}"
sudo -u www-data -g www-data svn checkout file:///var/lib/svn/LocalSVNextensions "${usr_share_extension_path}"
#######################################

echo    "#   NOTICE: Take care of subversion’s checkout as well in /var/lib/svn/hooks/post-commit"
echo    "#   to get ${usr_share_extension_path} updated upon every subversion commit"
echo    "# "
echo    "#   Extensions managed by composer get overwritten and are active immediately on all depending wikis."
echo    "#   Manage those extensions in composer.local.json not in composer.json. After you have set up a correct composer file you can execute:"
echo -e "#     \e[1mcd /path/to/v-hostwiki && sudo php /usr/local/bin/composer.phar update --no-dev\e[0m"
echo    "#   Note that some extensions need also a maintenace update (consult extension’s docu before):"
echo -e "#    \e[1mcd /path/to/v-hostwiki && sudo -u www-data php ./maintenance/update.php --dbuser wikiadmin --quick --conf ./LocalSettings.php\e[0m"
Retrieved from "https://biowikifarm.net/metawiki/index.php?title=Mediawiki_installation/create_or_upgrade_shared-wiki-extensions.sh&oldid=6000"