Mediawiki installation/create or upgrade shared-wiki-extensions.sh

From Biowikifarm Metawiki
< Mediawiki installation
Revision as of 13:52, 26 February 2016 by Andreas Plank (Talk | contribs) (→‎Bash script: samefile on $usr_share_extension_path not on samefile $usr_share_wiki_path/index.php)

Jump to: navigation, search

Run this as a script in normal mode not with sudo and not via copy and paste! 

./create_or_upgrade_shared-wiki-extensions.sh

The script does:

  • extract listed extensions in the script from /usr/share/mw-wmf-clone/extensions to a shared resource /usr/share/mediawikiXXX
  • list wikis depending on /usr/share/mediawikiXXX
  • if the requested git_tag_or_branch is not available whatsoever the script extracts origin/master then
  • the script needs running sudo without login prompt

The script does not:

  • manage composer extension. Read Composer (PHP) or mediawikiwiki:Composer and manage extensions in composer.local.json not in composer.json. Note that composer managed extensions are immediately active and some also need a maintenance update, e.g.
    cd /path/to/v-hostwiki && sudo -u www-data php ./maintenance/update.php --dbuser wikiadmin --quick --conf ./LocalSettings.php

Requirements:

Notes:

  • take care of subversion’s automatic checkout upon commit as well in /var/lib/svn/hooks/post-commit

Extension:VisualEditor

Required an extra git clone (according to VisualEditor documentation), using the normal git archive caused missing resources and files from it's submodule. Untested: possibly https://github.com/Kentzo/git-archive-all would be worth a try to get also submodules, but getting a shallow git clone did work: 

cd /usr/share/mediawiki27wmf/extensions-simple-features
# cd /usr/share/mediawiki27wmf/extensions-rich-features
if [[ -d VisualEditor ]];then sudo mv VisualEditor VisualEditor_backup`date +%Y%m%d%H%M`; fi
sudo git clone --depth 1 --branch origin/wmf/1.27.0-wmf.9 https://gerrit.wikimedia.org/r/p/mediawiki/extensions/VisualEditor.git
cd VisualEditor && sudo git submodule update --init --recursive
cd lib/ve && sudo git checkout origin/wmf/1.27.0-wmf.9

Not sure how to handle submodules or if /usr/share/mw-wmf-clone/extensions/VisualEditor contains a valid directory structure that cause no missing resource files.

Bash script

You can make the script executable by 

# u o g means: user who owns it , group users in the file's group, other users
# add executable mode for the owned user and groups
chmod ug+x create_or_upgrade_shared-wiki-extensions.sh


Bash script:
./create_or_upgrade_shared-wiki-extensions.sh 

#!/bin/bash
# Warning: run this in normal mode not as sudo and not as copy and paste in console!!
# @description: create or upgrade extensions in a shared MediaWiki resource
# @requires: running sudo without prompt
# @requires: a ready to use shared-wiki-resource e.g. in /usr/share/mediawiki26
# @requires: depending on the desired git_branch you need an 
#   up to date git clone of /usr/share/mw-wmf-clone/extensions
######################################################
 
##↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓
#  # Update the git clone on biowikifarm:
#  cd /usr/share/mw-wmf-clone/core; sudo git fetch; 
#  cd /usr/share/mw-wmf-clone/extensions && sudo git pull; sudo git submodule update --init --recursive; sudo /usr/share/mw-wmf-clone/extensions/quick-update
#  cd /usr/share/mw-wmf-clone/skins; sudo git pull; sudo git submodule update --init --recursive;
#  
#  # OPTIONAL: list information on tags etc:
#  cd /usr/share/mw-wmf-clone/core; 
#    git branch --remotes | sort --version-sort # list branches
#    git tag --list | sort --version-sort # list tags
#  cd /usr/share/mw-wmf-clone/extensions; 
#    git branch --remotes | sort --version-sort # list branches
#    git tag --list | sort --version-sort # list tags
##↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑
 
##↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓
# Settings: edit the following settings at your convenience
usr_share_wiki_path="/usr/share/mediawiki27wmf"
usr_share_wiki_path="/usr/share/mediawiki26"
git_tag_or_branch="origin/REL1_26" ## OR "origin/wmf/1.27.0-wmf.7" for wmf version, OR REL1_25
# extension folder
usr_share_extension_path="${usr_share_wiki_path}/extensions-simple-features"
# usr_share_extension_path="${usr_share_wiki_path}/extensions-rich-features"
 
# Note extensions managed by composer get overwritten and are not managed in this script
#   GraphViz
#   ImageMap
#   Maps
#   SemanticForms
#   SemanticMediaWiki
#   SemanticMaps
#   SemanticResultFormats
#   Translate mediawiki/translate
#   Translate suggest: mediawiki/babel
#   UniversalLanguageSelector - mediawiki/universal-language-selector
#   Translate suggest: mediawiki/translation-notifications
#   Validator: mediawiki/validator
 
# List of EXTENSIONS can manage as follows
# ExtensionName1 (and no specific tag_or_branch -> global git_tag_or_branch is used)
# ExtensionName2, comma and tag_or_branch (this specific given tag_or_branch is used instead of global)
EXTENSIONS="
Arrays
BetaFeatures
Campaigns
CategoryTree
CentralAuth
CharInsert
CheckUser
CirrusSearch
Cite
CleanChanges
CodeReview
Collection
CollapsibleVector
CommonsMetadata
ConfirmAccount
ConfirmEdit
ContactPage
DataTransfer
DismissableSiteNotice
DynamicSidebar
Echo
EducationProgram
Elastica
ExternalData, 1.8.2
FlaggedRevs
Flow
Gadgets
GeoData
GlobalUsage
HeaderTabs, 1.0.2
InputBox
Interwiki
LiquidThreads
LocalisationUpdate
Lockdown
Loops
MassEditRegex
Math
MediaFunctions
MwEmbedSupport, origin/master
MultimediaViewer
Nuke
OAuth
OpenID
PageTools
PagedTiffHandler
ParserFunctions
PdfExport
PdfHandler
Poem
ProofreadPage
Quiz
Renameuser
ReplaceText
SemanticCompoundQueries, 0.4
SemanticDrilldown, 2.0.1
SemanticFormsInputs
SemanticInternalObjects, 0.8.1
SpamBlacklist
SyntaxHighlight_GeSHi
Thanks
TimedMediaHandler, 0.5
TitleBlacklist
TranslationNotifications
TitleKey
UploadWizard
UserMerge
Variables
VisualEditor
Widgets
WikiEditor
Wikibase
WikimediaMessages
VisualEditor
"
# just some extensions:"
EXTENSIONS="
MultimediaViewer
"
# End of settings
##↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑
 
 
# Fix
EXTENSIONS=`echo "$EXTENSIONS" | sed '/^$/d'` # remove blank lines
 
# Check shared MediaWiki core 
if [[ ! -d ${usr_share_wiki_path} ]]; then 
  echo "### Shared resource does not exist: '${usr_share_wiki_path}' (stop)"
  echo "### Run script create_or_upgrade_shared_wiki-core-resource-structure.sh (see biowikifarm.net)"
  exit 1;
fi
 
# prompt User
echo    "###############################################################"
 
[[ -d $usr_share_extension_path ]] && \
echo -e "# Upgrade shared Wiki extensions to branch \e[31m${git_tag_or_branch}\e[0m" || \
echo -e "# Create shared Wiki extensions to branch \e[31m${git_tag_or_branch}\e[0m"
 
echo    "#   /usr/share/mw-wmf-clone/extensions → $usr_share_extension_path"
echo    "# Check before: up-to-date?"
echo    "#   Did you run sudo git pull and script quick-update in /usr/share/mw-wmf-clone/extensions?"
echo    "# Check before: information on branches, tags"
echo    "#   Run perhaps script write_wmf-shared_git-branches-tags2home_ext-tags-and-branches.sh"
echo    "#   to get all available branches and tags available in /usr/share/mw-wmf-clone/extensions"
echo    "# Extensions:"
echo -e "#   If an extension is not available as ${git_tag_or_branch} the script uses \e[1morigin/master\e[0m then."
echo -e "#   The following extensions are going to be updated (${git_tag_or_branch}):"
echo "$EXTENSIONS"  | sed 's@^ *\(.\+\)[ ]*$@\1@; s@\(.\+\)[,][ ]*\(.\+\)@\1 (\2)@g; $!{ s@$@;@;}' | fmt - | sed 's@^\(.\+\)@#     \1@;'
echo     "# "
if [[ -d $usr_share_wiki_path ]]; then 
echo     "# Information: the following wikis using the same setup. So take really care of what you do ;-) ..."
echo     "# (May be you need a maintenace update hereinafter as well)"
echo     "#   $usr_share_wiki_path"
  # avoid error messages 2>/dev/null
  find -L /var/www/ -maxdepth 3 -samefile $usr_share_extension_path -exec dirname '{}' ';' 2>/dev/null | sort | awk '{print "#   " NR " -> " $0}'
fi
 
CAN_I_RUN_SUDO=$(sudo -n uptime 2>&1|grep "load"|wc -l)
if [[ ! $CAN_I_RUN_SUDO -gt 0 ]]; then
echo     "# "
echo -e  "# The script was stopped because it \e[1mrequires sudo\e[0m running without promt. It will fail to work now. Please run any sudo"
echo     "# action before to have sudo action available without password prompt, e.g. list directory as sudo"
echo -e  "#   \e[1msudo ls\e[0m"
echo     "# "
exit 1
fi
 
[[ -d $usr_share_extension_path ]] && \
echo -ne "# Do you want to upgrade (\e[32m$usr_share_extension_path\e[0m)?\n[yes or no (default: no)]: " || \
echo -ne "# Do you want to create (\e[32m$usr_share_extension_path\e[0m)?\n[yes or no (default: no)]: "
read yno
case $yno in
  [yY]|[yY][Ee][Ss])
    echo "Continue ..."
  ;;
  [nN]|[nN][oO])
    echo "Stop";
    exit 1
  ;;
  *) echo "Invalid or no input (stop)"
    exit 1
  ;;
esac
 
#######################################
# 2. Create MediaWiki extensions in usr/share
# IMPORTANT: extensions managed by PHP script composer.phar and
# their dependencies must not be included here, presently these are:
# "mediawiki/graph-viz"
# "mediawiki/image-map"
# "mediawiki/semantic-media-wiki"
# "mediawiki/semantic-breadcrumb-links"
# "mediawiki/semantic-result-formats"
# "mediawiki/semantic-forms"
# "mediawiki/semantic-maps"
 
##↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓
echo "# Set ${usr_share_extension_path} to be owned by www-data (subversion post-commit demands these rights)"
if [ -d "${usr_share_extension_path}" ];then
  sudo chown www-data:www-data "${usr_share_extension_path}" -R
else
  echo "# Create ${usr_share_extension_path} ..."
  sudo -u root -g mwadmin mkdir --parents "${usr_share_extension_path}"
  sudo chown www-data:www-data "${usr_share_extension_path}" -R
fi
echo "# Update extensions in ${usr_share_extension_path} ..."
 
# --
# split line at ", ". If only one value, export global git_tag_or_branch, else first value is branch/tag. Example:
#   SemanticMediaWiki, 1.8
#      ↓                   ↓
#   $extension, $this_clone_version
#   will export tag 1.8
# NOTE: check the line (similar to ternary operator: if condition ? dothis : doOtherThings)
# $( ... ) substitutes/returns the command's output
# ${ext%,*}    check from end       of $ext → delete shortest (%) or longest (%%) match
# ${ext#*,* }  check from beginning of $ext → delete shortest (#) or longest (##) match
# --------------------------------------
 
this_ext_update_status=""
IFS=$'\n'; # IFS=script (I)nternal (F)ield (S)eparator -> set to newline
i_ext=1; n_ext=` echo "$EXTENSIONS" | wc --lines`
do_substitute_with_originMaster="" # empty or any value
 
for ext in $EXTENSIONS; do
  # test zero string length
  if [[ ! -z ${ext// /} ]];then
    this_extension=$([ "${ext%,*}" == "$ext" ] && echo "$ext" || echo "${ext%%,*}");
    this_clone_version=$([ "${ext%,*}" == "$ext" ] && echo "$git_tag_or_branch" || echo "${ext#*, *}");
    if ! [[ -d /usr/share/mw-wmf-clone/extensions/${this_extension} ]]; then
      echo -e "#   Shared extension resource: \e[31m/usr/share/mw-wmf-clone/extensions/${this_extension}\e[0m \e[1mDOES NOT EXIST AS GIT SOURCE\e[0m"
      echo    "#     Read documentation https://www.mediawiki.org/wiki/Extension:$this_extension";
      echo    "#     check the extension's name or path or it may be an external extension (skipped)";
    else
      if [[ ! -d ${usr_share_extension_path}/${this_extension} ]]; then
        this_ext_update_status="+";
        sudo mkdir --parents "${usr_share_extension_path}/${this_extension}";
      # extension folder must be owned by www-data because of the subversion post-commit update
        sudo chown www-data:www-data "${usr_share_extension_path}/${this_extension}";
      else
        this_ext_update_status="u";
      fi;
      echo "# "
      printf "#   Shared extension resource: %03s of %03s: %1s %-30s (tag or branch: %s)\n" $i_ext $n_ext $this_ext_update_status $this_extension $this_clone_version;
      if [[ -d /usr/share/mw-wmf-clone/extensions/${this_extension} ]]; then
        cd "/usr/share/mw-wmf-clone/extensions/${this_extension}";
        AVAILABLE_remoteBRANCH_VERSIONS=`git branch --remote | sort --version-sort`;
        AVAILABLE_TAGS_VERSIONS=`git tag --list | sort --version-sort`;
        ##### FOLLOWING LINES CAUSE ERROR -> CHECKING code temporarily DISABLED
        if [[ `echo "$AVAILABLE_remoteBRANCH_VERSIONS" | grep --count "$this_clone_version"` -eq 0 ]]; then
          if  [[ `echo "$AVAILABLE_TAGS_VERSIONS" | grep --count "$this_clone_version"` -eq 0 ]]; then
            echo -e "#   \e[31mClone version of tag or branch $this_clone_version of $this_extension is not available!!\e[0m";
            echo    "#   Check available branches:";
            echo    "#     cd /usr/share/mw-wmf-clone/extensions/${this_extension} && git branch --remotes  | sort --version-sort";
            echo    "$AVAILABLE_remoteBRANCH_VERSIONS" | column -c 100;
            echo    "#   Check available tags:";
            echo    "#     cd /usr/share/mw-wmf-clone/extensions/${this_extension} &&  git tag --list  | sort --version-sort";
            echo    "$AVAILABLE_TAGS_VERSIONS" | column -c 100;
            do_substitute_with_originMaster="1"
          else
            do_substitute_with_originMaster=""
            echo "#     OK $this_clone_version is available";
          fi
        else
          do_substitute_with_originMaster=""
          echo "#     OK $this_clone_version is available";
        fi;
        # test zero string length
        if [[ -z ${do_substitute_with_originMaster// /} ]];then
          # EXAMPLE: sudo git archive --prefix=AdminLinks/ HEAD | sudo -u www-data tar --extract --overwrite --directory=/usr/share/mediawiki25/extensions
          # TODO not sure if VisualEditor can be used from /usr/share/mw-wmf-clone/extensions/VisualEditor 
          # or whether it does need an addition al git fetch in /usr/share/mediawiki27wmf/extensions
          [[ "VisualEditor" == ${this_extension} ]] && \
            echo "#   Shared extension resource: WARNING YOU HAVE TO DO a git clone into ./extensions/VisualEditor MANUALLY !! (see web documentation)"
          
          if [[ -d ${usr_share_extension_path}/${this_extension} ]];then
            echo "#     clean ${usr_share_extension_path}/${this_extension}"
            sudo rm --recursive -- "${usr_share_extension_path}/${this_extension}"/*
          fi
          echo "#     sudo git archive --prefix=\"${this_extension}/\" \"$this_clone_version\"" 
          sudo git archive --prefix="${this_extension}/" "$this_clone_version" | sudo -u www-data tar --extract --overwrite --directory="${usr_share_extension_path}";
          # VE-check fi
        else
          echo -e "#     $this_clone_version of ${this_extension} not available, \e[31mSUBSTITUTE with origin/master\e[0m"
          sudo git archive --prefix="${this_extension}/" "origin/master" | sudo -u www-data tar --extract --overwrite --directory="${usr_share_extension_path}";
        fi
      else
        echo -e "#     Warning '\e[31m/usr/share/mw-wmf-clone/extensions/${this_extension}\e[0m' not found!! Check extension name or git pull (skipped this step)";
      fi;
    fi;
  fi;
  i_ext=$((i_ext + 1))
done;
cd ..
IFS=$' \t\n'; # reset IFS=script (I)nternal (F)ield (S)eparator
##↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑
 
 
#######################################
# subversion step 2: biowikifarm extensions 
cd "${usr_share_extension_path}"
echo "#   Update: subversion extensions of biowikifarm to ${usr_share_extension_path}"
sudo -u www-data -g www-data svn checkout file:///var/lib/svn/LocalSVNextensions "${usr_share_extension_path}"
#######################################
 
echo    "#   NOTICE: Take care of subversion’s checkout as well in /var/lib/svn/hooks/post-commit"
echo    "#   to get ${usr_share_extension_path} updated upon every subversion commit"
echo    "# "
echo    "#   Extensions managed by composer get overwritten and are active immediately on all depending wikis."
echo    "#   Manage those extensions in composer.local.json not in composer.json. After you have set up a correct composer file you can execute:"
echo -e "#     \e[1mcd /path/to/v-hostwiki && sudo php /usr/local/bin/composer.phar update --no-dev\e[0m"
echo    "#   Note that some extensions need also a maintenace update (consult extension’s docu before):"
echo -e "#    \e[1mcd /path/to/v-hostwiki && sudo -u www-data php ./maintenance/update.php --dbuser wikiadmin --quick --conf ./LocalSettings.php\e[0m"
Retrieved from "https://biowikifarm.net/metawiki/index.php?title=Mediawiki_installation/create_or_upgrade_shared-wiki-extensions.sh&oldid=5784"