Mediawiki installation/create or upgrade shared wiki-core-resource-structure.sh

From Biowikifarm Metawiki
< Mediawiki installation
Revision as of 17:12, 4 March 2016 by Andreas Plank (Talk | contribs) (→‎Bash script: cp --preserve)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

Run this as a script ./create_or_upgrade_shared_wiki-core-resource-structure.sh not via copy and paste!!! The script is interactive prompts before running. The concept is to have two different setups, one for simple feature-wikis and one for rich feature-wikis.

  • it takes care of wmf-skins but extensions must be setup separately it just provides the structure for extensions.
  • it excludes /images, /mw-config and on existing /usr/share/mediawiki27wmf (=upgrade) excludes /cache, /extensions as well (a correct previous setup is assumed)
  • it does not set up /media (this should be done when creating a v-host-wiki)
  • note that if you are upgrading the shared wiki resource (e.g /usr/share/mediawiki27wmf) you have to bare in mind the depending vhost-wikis
  • it should be run only for the first set up or when update needed then during testing phase
  • the script lists depending symlinks pointing back to /usr/share/mediawikixxx before running

To set up a vhost-Wiki use ./create_or_upgrade_vhost-wiki.sh.

To set up git extensions use ./create_or_upgrade_shared-wiki-extensions.sh. Note that composer managed extensions overwrite existing extensions.

For the different needed features the script sets up the left part, the right part must be symlinked afterwards (use ./create_or_upgrade_vhost-wiki.sh). For rich and simple features the following structure is set up: 

/usr/share/mediawiki27wmf                   v-host-wiki
├── extensions-rich-features/            -> ├── extensions/        
├── vendor-rich-features/                -> ├── vendor/            
├── composer-rich-features.json          -> ├── composer.json      
├── composer-rich-features.local.json    -> ├── composer.local.json (manage composer extensions here!!)
└── composer-rich-features.lock          -> └── composer.lock      

/usr/share/mediawiki27wmf                   v-host-wiki
├── extensions-simple-features/          -> ├── extensions/        
├── vendor-simple-features/              -> ├── vendor/            
├── composer-simple-features.json        -> ├── composer.json      
├── composer-simple-features.local.json  -> ├── composer.local.json (manage composer extensions here!!)
└── composer-simple-features.lock        -> └── composer.lock

To get a symlinked v-host-wiki functioning make sure to set up: 

# LocalSettings.php
putenv('MW_INSTALL_PATH=/path/to/v-host-wiki');"

Otherwise the update.php will not work. Although usr_share_wiki_path/vendor is not used in v-host-wikis, the existence of usr_share_wiki_path/vendor/autoload.php is tested otherwise the wiki updater will complain about missing external dependencies. So do once: 

cd /usr/share/mediawikiXXX && sudo php /usr/local/bin/composer.phar update --no-dev

Bash script

Bash script: ./create_or_upgrade_shared_wiki-core-resource-structure.sh. You can make the script executable by 

# u o g means: user who owns it (u), other users in the file's group (g), other users (o)
chmod ug+x create_or_upgrade_shared_wiki-core-resource-structure.sh # add executable mode for the owned user

#!/bin/bash
# @description: Interactive script to create or update a shared Mediawiki
#   This setup will create a simple and rich extension setup
#   the simple setup has extensions-simple-features/, vendor-simple-features/, composer-simple-features.json etc.
#   and analogue for rich features. Those separate settings must be symlinked to the vhost-wiki as
#   extensions-simple-features/ --> extensions/ etc.
# @requires running sudo without prompt, but the very script itself should not run under sudo ./this-script
# @requires git pull of wmf-core  /usr/share/mw-wmf-clone/core
# @requires git pull of wmf-skins /usr/share/mw-wmf-clone/skins
# @requires git pull of wmf-extensions /usr/share/mw-wmf-clone/extensions

##↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓
#  # Update the git clone on biowikifarm before you run this script:
#  cd /usr/share/mw-wmf-clone/core; sudo git fetch; # first time
#  cd /usr/share/mw-wmf-clone/core && sudo git pull; # subsequent times
#  cd /usr/share/mw-wmf-clone/extensions && sudo git pull; sudo git submodule update --init --recursive; sudo /usr/share/mw-wmf-clone/extensions/quick-update
#  cd /usr/share/mw-wmf-clone/skins && sudo git pull && sudo git submodule update --init --recursive;
#  
#  # OPTIONAL: list information on tags etc:
#  cd /usr/share/mw-wmf-clone/core; 
#    git branch --remotes | sort --version-sort # list branches
#    git tag --list | sort --version-sort # list tags
#  cd /usr/share/mw-wmf-clone/extensions; 
#  cd /usr/share/mw-wmf-clone/extensions/ExtensionsName; 
#    git branch --remotes | sort --version-sort # list branches
#    git tag --list | sort --version-sort # list tags
#  cd /usr/share/mw-wmf-clone/skins/skinName; 
#    git branch --remotes | sort --version-sort # list branches
#    git tag --list | sort --version-sort # list tags
##↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑

##↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓
# Start of setting global variables
# usr_share_wiki_path="/usr/share/mediawiki27wmf"
usr_share_wiki_path="/usr/share/mediawiki26"
git_branch_or_tag="origin/REL1_26" ## OR "origin/wmf/1.27.0-wmf.7" for wmf version, OR REL1_25
# extensions
usr_share_extension_path="${usr_share_wiki_path}/extensions"
usr_share_extension_path_simple="${usr_share_wiki_path}/extensions-simple-features"
usr_share_extension_path_rich="${usr_share_wiki_path}/extensions-rich-features"
# composer
usr_share_vendor_path_simple="${usr_share_wiki_path}/vendor-simple-features"
usr_share_vendor_path_rich="${usr_share_wiki_path}/vendor-rich-features"
# skins
usr_share_skin_path="${usr_share_wiki_path}/skins"

# allowed values as follows
# list_of_wmfskins="
# skinname1
# skinname2, specific-exisiting-tag-or-branch
# skinname3
# "

list_of_wmfskins="Vector
CologneBlue
MonoBook
Modern"
# End of global variables
##↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑


# Fix
list_of_wmfskins=`echo "$list_of_wmfskins" | sed '/^$/d'` # remove blank lines

# prompt User
echo     "###############################################################"
[[ -d $usr_share_wiki_path ]] && \
echo -e  "# Upgrade shared Wiki resource to branch or tag \e[31m${git_branch_or_tag}\e[0m" || \
echo -e  "# Create shard Wiki resource to branch or tag \e[31m${git_branch_or_tag}\e[0m"

[[ -d $usr_share_wiki_path ]] && \
echo     "#   Step 1: update shared wiki resource ($usr_share_wiki_path)" || \
echo     "#   Step 1: create shared wiki resource ($usr_share_wiki_path)"
echo     "#     /usr/share/mw-wmf-clone -> $usr_share_wiki_path"

[[ -d $usr_share_wiki_path/skins ]] && \
echo     "#   Step 2: update shared wiki skins " || \
echo     "#   Step 2: create shared wiki skins "

echo     "#   Step 3: set up structrue for simple-features, rich-features"
echo     "# "
echo     "# Check before: up-to-date? (/usr/share/mw-wmf-clone)"
echo     "#   Did you run sudo git pull in /usr/share/mw-wmf-clone for ./core ./skins and ./extensions?"
echo     "# Check before: skins?      (/usr/share/mw-wmf-clone/skins)"
skins_available=`find /usr/share/mw-wmf-clone/skins/* -maxdepth 0 -not -name '.*' -type d -printf '%f ' | sort --ignore-case | fold --spaces | sed 's@^@#   skins available: @;'`
echo     "$skins_available"
echo     "#  "
echo     "#   The following skins are going to be copied from /usr/share/mw-wmf-clone/skins:"
echo     $list_of_wmfskins | fold | sed 's@^@#      @g;'
echo     "# "
if [[ -d $usr_share_wiki_path ]]; then 
echo     "# Information: the following wikis using the same setup. So take really care of what you do ;-) ..."
echo     "#   $usr_share_wiki_path"
  find -L /var/www/ -maxdepth 2 -samefile $usr_share_wiki_path/index.php -exec dirname '{}' ';' | sort | awk '{print "#   " NR " -> " $0}'
fi
echo     "# "
echo -e  "# Check before: Note this script \e[1mrequires sudo\e[0m. It will only work if you have done a sudo action before, eg. sudo ls"
echo     "# then you do not need to provide the sudo password during script runnig, wich will fail."
echo     "# "
echo -ne "# Do you want to upgrade shared wiki resources?\n[yes or no (default: no)]: "
read yno
case $yno in
  [yY]|[yY][Ee][Ss])
    echo "Continue ..."
  ;;
  [nN]|[nN][oO])
    echo "Stop";
    exit 1
  ;;
  *) 
    echo "Invalid or no input (stop)"
    exit 1
  ;;
esac
 
# CREATE MediaWiki core for specific version
if [[ ! -d $usr_share_wiki_path ]]; then 
  echo "### Shared-Wiki-resource (step 1): Create directory '${usr_share_wiki_path}'"
  sudo -u root -g mwadmin mkdir --parent "${usr_share_wiki_path}"
  # Extract MediaWiki core files from git to usr-share-folder
  ### EXAMPLE without vars:
  ### cd /usr/share/mw-wmf-clone/core; sudo git archive origin/REL1_23 | sudo -u root -g mwadmin tar --extract --overwrite --directory=/usr/share/mediawiki23
  echo "#   Extract archived core (git branch or tag: $git_branch_or_tag) into '${usr_share_wiki_path}' ..."
  echo "#   exclude /images, /mw-config, /cache ..."
  cd /usr/share/mw-wmf-clone/core && sudo git archive $git_branch_or_tag | sudo -u root -g mwadmin tar --extract \
    --anchored --recursive-unlink \
     --exclude="images*" --exclude="cache*" --exclude="mw-config*" \
    --overwrite --directory="${usr_share_wiki_path}"
  if [ -d "${usr_share_wiki_path}/mw-config" ];then 
    echo "#   Remove mw-config (we do not use this web install directory) ...";
    sudo rm --recursive "${usr_share_wiki_path}/mw-config"
  fi
  if [ -d "${usr_share_wiki_path}/extensions" ];then
    echo "#   ";
    echo "#   Set rights to www-data:www-data in '${usr_share_extension_path}' ...";
    echo "#   unused in favour of a simple and rich-feature extensions setup";
    echo "#   (may be used as a test-setup besides the simple- and rich-feature setup)";
    sudo chown www-data:www-data --recursive "${usr_share_extension_path}"
  fi
  # for composer on wikifarm: create a vendor folder in usr-share-wiki
  if ! [ -d "${usr_share_wiki_path}/vendor" ];then 
    echo "#   ";
    echo "#   Add vendor directory for composer manager '${usr_share_wiki_path}/vendor' ...";
    echo "#   unused in favour of a simple and rich-feature extensions setup";
    echo "#   (may be used as a test-setup besides the simple- and rich-feature setup)";
    sudo -u root -g mwadmin mkdir --parent "${usr_share_wiki_path}/vendor"
  fi
else
  # UPGRADE MediaWiki core
  echo "### Shared-Wiki-resource (step 1): Upgrade shared directory '${usr_share_wiki_path}' ... (overwrite partially)"
  echo "#   ASSUME correct installed shared wiki resources ..."
  echo "#   Clean and overwrite core (git branch or tag: $git_branch_or_tag) to '${usr_share_wiki_path}'"
  echo "#   but exclude export of /cache, /extensions, /images, /mw-config, /skins ..."
  cd /usr/share/mw-wmf-clone/core && sudo git archive $git_branch_or_tag | sudo -u root -g mwadmin tar --extract \
    --anchored --recursive-unlink \
    --exclude="cache*" --exclude="extensions*" --exclude="images*" --exclude="skins*" --exclude="mw-config*" \
    --overwrite --directory="${usr_share_wiki_path}"
fi

############
# skins
echo "### Shared-Wiki-skins (step 2): '${usr_share_wiki_path}/skins' ..."
if [[ ! -d ${usr_share_wiki_path}/skins ]];then
  echo "#   added missing '${usr_share_wiki_path}/skin"
  sudo -u root -g mwadmin mkdir --parent "${usr_share_wiki_path}/skins"
fi
# get default skins from separate skin git
## NEXT TIME: extract (git archive) from /usr/share/mw-wmf-clone/skins!
##DISABLED, IS IN COMPOSER## sudo git clone https://git.wikimedia.org/git/mediawiki/skins/Vector.git
# optionally get special skins from separate git(s)
cd "${usr_share_wiki_path}/skins"
if [[ -d /usr/share/mw-wmf-clone/skins ]];then
  IFS=$'\n'; # change internal Field Separator
  i_skin=1; n_skin=` echo "$list_of_wmfskins" | wc --lines`

  for skin in $list_of_wmfskins;do 
    this_skin=$([ "${skin%,*}" == "$skin" ] && echo "$skin" || echo "${skin%%,*}");
    this_clone_version=$([ "${skin%,*}" == "$skin" ] && echo "$git_branch_or_tag" || echo "${skin#*, *}");

    echo   "# "
    printf "#   Shared skin resource: %03s of %03s: %-30s (tag or branch: %s)\n" $i_skin $n_skin $this_skin $this_clone_version;
    if [[ -d /usr/share/mw-wmf-clone/skins/${this_skin} ]];then
      cd /usr/share/mw-wmf-clone/skins/${this_skin}
      AVAILABLE_remoteBRANCH_VERSIONS=`git branch --remote | sort --version-sort`;
      AVAILABLE_TAGS_VERSIONS=`git tag --list | sort --version-sort`;
      ##### FOLLOWING LINES CAUSE ERROR -> CHECKING code temporarily DISABLED
      if [ "`echo "$AVAILABLE_remoteBRANCH_VERSIONS" | grep --count "$this_clone_version"`" == "0" ]; then
        if  [ "`echo "$AVAILABLE_TAGS_VERSIONS" | grep --count "$this_clone_version"`" == "0" ]; then
          echo    "# ";
          echo -e "#   \e[31mClone version of branch or tag $this_clone_version of $this_skin is not available!!\e[0m";
          echo    "#   Check available branches ($this_skin):";
          echo    "#     cd /usr/share/mw-wmf-clone/skins/${this_skin} &&  git branch --remotes  | sort --version-sort";
          echo    "$AVAILABLE_remoteBRANCH_VERSIONS" | column -c 100;
          echo    "#   Check available tags ($this_skin):";
          echo    "#     cd /usr/share/mw-wmf-clone/skins/${this_skin} && git tag --list  | sort --version-sort";
          echo    "$AVAILABLE_TAGS_VERSIONS" | column -c 100;
          do_substitute_with_originMaster="1"
        else
          do_substitute_with_originMaster=""
          echo "#   OK: $this_clone_version is available";
        fi
      else
        do_substitute_with_originMaster=""
        echo "#   OK: $this_clone_version is available";
      fi;
      # check existing directory
      [[ ! -d  ${usr_share_wiki_path}/skins/${this_skin} ]] && sudo mkdir --parents "${usr_share_wiki_path}/skins/${this_skin}";
      # test zero string length
      if [[ -z ${do_substitute_with_originMaster// /} ]];then
        echo "#   Clean and overwrite skin $this_skin into ${usr_share_wiki_path}/skins ..."
        cd /usr/share/mw-wmf-clone/skins/${this_skin} && sudo git archive $git_branch_or_tag | \
          sudo -u root -g mwadmin tar --extract \
          --recursive-unlink \
          --overwrite --directory="${usr_share_wiki_path}/skins/${this_skin}"
      else
        echo    "#   Clean and overwrite skin $this_skin into ${usr_share_wiki_path}/skins ..."
        echo -e "#   $this_clone_version of ${this_skin} not available, \e[31mSUBSTITUTE with origin/master\e[0m"
        cd /usr/share/mw-wmf-clone/skins/${this_skin} && sudo git archive origin/master | \
          sudo -u root -g mwadmin tar --extract \
          --recursive-unlink \
          --overwrite --directory="${usr_share_wiki_path}/skins/${this_skin}"
      fi
  else
    echo "#   WARNING $this_skin folder does not exist: /usr/share/mw-wmf-clone/skins/${this_skin} ... (skipped)"
  fi
  i_skin=$((i_skin + 1))

  done
  IFS=$' \t\n'; # back to default
else
  echo "#   skin WARNING, please set up a git skin clone in /usr/share/mw-wmf-clone/skins ... (skipped skins)"
fi

############
# simple features (extensions, vendor, composer.json)
echo "### Shared-Wiki-setup (step 3): simple extension features ..."
if  [[ ! -d $usr_share_extension_path_simple ]];then
  echo "#   Add extension folder for simple features '$usr_share_extension_path_simple' ...";
  sudo -u root -g mwadmin mkdir --parent "$usr_share_extension_path_simple"
  sudo chown www-data:www-data --recursive "$usr_share_extension_path_simple"
  echo "#   NOTE it shall be linked '$usr_share_extension_path_simple' as /extensions in a v-host wiki";
fi

if  [[ ! -d $usr_share_vendor_path_simple ]];then
  echo "#   Add vendor-simple-directory for composer manager '${usr_share_vendor_path_simple}' ...";
  sudo -u root -g mwadmin mkdir --parent "${usr_share_vendor_path_simple}"
fi
# create composer-simple-features.json
[[ ! -e ${usr_share_wiki_path}/composer-simple-features.json ]] && \
  sudo cp --preserve "${usr_share_wiki_path}/composer.json" "${usr_share_wiki_path}/composer-simple-features.json" || \
  sudo cp --interactive --preserve "${usr_share_wiki_path}/composer.json" "${usr_share_wiki_path}/composer-simple-features.json"

sudo touch "${usr_share_wiki_path}/composer-simple-features.local.json" # creates empty file as may be the case
sudo touch "${usr_share_wiki_path}/composer-simple-features.lock" # creates empty file as may be the case

############
# rich features (extensions, vendor, composer.json)
echo "### Shared-Wiki-setup (step 3): rich extension features ..."
if  [[ ! -d $usr_share_extension_path_rich ]];then
  echo "#   Add extension folder for rich features '$usr_share_extension_path_rich' ...";
  sudo -u root -g mwadmin mkdir --parent "$usr_share_extension_path_rich"
  sudo chown www-data:www-data --recursive "$usr_share_extension_path_rich"
  echo "#   NOTE it shall be linked '$usr_share_extension_path_rich' as /extensions in a v-host wiki";
else
  echo "#   OK extension folder for rich feature wikis exist: '$usr_share_extension_path_rich' (skipped)";
fi
if [[ ! -d $usr_share_vendor_path_rich ]];then
  echo "#   Add vendor-rich-directory for composer manager '${usr_share_vendor_path_rich}' ...";
  sudo -u root -g mwadmin mkdir --parent "${usr_share_vendor_path_rich}"
else
  echo "#   OK vendor folder for rich feature wikis exist: '${usr_share_vendor_path_rich}' (skipped)";
fi

[[ ! -e ${usr_share_wiki_path}/composer-rich-features.json ]] && \
  sudo cp --preserve "${usr_share_wiki_path}/composer.json" "${usr_share_wiki_path}/composer-rich-features.json" || \
  sudo cp --interactive --preserve "${usr_share_wiki_path}/composer.json" "${usr_share_wiki_path}/composer-rich-features.json"

sudo touch "${usr_share_wiki_path}/composer-rich-features.local.json"
sudo touch "${usr_share_wiki_path}/composer-rich-features.lock"

echo    "# (Done)"

echo    "# "
echo    "#   NOTE: manage your composer extensions only in *.local.json not in composer.json!!"
echo    "#     Note the following setup for rich and simple features in ${usr_share_wiki_path} ..."
echo    "#     Make correct symlinks depending of the wiki setup (simple or rich features):"
echo    "#       usr_share_wiki_path/extensions-rich-features          -> v-host-wiki/extensions"
echo    "#       usr_share_wiki_path/composer-rich-features.local.json -> v-host-wiki/composer.local.json"
echo    "#       usr_share_wiki_path/composer-rich-features.lock       -> v-host-wiki/composer.lock"
echo    "#       usr_share_wiki_path/vendor-rich-features              -> v-host-wiki/vendor"
echo    "#       etc."
echo    "#     See the following folders and files prepared for simple a rich feature setup:"
  cd    "${usr_share_wiki_path}" &&  find . -maxdepth 1 -name '*features*' | sort | sed 's@.*@#      &@g'
echo    "#   NOTE make sure in LocalSettings.php to have set:"
echo    "#     putenv('MW_INSTALL_PATH=/path/to/v-host-wiki');"
echo    "#     this ensures a functioning maintenance/update.php"
echo    "#   NOTE although usr_share_wiki_path/vendor is not used in v-host-wikis, the existence of"
echo    "#     usr_share_wiki_path/vendor/autoload.php is tested otherwise the wiki updater will complain"
echo    "#     about missing external dependencies. So do once:"
echo -e "#     \e[31mcd ${usr_share_wiki_path} && sudo php /usr/local/bin/composer.phar update --no-dev\e[0m"
echo    "#   NOTE set up extensions manually (MediaWikiCommonSettings.php, LocalSettings.php, composer)"
echo    "#   NOTE set up cache, media on vhost-wiki level"
echo    "#   NOTE the default ${usr_share_wiki_path}/extensions, ./vendor, composer.local.json etc. are unused but could serve as a 3rd setup"
echo    "#     unused at the moment but could serve as a 3rd setup possibility for a test wiki case aside from simple- and rich-features"
echo    "#     that are used in production"
echo    "# "
echo    "#   Check also RELEASE NOTES, e.g. '${usr_share_wiki_path}/RELEASE-NOTES-X.XX'"
find -L "${usr_share_wiki_path}" -maxdepth 1 -type f -regex '.*RELEASE.*NOTES.*' \
  -exec echo -en "# navigate in more forwardly by <enter> or <space> and backwardly by <b>, <q> quits\n   more " '{}' "\n" ';'
##↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑
Retrieved from "https://biowikifarm.net/metawiki/index.php?title=Mediawiki_installation/create_or_upgrade_shared_wiki-core-resource-structure.sh&oldid=5797"