Mediawiki subversion (SVN) to git migration

From Biowikifarm Metawiki
Revision as of 11:27, 12 December 2012 by WikiSysop (Talk | contribs) (intermediate save, unfinished)

Jump to: navigation, search

This is a planning document

How-to is here: https://www.mediawiki.org/wiki/Download_from_Git . Checking extensions: Which extensions previously obtained from WMF-Subversion (i.e. excluding those we have in local SVN) are Not in the following list of git-migrated mediawiki extensions? https://gerrit.wikimedia.org/r/#/admin/projects/

Useful resources for understanding git checkout, head, detached head, --track:


Comparing 1.20 git-extensions with 1.18 and above, Nov. 2012

Local SVN are now in same folder as git checkout = OK

Manually copied i.e. now available in the 1.20git extension folder:

  • SMWAskAPI - only installed on offene-naturfuehrer = OK

Missing but no longer used = OK

  • AddMediaWizard (no longer found by find/grep)
  • ArrayExtension replaced by extension: Arrays (all seem to be updated)
  • SyntaxHighlight (no longer used, all occurrences refer to SyntaxHighlight_GeSHi)

Missing, now disabled in common/local settings = OK

  • Disabling MassEditRegex - Installed on all wikis, but not used because the currently installed version (r4) is not compatible with MediaWiki 1.18. We have tested this extension on 1.19 where it works. For some wikis this extension would be very useful.
  • Disabling MathStatFunctions - probably never used.
  • Disabling DynamicPageList - only installed 4 wiki, was deprecated a long time; all pages checked and function removed.
  • Disabling FileAttach: Used only as testing in testwikis, now removed there.


THE FOLLOWING probably may have to be disabled when migrating:


Installing mediawiki from git (1.20 and later)

Main source: mediawiki.org: Download from Git. See also overview of wikimedia projects in Git.

IMPORTANT CHANGE in comparison to earlier subversion setup: we directly clone all extensions into the /extension folder, instead of cloning elsewhere and then symlinking.

Memo: use "git tag -l | sort -V" um tags sortiert anzuzeigen

First installation only: Create normal git-clone

A biowikifarm git clone is updated Core and extensions are exported by a tar archive export out of a git clone. (HINT: before starting the script do a sudo command, e.g. sudo ls so the bash prompt knows your password and you don't have to give your password over and over again) 

# 1. FIRST create a biowikifarm git-clone (non-bare) for mediawiki-core and mw-extensions (submodules):
  cd /usr/share; sudo mkdir mw-wmf-clone; cd /usr/share/mw-wmf-clone
  sudo git clone https://gerrit.wikimedia.org/r/p/mediawiki/core.git core
  sudo git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions.git extensions
  cd /usr/share/mw-wmf-clone/extensions; sudo git submodule update --init --recursive
  # Note: with after init, one can use "git foreach". Example:
  #   git submodule foreach 'echo $path `git rev-parse HEAD`' 
  #   will show the path and currently checked out commit for each submodule.

Update git and export versions to folders

This works for creating the first version as well as updating. 

 # 1. Update LOCAL-git core and extensions from WMF-git:
 cd /usr/share/mw-wmf-clone/core; sudo git fetch; cd ../extensions; sudo git pull; sudo git submodule update --init --recursive;

 # 2. now we export (git archive) a specific MediaWiki versions into other folders:
  sudo mkdir /usr/share/mw1.20
  cd /usr/share/mw-wmf-clone/core
  sudo git archive 1.20.2 | sudo tar --extract --overwrite --directory=/usr/share/mw1.20
  # Note: tar provides a "--recursive-unlink" option for extract, which removes the content of the target folder
  #  However, our mediawiki installations require many additional files (svn, widget-right changes, smarty, commons-bot)
  #  which would have to be manually restored. Therefore, do not use --recursive-unlink

 extpathto="/usr/share/mw1.20/extensions/"
 extpathsrc="/usr/share/mw-wmf-clone/extensions/"
 # specify version control like this:
 #"SemanticMediaWiki"           → version HEAD + extension SemanticMediaWiki
 #"tags/1.8, SemanticMediaWiki" → version  1.8 + extension SemanticMediaWiki
 EXTENSIONS="AccessControl
AdminLinks
APC
Arrays
ArticleFeedback
ArticleFeedbackv5
AssertEdit
CategoryTree
CentralAuth
CentralNotice
CharInsert
CheckUser
Cite
cldr
ClickTracking
CodeEditor
Collection
ConditionalShowSection
Configure
ConfirmAccount
ConfirmEdit
ContactPage
Contest
ContributionReporting
DataTransfer
DiscussionThreading
DismissableSiteNotice
DoubleWiki
DumpHTML
DynamicPageList
EmailCapture
ExpandTemplates
ExternalData
FlaggedRevs
FormPreloadPostCache
Gadgets
GlobalBlocking
GlobalUsage
GlobalUserGroups
GraphViz
HeaderTabs
ImageMap
InputBox
Interlanguage
intersection
Interwiki
LabeledSectionTransclusion
LiquidThreads
Lockdown
Loops
Maintenance
tags/2.0, Maps
Math
MediaFunctions
MobileFrontend
MoodBar
MWSearch
Nuke
OAI
OggHandler
OpenID
OpenSearchXml
PagedTiffHandler
ParserFunctions
PdfHandler
Persona
Poem
ProofreadPage
Quiz
Renameuser
ReplaceText
RSS
Scribunto
tags/0.3.4, SemanticCompoundQueries
tags/1.2.4, SemanticDrilldown
tags/2.5.1, SemanticForms
SemanticFormsInputs
SemanticImageInput
tags/0.7, SemanticInternalObjects
tags/2.0, SemanticMaps
tags/1.8, SemanticMediaWiki
tags/1.8, SemanticResultFormats
SemanticSignup
SemanticWatchlist
SimpleSurvey
SocialProfile
SpamBlacklist
Sudo
Suhosin
SwiftCloudFiles
SyntaxHighlight_GeSHi
TimedMediaHandler
TitleKey
UploadLocal
UploadWizard
tags/0.5.1, Validator
Variables
Vector
VipsScaler
VisualEditor
WebFonts
Widgets
Wikibase
WikibaseSolr
WikiEditor
WikiLove
WikimediaMessages"

 IFS=$'\n' # set field separator to newline
 for ext in $EXTENSIONS; do
  # split line on ", ". If only one value, export HEAD, else first value is branch/tag. Example:
  #   "tags/1.8, SemanticMediaWiki"
  #      ↓             ↓
  #   $cloneVersion  $extension
  # will export tag 1.8. 
  
  # get cloneVersion and extension (as part of the line-string)
  cloneVersion=$([ "${ext%,*}" == "$ext" ] && echo "HEAD" || echo "${ext%,*}")
  extension=$([ "${ext%,*}" == "$ext" ] && echo "$ext" || echo "${ext##*,* }")
  if [ ! -d "${extpathto}${extension}" ]; then
    sudo mkdir --parents "${extpathto}${extension}"
  fi
  if [ ! -d "${extpathsrc}${extension}" ]; then
    echo "${extpathsrc}${extension} DOES NOT EXIST to export a git archive from!! (Skip this step)"
  else
    printf "Git archive export: %-30s %s\n" $extension $cloneVersion
    # sudo git archive --prefix=AdminLinks/ HEAD  | sudo tar --extract --overwrite --directory=/usr/share/mw1.20/extensions
    cd "${extpathsrc}${extension}" && sudo git archive $cloneVersion  --prefix="${extension}/" | sudo tar --extract --recursive-unlink --directory="${extpathto}"
  fi
 done






# 3. Also insert the local-svn-managed extensions into the same folder:
# Note: www-data cannot create the folders, but must be able to write the updates (svn-post-commit hook) -> temp folder
# Remaining Problem: when a new svn folder is added after this initial setup to the local subversion repo,
#   that folder must manually chown to www-data; else svn co post-commit errors will occur.
cd /usr/share/mediawiki20/
sudo mkdir TEMP-LOCAL-svn; ## temp folder
sudo svn checkout file:///var/lib/svn/LocalSVNextensions ./TEMP-LOCAL-svn
sudo chown -R  www-data.www-data ./TEMP-LOCAL-svn
sudo mv ./TEMP-LOCAL-svn/* ./extensions; sudo mv ./TEMP-LOCAL-svn/.svn ./extensions
sudo rm ./TEMP-LOCAL-svn -r

# 4. Finishing touches. Widgets needs a writable folder:
cd /usr/share/mediawiki20/
sudo chown -R  www-data.www-data ./extensions/Widgets/compiled_templates; 
# And widgets does not contain the necessary code:
# (Warning: require_once(/usr/share/mediawiki20/phase3/extensions/Widgets/smarty/Smarty.class.php): failed to open stream: No such file or directory in /usr/share/mediawiki20/phase3/extensions/Widgets/Widgets.php on line 53 etc.)
# In the absence of installation instructions, using the previous 1.18 code: 
cp -r /usr/share/mediawikistaging/ext-tagged-release/Widgets/smarty ./extensions/Widgets
cp -r /usr/share/mediawikistaging/ext-tagged-release/Widgets/googlecode ./extensions/Widgets
# Note: a list of UNUSED extensions is available on earlier versions of this page
# (before 9 Dec. 2012); may be useful for easy copying or checking additional extensions.

# A second clone:
cd /usr/share; sudo git clone --local /usr/share/mediawiki-git/core mediawikiPP
cd /usr/share/mediawikiPP/extensions;
## (EXECUTE EXTENSION CLONE-COMMANDS, SVN, AND FINISHING FROM ABOVE with modified path HERE AS WELL)


Updating mediawiki (git+svn)

 # 2. Reset local mediawiki installations prior to the update:
 # (Note: HEAD is head of current branch i.e. head after checkout REL1_20 is 1.20, not master.
 # (Use "git branch" or "git branch -a" to check which branch is active.)
 cd /usr/share/mediawikiPP; sudo git checkout origin/master; cd extensions; (for f in *; do cd "$f" && sudo git checkout origin/master && cd ..; done);
 cd /usr/share/mediawiki20; sudo git checkout origin/master; cd extensions; (for f in *; do cd "$f" && sudo git checkout origin/master && cd ..; done);
 # 3. Update local mediawiki20, mediawikiPP from LOCAL-git and LOCAL-svn with:
 cd /usr/share/mediawiki20; sudo git pull; cd extensions; (for f in *; do cd "$f" && sudo git pull origin/master && cd ..; done); /usr/bin/svn update;
 cd /usr/share/mediawikiPP; sudo git pull; cd extensions; (for f in *; do cd "$f" && sudo git pull origin/master && cd ..; done); /usr/bin/svn update;
 ## Note: LATER check whether the git pull commands above are really necessary. It always reports "already up-to-date" - perhaps due to the hardlinks?

 # 4. switch to the desired version again (AFTER UPDATING), see next section...

Updating mediawiki to specific versions

SWITCH INSTALLATIONS TO SPECIFIC TAGS: 

 # Note: to see all branches/tags do: 
 git branch -r | sort --version-sort
 git tag -l | sort --version-sort



 # Mediawiki core:
 cd /usr/share/mediawikiPP; sudo git checkout origin/REL1_19
 cd /usr/share/mediawiki20; sudo git checkout origin/REL1_20
 # SMW extensions:
 # find tags online: https://gerrit.wikimedia.org/r/gitweb?p=mediawiki/extensions/SemanticMediaWiki.git;a=tags
 # on the commend line: git tag -l | sort --version-sort
 cd /usr/share/mediawiki20/extensions/SemanticMediaWiki; sudo git checkout tags/1.8
 cd /usr/share/mediawiki20/extensions/SemanticCompoundQueries; sudo git checkout tags/0.3.4
 cd /usr/share/mediawiki20/extensions/SemanticDrilldown; sudo git checkout tags/1.2.4
 cd /usr/share/mediawiki20/extensions/SemanticForms; sudo git checkout tags/2.5.1
 # cd /usr/share/mediawiki20/extensions/SemanticFormsInputs; sudo git checkout tags/DOES NOT EXIST YET
 cd /usr/share/mediawiki20/extensions/SemanticMaps; sudo git checkout tags/2.0
 cd /usr/share/mediawiki20/extensions/SemanticInternalObjects; sudo git checkout tags/0.7
 cd /usr/share/mediawiki20/extensions/SemanticResultFormats; sudo git checkout tags/1.8
 cd /usr/share/mediawiki20/extensions/Maps; sudo git checkout tags/2.0
 cd /usr/share/mediawiki20/extensions/Validator; sudo git checkout tags/0.5.1



 cd /var/www/testwiki_v20;        sudo -u www-data php ./maintenance/update.php  --quick --conf ./LocalSettings.php 
 cd /var/www/v-practicalplants/w; sudo -u www-data php ./maintenance/update.php  --quick --conf ./LocalSettings.php
 sudo /usr/sbin/apache2ctl -k graceful  && sudo /etc/init.d/nginx restart



After this: 

cd /var/www/testwiki_v20; sudo -u www-data php ./maintenance/update.php  --quick --conf ./LocalSettings.php

started to run, but terminated with error: "Database returned error "1170: BLOB/TEXT column 'acr_email' used in key specification without a key length (localhost)""

  • creating a new db with all shared tables locally and a setting in LocalSettings: $wgSharedTables = array("interwiki"); did not help.
  • Changed field type, see [1]* ./maintenance/update.php running through.

Main page loads, but Special pages cause error (in biowikifarm.net-error.log): 2012/10/13 00:55:52 [error] 1546#0: *723661 FastCGI sent in stderr: "PHP message: PHP Fatal error: Class 'SMWSQLStore2' not found in /usr/share/mediawiki20/phase3/extensions/SemanticInternalObjects/SemanticInternalObjects_body.php on line 56" while reading response header from upstream, client...

Source are SemanticForms and SemanticInternalObjects.

TESTING: http://biowikifarm.net/test_v20/Main_Page versus http://biowikifarm.net/test_v18/Main_Page (test with normal staging version, just for identical web server setup)

After waiting several weeks, the bug did not go away. Submitted bug reports: https://bugzilla.wikimedia.org/show_bug.cgi?id=41640 https://bugzilla.wikimedia.org/show_bug.cgi?id=41641

Retrieved from "https://biowikifarm.net/metawiki/index.php?title=Mediawiki_subversion_(SVN)_to_git_migration&oldid=4187"