Mediawiki subversion (SVN) to git migration
From Biowikifarm Metawiki
Revision as of 23:36, 10 December 2012 by Andreas Plank (Talk | contribs) (→Updating mediawiki to specific versions: reduced git tags)
This is a planning document
How-to is here: https://www.mediawiki.org/wiki/Download_from_Git . Checking extensions: Which extensions previously obtained from WMF-Subversion (i.e. excluding those we have in local SVN) are Not in the following list of git-migrated mediawiki extensions? https://gerrit.wikimedia.org/r/#/admin/projects/
Useful resources for understanding git checkout, head, detached head, --track:
- http://www.kernel.org/pub/software/scm/git/docs/git-checkout.html
- http://developer.imagej.net/using-git
Contents
Comparing 1.20 git-extensions with 1.18 and above, Nov. 2012
Local SVN are now in same folder as git checkout = OK
Manually copied i.e. now available in the 1.20git extension folder:
- SMWAskAPI - only installed on offene-naturfuehrer = OK
Missing but no longer used = OK
- AddMediaWizard (no longer found by find/grep)
- ArrayExtension replaced by extension: Arrays (all seem to be updated)
- SyntaxHighlight (no longer used, all occurrences refer to SyntaxHighlight_GeSHi)
Missing, now disabled in common/local settings = OK
- Disabling MassEditRegex - Installed on all wikis, but not used because the currently installed version (r4) is not compatible with MediaWiki 1.18. We have tested this extension on 1.19 where it works. For some wikis this extension would be very useful.
- Disabling MathStatFunctions - probably never used.
- Disabling DynamicPageList - only installed 4 wiki, was deprecated a long time; all pages checked and function removed.
- Disabling FileAttach: Used only as testing in testwikis, now removed there.
THE FOLLOWING probably may have to be disabled when migrating:
- MultipleUpload (also called MultiUpload) - only installed on OpenMedia, KeytoNature and OroWiki
- jQueryMsg - installed on all wikis, but obsolete in v.1.20!!
- http://www.mediawiki.org/wiki/Git/Conversion/Extensions_still_in_svn says "jQueryMsg - Abandoned - Available in Core"
New mediawiki git notes - LATER MOVE TO MW Maintenance!
Main source: mediawiki.org: Download from Git. See also overview of wikimedia projects in Git.
IMPORTANT CHANGE in comparison to earlier subversion setup: we directly clone all extensions into the /extension folder, instead of cloning elsewhere and then symlinking.
Background: git creates a full copy of the repository. Since we need several parallel branches, we clone once into mediawiki-git, and from there locally into further repose. Local repos use hardlinks, so they do not need much additional hard drive space (Note: git uses hardlinks inside .git, see ls -l: hardlink number is col. before owner; the checkout is NOT hardlinked and requires additional space.).
# 1. FIRST create a local git-clone for mediawiki-core (bare) and mw-extensions (submodules, cannot be bare): cd /usr/share; sudo mkdir mediawiki-git; cd /usr/share/mediawiki-git sudo git clone --bare https://gerrit.wikimedia.org/r/p/mediawiki/core.git core sudo git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions.git extensions cd /usr/share/mediawiki-git/extensions; sudo git submodule update --init --recursive
# 2. THEN Create LOCAL git-clones (with hardlinks to mediawiki-git!) cd /usr/share; sudo git clone --local /usr/share/mediawiki-git/core mediawiki20 # NOW clone individual desired extensions from LOCAL: # (When cloning extension as submodules, no hardlinks would be used.) # Note: extensions are cloned not parallel to core, but inside core/extensions folder cd /usr/share/mediawiki20/extensions; sudo git clone --local /usr/share/mediawiki-git/extensions/AccessControl sudo git clone --local /usr/share/mediawiki-git/extensions/AdminLinks sudo git clone --local /usr/share/mediawiki-git/extensions/APC sudo git clone --local /usr/share/mediawiki-git/extensions/Arrays sudo git clone --local /usr/share/mediawiki-git/extensions/ArticleFeedback sudo git clone --local /usr/share/mediawiki-git/extensions/ArticleFeedbackv5 sudo git clone --local /usr/share/mediawiki-git/extensions/AssertEdit sudo git clone --local /usr/share/mediawiki-git/extensions/CategoryTree sudo git clone --local /usr/share/mediawiki-git/extensions/CentralAuth sudo git clone --local /usr/share/mediawiki-git/extensions/CentralNotice sudo git clone --local /usr/share/mediawiki-git/extensions/CharInsert sudo git clone --local /usr/share/mediawiki-git/extensions/CheckUser sudo git clone --local /usr/share/mediawiki-git/extensions/Cite sudo git clone --local /usr/share/mediawiki-git/extensions/cldr sudo git clone --local /usr/share/mediawiki-git/extensions/ClickTracking sudo git clone --local /usr/share/mediawiki-git/extensions/CodeEditor sudo git clone --local /usr/share/mediawiki-git/extensions/Collection sudo git clone --local /usr/share/mediawiki-git/extensions/ConditionalShowSection sudo git clone --local /usr/share/mediawiki-git/extensions/Configure sudo git clone --local /usr/share/mediawiki-git/extensions/ConfirmAccount sudo git clone --local /usr/share/mediawiki-git/extensions/ConfirmEdit sudo git clone --local /usr/share/mediawiki-git/extensions/ContactPage sudo git clone --local /usr/share/mediawiki-git/extensions/Contest sudo git clone --local /usr/share/mediawiki-git/extensions/ContributionReporting sudo git clone --local /usr/share/mediawiki-git/extensions/DataTransfer sudo git clone --local /usr/share/mediawiki-git/extensions/DiscussionThreading sudo git clone --local /usr/share/mediawiki-git/extensions/DismissableSiteNotice sudo git clone --local /usr/share/mediawiki-git/extensions/DoubleWiki sudo git clone --local /usr/share/mediawiki-git/extensions/DumpHTML sudo git clone --local /usr/share/mediawiki-git/extensions/DynamicPageList sudo git clone --local /usr/share/mediawiki-git/extensions/EmailCapture sudo git clone --local /usr/share/mediawiki-git/extensions/ExpandTemplates sudo git clone --local /usr/share/mediawiki-git/extensions/ExternalData sudo git clone --local /usr/share/mediawiki-git/extensions/FlaggedRevs sudo git clone --local /usr/share/mediawiki-git/extensions/FormPreloadPostCache sudo git clone --local /usr/share/mediawiki-git/extensions/Gadgets sudo git clone --local /usr/share/mediawiki-git/extensions/GlobalBlocking sudo git clone --local /usr/share/mediawiki-git/extensions/GlobalUsage sudo git clone --local /usr/share/mediawiki-git/extensions/GlobalUserGroups sudo git clone --local /usr/share/mediawiki-git/extensions/GraphViz sudo git clone --local /usr/share/mediawiki-git/extensions/HeaderTabs sudo git clone --local /usr/share/mediawiki-git/extensions/ImageMap sudo git clone --local /usr/share/mediawiki-git/extensions/InputBox sudo git clone --local /usr/share/mediawiki-git/extensions/Interlanguage sudo git clone --local /usr/share/mediawiki-git/extensions/intersection sudo git clone --local /usr/share/mediawiki-git/extensions/Interwiki sudo git clone --local /usr/share/mediawiki-git/extensions/LabeledSectionTransclusion sudo git clone --local /usr/share/mediawiki-git/extensions/LiquidThreads sudo git clone --local /usr/share/mediawiki-git/extensions/Lockdown sudo git clone --local /usr/share/mediawiki-git/extensions/Loops sudo git clone --local /usr/share/mediawiki-git/extensions/Maintenance sudo git clone --local /usr/share/mediawiki-git/extensions/Maps sudo git clone --local /usr/share/mediawiki-git/extensions/Math sudo git clone --local /usr/share/mediawiki-git/extensions/MediaFunctions sudo git clone --local /usr/share/mediawiki-git/extensions/MobileFrontend sudo git clone --local /usr/share/mediawiki-git/extensions/MoodBar sudo git clone --local /usr/share/mediawiki-git/extensions/MWSearch sudo git clone --local /usr/share/mediawiki-git/extensions/Nuke sudo git clone --local /usr/share/mediawiki-git/extensions/OAI sudo git clone --local /usr/share/mediawiki-git/extensions/OggHandler sudo git clone --local /usr/share/mediawiki-git/extensions/OpenID sudo git clone --local /usr/share/mediawiki-git/extensions/OpenSearchXml sudo git clone --local /usr/share/mediawiki-git/extensions/PagedTiffHandler sudo git clone --local /usr/share/mediawiki-git/extensions/ParserFunctions sudo git clone --local /usr/share/mediawiki-git/extensions/PdfHandler sudo git clone --local /usr/share/mediawiki-git/extensions/Persona sudo git clone --local /usr/share/mediawiki-git/extensions/Poem sudo git clone --local /usr/share/mediawiki-git/extensions/ProofreadPage sudo git clone --local /usr/share/mediawiki-git/extensions/Quiz sudo git clone --local /usr/share/mediawiki-git/extensions/Renameuser sudo git clone --local /usr/share/mediawiki-git/extensions/ReplaceText sudo git clone --local /usr/share/mediawiki-git/extensions/RSS sudo git clone --local /usr/share/mediawiki-git/extensions/Scribunto sudo git clone --local /usr/share/mediawiki-git/extensions/SemanticCompoundQueries sudo git clone --local /usr/share/mediawiki-git/extensions/SemanticDrilldown sudo git clone --local /usr/share/mediawiki-git/extensions/SemanticForms sudo git clone --local /usr/share/mediawiki-git/extensions/SemanticFormsInputs sudo git clone --local /usr/share/mediawiki-git/extensions/SemanticImageInput sudo git clone --local /usr/share/mediawiki-git/extensions/SemanticInternalObjects sudo git clone --local /usr/share/mediawiki-git/extensions/SemanticMaps sudo git clone --local /usr/share/mediawiki-git/extensions/SemanticMediaWiki sudo git clone --local /usr/share/mediawiki-git/extensions/SemanticResultFormats sudo git clone --local /usr/share/mediawiki-git/extensions/SemanticSignup sudo git clone --local /usr/share/mediawiki-git/extensions/SemanticWatchlist sudo git clone --local /usr/share/mediawiki-git/extensions/SimpleSurvey sudo git clone --local /usr/share/mediawiki-git/extensions/SocialProfile sudo git clone --local /usr/share/mediawiki-git/extensions/SpamBlacklist sudo git clone --local /usr/share/mediawiki-git/extensions/Sudo sudo git clone --local /usr/share/mediawiki-git/extensions/Suhosin sudo git clone --local /usr/share/mediawiki-git/extensions/SwiftCloudFiles sudo git clone --local /usr/share/mediawiki-git/extensions/SyntaxHighlight_GeSHi sudo git clone --local /usr/share/mediawiki-git/extensions/TimedMediaHandler sudo git clone --local /usr/share/mediawiki-git/extensions/TitleKey sudo git clone --local /usr/share/mediawiki-git/extensions/UploadLocal sudo git clone --local /usr/share/mediawiki-git/extensions/UploadWizard sudo git clone --local /usr/share/mediawiki-git/extensions/Validator sudo git clone --local /usr/share/mediawiki-git/extensions/Variables sudo git clone --local /usr/share/mediawiki-git/extensions/Vector sudo git clone --local /usr/share/mediawiki-git/extensions/VipsScaler sudo git clone --local /usr/share/mediawiki-git/extensions/VisualEditor sudo git clone --local /usr/share/mediawiki-git/extensions/WebFonts sudo git clone --local /usr/share/mediawiki-git/extensions/Widgets sudo git clone --local /usr/share/mediawiki-git/extensions/Wikibase sudo git clone --local /usr/share/mediawiki-git/extensions/WikibaseSolr sudo git clone --local /usr/share/mediawiki-git/extensions/WikiEditor sudo git clone --local /usr/share/mediawiki-git/extensions/WikiLove sudo git clone --local /usr/share/mediawiki-git/extensions/WikimediaMessages
# 3. Also insert the local-svn-managed extensions into the same folder: # Note: www-data cannot create the folders, but must be able to write the updates (svn-post-commit hook) -> temp folder # Remaining Problem: when a new svn folder is added after this initial setup to the local subversion repo, # that folder must manually chown to www-data; else svn co post-commit errors will occur. cd /usr/share/mediawiki20/ sudo mkdir TEMP-LOCAL-svn; ## temp folder sudo svn checkout file:///var/lib/svn/LocalSVNextensions ./TEMP-LOCAL-svn sudo chown -R www-data.www-data ./TEMP-LOCAL-svn sudo mv ./TEMP-LOCAL-svn/* ./extensions; sudo mv ./TEMP-LOCAL-svn/.svn ./extensions sudo rm ./TEMP-LOCAL-svn -r
# 4. Finishing touches. Widgets needs a writable folder: cd /usr/share/mediawiki20/ sudo chown -R www-data.www-data ./extensions/Widgets/compiled_templates; # And widgets does not contain the necessary code: # (Warning: require_once(/usr/share/mediawiki20/phase3/extensions/Widgets/smarty/Smarty.class.php): failed to open stream: No such file or directory in /usr/share/mediawiki20/phase3/extensions/Widgets/Widgets.php on line 53 etc.) # In the absence of installation instructions, using the previous 1.18 code: cp -r /usr/share/mediawikistaging/ext-tagged-release/Widgets/smarty ./extensions/Widgets cp -r /usr/share/mediawikistaging/ext-tagged-release/Widgets/googlecode ./extensions/Widgets # Note: a list of UNUSED extensions is available on earlier versions of this page # (before 9 Dec. 2012); may be useful for easy copying or checking additional extensions.
# A second clone: cd /usr/share; sudo git clone --local /usr/share/mediawiki-git/core mediawikiPP cd /usr/share/mediawikiPP/extensions; ## (EXECUTE EXTENSION CLONE-COMMANDS, SVN, AND FINISHING FROM ABOVE with modified path HERE AS WELL)
Updating mediawiki (git+svn)
# 1. Update LOCAL-git core and extensions from WMF-git:
cd /usr/share/mediawiki-git/core; sudo git fetch; cd ../extensions; sudo git pull; sudo git submodule update --init --recursive;
# 2. Reset local mediawiki installations prior to the update:
# (Note: HEAD is head of current branch i.e. head after checkout REL1_20 is 1.20, not master.
# (Use "git branch" or "git branch -a" to check which branch is active.)
cd /usr/share/mediawikiPP; sudo git checkout origin/master; cd extensions; (for f in *; do cd "$f" && sudo git checkout origin/master && cd ..; done);
cd /usr/share/mediawiki20; sudo git checkout origin/master; cd extensions; (for f in *; do cd "$f" && sudo git checkout origin/master && cd ..; done);
# 3. Update local mediawiki20, mediawikiPP from LOCAL-git and LOCAL-svn with:
cd /usr/share/mediawiki20; sudo git pull; cd extensions; (for f in *; do cd "$f" && sudo git pull origin/master && cd ..; done); /usr/bin/svn update;
cd /usr/share/mediawikiPP; sudo git pull; cd extensions; (for f in *; do cd "$f" && sudo git pull origin/master && cd ..; done); /usr/bin/svn update;
## Note: LATER check whether the git pull commands above are really necessary. It always reports "already up-to-date" - perhaps due to the hardlinks?
# 4. switch to the desired version again (AFTER UPDATING), see next section...
Updating mediawiki to specific versions
SWITCH INSTALLATIONS TO SPECIFIC TAGS:
# Note: to see all branches/tags do:
git branch -r | sort --version-sort
git tag -l | sort --version-sort
# Mediawiki core:
cd /usr/share/mediawikiPP; sudo git checkout origin/REL1_19
cd /usr/share/mediawiki20; sudo git checkout origin/REL1_20
# SMW extensions:
# find tags online: https://gerrit.wikimedia.org/r/gitweb?p=mediawiki/extensions/SemanticMediaWiki.git;a=tags
# on the commend line: git tag -l | sort --version-sort
cd /usr/share/mediawiki20/extensions/SemanticMediaWiki; sudo git checkout tags/1.8
cd /usr/share/mediawiki20/extensions/SemanticCompoundQueries; sudo git checkout tags/0.3.4
cd /usr/share/mediawiki20/extensions/SemanticDrilldown; sudo git checkout tags/1.2.4
cd /usr/share/mediawiki20/extensions/SemanticForms; sudo git checkout tags/2.5.1
# cd /usr/share/mediawiki20/extensions/SemanticFormsInputs; sudo git checkout tags/DOES NOT EXIST YET
cd /usr/share/mediawiki20/extensions/SemanticMaps; sudo git checkout tags/2.0
cd /usr/share/mediawiki20/extensions/SemanticInternalObjects; sudo git checkout tags/0.7
cd /usr/share/mediawiki20/extensions/SemanticResultFormats; sudo git checkout tags/1.8
cd /usr/share/mediawiki20/extensions/Maps; sudo git checkout tags/2.0
cd /usr/share/mediawiki20/extensions/Validator; sudo git checkout tags/0.5.1
cd /var/www/testwiki_v20; sudo -u www-data php ./maintenance/update.php --quick --conf ./LocalSettings.php
cd /var/www/v-practicalplants/w; sudo -u www-data php ./maintenance/update.php --quick --conf ./LocalSettings.php
sudo /usr/sbin/apache2ctl -k graceful && sudo /etc/init.d/nginx restart
After this:
cd /var/www/testwiki_v20; sudo -u www-data php ./maintenance/update.php --quick --conf ./LocalSettings.php
started to run, but terminated with error: "Database returned error "1170: BLOB/TEXT column 'acr_email' used in key specification without a key length (localhost)""
- creating a new db with all shared tables locally and a setting in LocalSettings: $wgSharedTables = array("interwiki"); did not help.
- Changed field type, see [1]* ./maintenance/update.php running through.
Main page loads, but Special pages cause error (in biowikifarm.net-error.log): 2012/10/13 00:55:52 [error] 1546#0: *723661 FastCGI sent in stderr: "PHP message: PHP Fatal error: Class 'SMWSQLStore2' not found in /usr/share/mediawiki20/phase3/extensions/SemanticInternalObjects/SemanticInternalObjects_body.php on line 56" while reading response header from upstream, client...
Source are SemanticForms and SemanticInternalObjects.
TESTING: http://biowikifarm.net/test_v20/Main_Page versus http://biowikifarm.net/test_v18/Main_Page (test with normal staging version, just for identical web server setup)
After waiting several weeks, the bug did not go away. Submitted bug reports: https://bugzilla.wikimedia.org/show_bug.cgi?id=41640 https://bugzilla.wikimedia.org/show_bug.cgi?id=41641
