Difference between revisions of "Extension:CommonsMediaCaching"

From Biowikifarm Metawiki
Jump to: navigation, search
m
 
(5 intermediate revisions by the same user not shown)
Line 1: Line 1:
−
TODO documentation
+
(TODO check correct documentation)
  
 
This extension reads all new thumbs (from [https://commons.wikimedia.org/wiki/ commons.wikimedia.org/wiki/]) in the wiki <code>media</code> or <code>image</code> folder an if they are new it downloads them to a local specified Wiki or repository Wiki ([https://species-id.net/openmedia/ OpenMedia]).
 
This extension reads all new thumbs (from [https://commons.wikimedia.org/wiki/ commons.wikimedia.org/wiki/]) in the wiki <code>media</code> or <code>image</code> folder an if they are new it downloads them to a local specified Wiki or repository Wiki ([https://species-id.net/openmedia/ OpenMedia]).
  
−
== (1) Set up a <code>cronjob</code> for runJobs.php ==
+
Code see in https://biowikifarm.net/svn/LocalSVNextensions/CommonsMediaCaching/
  
−
TODO documentation
+
== (1) Set up which Wiki(s) shall run the caching ==
−
For instance in a file like <code>/etc/cron.d/wiki-run-jobs</code> you can specify how often the job should run the maintenance script <code>runJobs.php</code>
+
−
<syntaxhighlight lang="bash">
+
−
# RUN every 5 minutes  mediawiki Jobs: Species-ID, Offene Naturfuehrer, etc
+
−
# NOTE: without  &> /dev/null it will create mails to www-data!
+
−
3,8,13,18,23,28,33,38,43,48,53,58 * * * * www-data bash /usr/share/mediawiki26/extensions-rich-features/CommonsMediaCaching/runJobs.sh &> /dev/null
+
−
1,6,11,16,21,26,31,36,41,46,51,56 * * * * www-data bash /usr/share/mediawiki26/extensions-simple-features/CommonsMediaCaching/runJobs_simple-extensions.sh &> /dev/null
+
−
</syntaxhighlight>
+
  
−
The script …
+
<code>cacheCommons.sh</code> is used to download and cache the images from [https://commons.wikimedia.org/wiki/ commons.wikimedia.org/wiki/].
−
* <code>extensions-rich-features/…/runJobs.sh</code> contains all Wikis with extended extension set up
+
−
* and <code>extensions-simple-features/…/runJobs_simple-extensions.sh</code> for those with a simple extension setup.
+
  
−
== (2) Logrotate for runJobsOutput.log ==
+
== (2) Set up <code>logrotate</code> for logging outputs ==
  
−
Set up <code>logrotate</code>, eg. with a configuration in <code>/etc/logrotate.d/wiki-runJobs</code>
+
Recommendation summary:
 +
logrotate
 +
└ for ─┬─ Wikis in cacheCommons.sh (optional, download)
 +
        ├─ Wikis in runJobs.sh (optional, wiki page jobs)
 +
        ├─ Wikis in runJobs.sh_simple-extensions.sh (optional, wiki page jobs)
 +
        └─ …
 +
 
 +
If you have [[#set up cronjobs for runJobs|set up cron jobs]] then set up <code>logrotate</code> as well, eg. with a configuration in <code>/etc/logrotate.d/wiki-runJobs</code>
 
<syntaxhighlight lang="text">
 
<syntaxhighlight lang="text">
 
/usr/share/mediawiki26/extensions-rich-features/CommonsMediaCaching/runJobsOutput.log {
 
/usr/share/mediawiki26/extensions-rich-features/CommonsMediaCaching/runJobsOutput.log {
Line 42: Line 40:
 
</syntaxhighlight>
 
</syntaxhighlight>
  
−
== Set up a template in the local (repository) Wiki ==
+
If you also want to read the <code>cache/cacheCommons.log</code> (which are not created by default), similarly you should set up a <code>logrotate</code> for all the Wikis in <code>cacheCommons.sh</code>, which log their files into <code>cache/cacheCommons.log</code>, eg.:
 +
* <code>/var/www/testwiki/cache/cacheCommons.log</code>
 +
* <code>/path/to/wiki/cache/cacheCommons.log</code>
 +
 
 +
== (3) Set up a <code>cronjob</code> for runJobs.php (optional){{anchor|set up cronjobs for runJobs}} ==
 +
 
 +
For instance in a file like <code>/etc/cron.d/wiki-run-jobs</code> you can specify how often the job should run the maintenance script <code>runJobs.php</code>
 +
<syntaxhighlight lang="bash">
 +
# RUN every 10 minutes
 +
# NOTE: without  &> /dev/null it will create mails to the specified user (www-data)
 +
*/10 * * * * www-data bash /usr/share/mediawiki26/extensions-rich-features/CommonsMediaCaching/runJobs.sh &> /dev/null
 +
*/10 * * * * www-data bash /usr/share/mediawiki26/extensions-simple-features/CommonsMediaCaching/runJobs_simple-extensions.sh &> /dev/null
 +
</syntaxhighlight>
 +
 
 +
The script …
 +
* <code>extensions-rich-features/…/runJobs.sh</code> contains all Wikis with extended extension set up
 +
* and <code>extensions-simple-features/…/runJobs_simple-extensions.sh</code> for those with a simple extension setup.
 +
 
 +
== (4) Set up a template in the local (repository) Wiki ==
  
 
<code>Template:Cached Commons Copy</code> is used on File pages to store the information of the original commons file. It can look like this:
 
<code>Template:Cached Commons Copy</code> is used on File pages to store the information of the original commons file. It can look like this:
Line 64: Line 80:
  
 
[[Category:MediaWiki Extensions]]
 
[[Category:MediaWiki Extensions]]
 +
[[Category:Maintenance]]
 
{{DEFAULTSORT:CommonsMediaCaching}}
 
{{DEFAULTSORT:CommonsMediaCaching}}

Latest revision as of 13:19, 4 March 2020

(TODO check correct documentation)

This extension reads all new thumbs (from commons.wikimedia.org/wiki/) in the wiki media or image folder an if they are new it downloads them to a local specified Wiki or repository Wiki (OpenMedia).

Code see in https://biowikifarm.net/svn/LocalSVNextensions/CommonsMediaCaching/

(1) Set up which Wiki(s) shall run the caching

cacheCommons.sh is used to download and cache the images from commons.wikimedia.org/wiki/.

(2) Set up logrotate for logging outputs

Recommendation summary: 

logrotate
└ for ─┬─ Wikis in cacheCommons.sh (optional, download)
       ├─ Wikis in runJobs.sh (optional, wiki page jobs)
       ├─ Wikis in runJobs.sh_simple-extensions.sh (optional, wiki page jobs)
       └─ …

If you have set up cron jobs then set up logrotate as well, eg. with a configuration in /etc/logrotate.d/wiki-runJobs 

/usr/share/mediawiki26/extensions-rich-features/CommonsMediaCaching/runJobsOutput.log {
  missingok
  compress
  delaycompress
  daily
  size 1M
  create 644 www-data www-data
  rotate 10
}
/usr/share/mediawiki26/extensions-simple-features/CommonsMediaCaching/runJobsOutput.log {
  missingok
  compress
  delaycompress
  daily
  size 1M
  create 644 www-data www-data
  rotate 10
}

If you also want to read the cache/cacheCommons.log (which are not created by default), similarly you should set up a logrotate for all the Wikis in cacheCommons.sh, which log their files into cache/cacheCommons.log, eg.:

  • /var/www/testwiki/cache/cacheCommons.log
  • /path/to/wiki/cache/cacheCommons.log

(3) Set up a cronjob for runJobs.php (optional)

For instance in a file like /etc/cron.d/wiki-run-jobs you can specify how often the job should run the maintenance script runJobs.php 

# RUN every 10 minutes 
# NOTE: without  &> /dev/null it will create mails to the specified user (www-data)
*/10 * * * * www-data bash /usr/share/mediawiki26/extensions-rich-features/CommonsMediaCaching/runJobs.sh &> /dev/null 
*/10 * * * * www-data bash /usr/share/mediawiki26/extensions-simple-features/CommonsMediaCaching/runJobs_simple-extensions.sh &> /dev/null

The script …

  • extensions-rich-features/…/runJobs.sh contains all Wikis with extended extension set up
  • and extensions-simple-features/…/runJobs_simple-extensions.sh for those with a simple extension setup.

(4) Set up a template in the local (repository) Wiki

Template:Cached Commons Copy is used on File pages to store the information of the original commons file. It can look like this: 

<noinclude>A simplified version of [[Template:Cached Copy]] (see there), specific to Wikimedia commons. It also adds a new parameter: lastuploader  

The parameters are simplified to only:
* '''file=''' (without the File: prefix)
* '''lastuploader''' =for the name of the last user uploading this file on commons (not sufficient for licensing and attribution purposes, this user may only have modified someone elses original work.
* '''time=''' (optional).

Example: <nowiki>{{Cached_Commons_Copy|file=x.jpg|lastuploader=xxuser|time=2011-08-23 : 20:12}}</nowiki> results in:

{{Cached_Commons_Copy|file=x.jpg|lastuploader=xxuser|time=2011-08-23 : 20:12}}

</noinclude><includeonly><table cellspacing="4" cellpadding="4" style="margin:1em 0; width:99%; font-size:100%; background-color:#FFFFEE; border-style:solid; border-color:#BB4020; border-width: 2px; clear: both; position:relative; text-align:left;" width="99%"><tr><td>This media item is originally from ''http://commons.wikimedia.org/wiki/File:{{{file|'''The source file is missing!'''}}}''{{#if:{{{lastuploader|}}}|, last uploaded by [http://commons.wikimedia.org/wiki/User:{{anchorencode:{{{lastuploader}}}}} {{{lastuploader}}}]}}; it was copied to a local repository as a caching mechanism to speed up page rendering and to prevent links to break when media items are renamed on commons. For a full list of earlier authors and contributors please refer to the ''[http://commons.wikimedia.org/wiki/File:{{{file|}}}#filehistory file version history]'' and the ''[http://commons.wikimedia.org/w/index.php?title=File:{{{file|}}}&action=history metadata version history]'' in the repository from which this media item has been copied.

'''Below you find the original information from the metadata page at the time of copying{{#if:{{{time|}}}|&#32;({{{time}}})}}:'''
</td></tr></table><includeonly>{{#ifeq:{{NAMESPACE}}|{{ns:Category}}||[[Category:Cached Commons Images]]}}</includeonly>
Retrieved from "https://biowikifarm.net/metawiki/index.php?title=Extension:CommonsMediaCaching&oldid=6603"