<?xml version="1.0" encoding="utf-8"?><!DOCTYPE article  PUBLIC '-//OASIS//DTD DocBook XML V4.4//EN'  'http://www.docbook.org/xml/4.4/docbookx.dtd'><article><articleinfo><title>HelpOnXapian</title></articleinfo><section><title>Xapian</title><para>Using Xapian you can dramatically improve the performance of searching in moin and furthermore unlock some more features (see the search prefixes above) not possible with the legacy search engine. </para><section><title>Requirements</title><para>You must have Xapian itself and its Python bindings (xapian-core and xapian-bindings) from <ulink url="http://www.xapian.org/"/> at least in version 1.0.6 installed. In addition, Windows users need to install pywin32 from <ulink url="http://sourceforge.net/projects/pywin32/"/>. </para><para>To process attachment files, moin uses <code>filter</code> plugins - here is the list of filter plugins included: </para><informaltable><tgroup cols="3"><colspec colname="col_0" colwidth="25*"/><colspec colname="col_1"/><colspec colname="col_2"/><tbody><row rowsep="1"><entry colsep="1" rowsep="1"><para> <emphasis role="strong">File type</emphasis> </para></entry><entry colsep="1" rowsep="1"><para> <emphasis role="strong">Dependency</emphasis> </para></entry><entry colsep="1" rowsep="1"><para> <emphasis role="strong">Notes</emphasis> </para></entry></row><row rowsep="1"><entry colsep="1" rowsep="1"><para> Text files (.txt) </para></entry><entry colsep="1" rowsep="1"><para> - </para></entry><entry colsep="1" rowsep="1"><para> tries utf-8 and iso-8859-15 encodings (or forces to ASCII if those do not work) </para></entry></row><row rowsep="1"><entry colsep="1" rowsep="1"><para> JPEG images (.jpg) </para></entry><entry colsep="1" rowsep="1"><para> - </para></entry><entry colsep="1" rowsep="1"><para> EXIF data is extracted </para></entry></row><row rowsep="1"><entry colsep="1" rowsep="1"><para> Open Office files (.sx?) </para></entry><entry colsep="1" rowsep="1"><para> - </para></entry><entry colsep="1" rowsep="1"><para> e.g. from older <code>OpenOffice.org/StarOffice</code> versions </para></entry></row><row rowsep="1"><entry colsep="1" rowsep="1"><para> Open Document files (.od?) </para></entry><entry colsep="1" rowsep="1"><para> - </para></entry><entry colsep="1" rowsep="1"><para> e.g. from recent <code>OpenOffice.org/StarOffice</code> versions </para></entry></row><row rowsep="1"><entry colsep="1" rowsep="1"><para> Binary files </para></entry><entry colsep="1" rowsep="1"><para> - </para></entry><entry colsep="1" rowsep="1"><para> moin uses a <code>strings</code> like filter to process those, as well as a blacklist with stuff you don't want to search </para></entry></row><row rowsep="1"><entry colsep="1" rowsep="1"><para> MS Word files (.doc) </para></entry><entry colsep="1" rowsep="1"><para> antiword </para></entry><entry colsep="1" rowsep="1"><para> filter calls <code>antiword</code> </para></entry></row><row rowsep="1"><entry colsep="1" rowsep="1"><para> MS Excel files (.xls) </para></entry><entry colsep="1" rowsep="1"><para> catdoc </para></entry><entry colsep="1" rowsep="1"><para> filter calls <code>xls2csv</code> </para></entry></row><row rowsep="1"><entry colsep="1" rowsep="1"><para> MS Powerpoint files (.ppt) </para></entry><entry colsep="1" rowsep="1"><para> catdoc </para></entry><entry colsep="1" rowsep="1"><para> filter calls <code>catppt</code> </para></entry></row><row rowsep="1"><entry colsep="1" rowsep="1"><para> PDF files (.pdf) </para></entry><entry colsep="1" rowsep="1"><para> xpdf-utils or poppler-utils </para></entry><entry colsep="1" rowsep="1"><para> filter calls <code>pdftotext</code> </para></entry></row></tbody></tgroup></informaltable><para>After installing additional filters (or dependencies) you should (re)build your index. Xapian will find the new filters / support packages automagically. The next time your search results may contain results linking directly to your attachments.  </para></section><section><title>Configuration</title><para>In your wikiconfig, you have several options on how to configure Xapian: </para><para><informaltable><tgroup cols="3"><colspec colname="col_0"/><colspec colname="col_1"/><colspec colname="col_2"/><tbody><row rowsep="1"><entry colsep="1" rowsep="1"><emphasis role="strong">Variable name</emphasis></entry><entry colsep="1" rowsep="1"><emphasis role="strong">Default</emphasis></entry><entry colsep="1" rowsep="1"><emphasis role="strong">Description</emphasis>None</entry></row><row rowsep="1"><entry colsep="1" rowsep="1">xapian_index_dir</entry><entry colsep="1" rowsep="1"/><entry colsep="1" rowsep="1">False</entry></row><row rowsep="1"><entry colsep="1" rowsep="1">xapian_index_history</entry><entry colsep="1" rowsep="1"/><entry colsep="1" rowsep="1">False</entry></row><row rowsep="1"><entry colsep="1" rowsep="1">xapian_search</entry><entry colsep="1" rowsep="1"/><entry colsep="1" rowsep="1">False</entry></row><row rowsep="1"><entry colsep="1" rowsep="1">xapian_stemming</entry><entry colsep="1" rowsep="1"/><entry colsep="1" rowsep="1"/></row></tbody></tgroup></informaltable><!--The macro WikiConfigHelp caused an error and should be blacklisted. It returned the data 'Directory where the Xapian search index is stored (None = auto-configure wiki local storage)True to enable indexing of non-current page revisions.True to enable the fast, indexed search (based on the Xapian search library)True to enable Xapian word stemmer usage for indexing / searching.' which caused the docbook-formatter to choke. Please file a bug.--> </para><itemizedlist><listitem><para><emphasis role="strong">xapian_search</emphasis> (default: False) </para><itemizedlist><listitem override="none"><para>Setting this to True, enables Xapian search for your <ulink url="https://wiki.math.uzh.ch/public/HelpOnXapian/public/MoinMoin#">MoinMoin</ulink> wiki. </para><para><inlinemediaobject><imageobject><imagedata depth="16" fileref="https://wiki.math.uzh.ch/public/moin_static198/moniker/img/idea.png" width="16"/></imageobject><textobject><phrase>(!)</phrase></textobject></inlinemediaobject> Moin will auto disable xapian_search (and fall back to slow search) if it doesn't find a usable index. You can see whether it uses Xapian on <ulink url="https://wiki.math.uzh.ch/public/HelpOnXapian/public/SystemInfo#">SystemInfo</ulink>. </para></listitem></itemizedlist></listitem><listitem><para><emphasis role="strong">xapian_index_history</emphasis> (default: False) </para><itemizedlist><listitem override="none"><para>If this option is enabled, all revisions of all pages (except underlay, of which only one revision is available) are indexed. This allows users to search in older revisions of pages if enabled in the search dialogue on <ulink url="https://wiki.math.uzh.ch/public/HelpOnXapian/public/FindPage#">FindPage</ulink>. </para></listitem><listitem override="none"><para><inlinemediaobject><imageobject><imagedata depth="16" fileref="https://wiki.math.uzh.ch/public/moin_static198/moniker/img/alert.png" width="16"/></imageobject><textobject><phrase>/!\</phrase></textobject></inlinemediaobject> You need to rebuild your index if you change this option. Also check the size of your index after build if you're running a big wiki as this feature can eat up a lot of disk space. Creating the index might take rather long, if indexing history is enabled. </para></listitem></itemizedlist></listitem><listitem><para><emphasis role="strong">xapian_index_dir</emphasis> (default: None) </para><itemizedlist><listitem override="none"><para>This option lets you specify a non-default directory to save your index to. Initially, it gets saved to <emphasis>data_dir</emphasis>/cache/xapian/.  </para></listitem><listitem override="none"><para><inlinemediaobject><imageobject><imagedata depth="16" fileref="https://wiki.math.uzh.ch/public/moin_static198/moniker/img/alert.png" width="16"/></imageobject><textobject><phrase>/!\</phrase></textobject></inlinemediaobject> Don't forget to (re-)build an/the index/indices after enabling this! </para></listitem></itemizedlist></listitem><listitem><para><emphasis role="strong">xapian_stemming</emphasis> (default: False) </para><itemizedlist><listitem override="none"><para>If enabled, words will be indexed in their raw and stemmed forms and terms in your search query are stemmed in your language. This means that searching for &quot;testing&quot; will also yield pages containing the words &quot;tested&quot;, &quot;tester&quot; etc. </para></listitem><listitem override="none"><para><inlinemediaobject><imageobject><imagedata depth="16" fileref="https://wiki.math.uzh.ch/public/moin_static198/moniker/img/alert.png" width="16"/></imageobject><textobject><phrase>/!\</phrase></textobject></inlinemediaobject> Enabling/disabling this option needs a complete rebuild of your index! </para></listitem></itemizedlist></listitem></itemizedlist></section><section><title>(Re-)Building an index</title><para>You can use the supplied command line tool <emphasis>moin</emphasis> to initially build, completely rebuild or update an existing index. </para><para>To (re-)build your index please execute: </para><screen><![CDATA[moin --config-dir=/where/your/configdir/is --wiki-url=wiki-url/ index build --mode=rebuild]]></screen><para>For more information about the <code>moin index</code> command, see <ulink url="https://wiki.math.uzh.ch/public/HelpOnXapian/public/HelpOnMoinCommand#index-build">HelpOnMoinCommand</ulink>. </para><para><inlinemediaobject><imageobject><imagedata depth="16" fileref="https://wiki.math.uzh.ch/public/moin_static198/moniker/img/alert.png" width="16"/></imageobject><textobject><phrase>/!\</phrase></textobject></inlinemediaobject> Please note that you <emphasis role="strong">must</emphasis> rebuild your index if you change any of the xapian_index_history, xapian_index_dir or xapian_stemming configuration options! </para><para>If you have a large site, you may not wish for searching to be unavailable while your index rebuilds.  In this case, you can start with <code>moin index build --mode=buildnewindex</code>.  It is slow, but won't interfere with the running wiki.  Then, stop the wiki, run <code>moin index build --mode=usenewindex</code> to switch to the new index, and restart the wiki. </para></section><section><title>Testing</title><para>You can test if Xapian is enabled and if an index is available by checking <ulink url="https://wiki.math.uzh.ch/public/HelpOnXapian/public/SystemInfo#">SystemInfo</ulink>. To check if searches are performed using Xapian, enable show_timings in your wikiconfig, perform a search and look for _xapianSearch on the bottom of the page. </para></section><section><title>Usage</title><para>Xapian is basically used the same way as all other search engines. Due to Xapian's advanced features some new search term prefixed were introduced which are not already available in the legacy search engine (commonly referred to as moin search). See <ulink url="https://wiki.math.uzh.ch/public/HelpOnXapian/public/HelpOnSearching#">HelpOnSearching</ulink> for more information and/or use the new advanced search dialogue available on <ulink url="https://wiki.math.uzh.ch/public/HelpOnXapian/public/FindPage#">FindPage</ulink> to see what's available and possible. </para></section></section></article>