NAME¶

Catmandu::Importer::MediaWiki - Catmandu importer that imports pages from mediawiki

DESCRIPTION¶

This importer uses the query api from mediawiki to get a list of pages that match certain requirements.

It retrieves a list of pages and their content by using the generators from mediawiki:

<http://www.mediawiki.org/wiki/API:Query#Generators>

The default generator is 'allpages'.

The list could also be retrieved with the module 'list':

<http://www.mediawiki.org/wiki/API:Lists>

But this module 'list' is very limited. It retrieves a list of pages with a limited set of attributes (pageid, ns and title).

The module 'properties' on the other hand lets you add properties:

<http://www.mediawiki.org/wiki/API:Properties>

But the selecting parameters (titles, pageids and revids) are too specific to execute a query in one call. One should execute a list query, and then use the pageids to feed them to the submodule 'properties'.

To execute a query, and add properties to the pages in one call can be accomplished by use of generators.

<http://www.mediawiki.org/wiki/API:Query#Generators>

These parameters are set automatically, and cannot be overwritten:

action = "query" indexpageids = 1 generator = <generate> format = "json"

Additional parameters can be set in the constructor argument 'args'. Arguments for a generator origin from the list module with the same name, but must be prepended with 'g'.

ARGUMENTS¶

generate

type: string

explanation: type of generator to use. For a complete list, see <http://www.mediawiki.org/wiki/API:Lists>.
because Catmandu::Iterable already defines 'generator', this parameter has been renamed
to 'generate'.

default: 'allpages'.

args

type: hash

explanation: extra arguments. These arguments are merged with the defaults.

default:

    {
        prop => "revisions",
        rvprop => "ids|flags|timestamp|user|comment|size|content",
        gaplimit => 100,
        gapfilterredir => "nonredirects"
    }
    

which means:

    prop             add revisions to the list of page attributes
    rvprop           specific properties for the list of revisions
    gaplimit         limit for generator 'allpages' (every 'generator' has its own limit).
    gapfilterredir   filter out redirect pages
    

lgname

type: string

explanation: login name. Only used when both lgname and lgpassword are set.

<https://www.mediawiki.org/wiki/API:Login>

lgpassword

type: string

explanation: login password. Only used when both lgname and lgpassword are set.

SYNOPSIS¶

    use Catmandu::Sane;
    use Catmandu::Importer::MediaWiki;
    binmode STDOUT,":utf8";
    my $importer = Catmandu::Importer::MediaWiki->new(
        url => "http://en.wikipedia.org/w/api.php",
        generate => "allpages",
        args => {
            prop => "revisions",
            rvprop => "ids|flags|timestamp|user|comment|size|content",
            gaplimit => 100,
            gapprefix => "plato",
            gapfilterredir => "nonredirects"
        }
    );
    $importer->each(sub{
        my $r = shift;
        my $content = $r->{revisions}->[0]->{"*"};
        say $r->{title};
    });

AUTHORS¶

Nicolas Franck "<nicolas.franck at ugent.be>"

SEE ALSO¶

Catmandu, MediaWiki::API