.TH "libwget-robots" 3 "Tue Jan 26 2021" "Version 1.99.1" "wget2" \" -*- nroff -*-
.ad l
.nh
.SH NAME
libwget-robots \- Robots Exclusion file parser
.SH SYNOPSIS
.br
.PP
.SS "Functions"

.in +1c
.ti -1c
.RI "\fBwget_robots_t\fP * \fBwget_robots_parse\fP (const char *data, const char *client)"
.br
.ti -1c
.RI "void \fBwget_robots_free\fP (\fBwget_robots_t\fP **robots)"
.br
.in -1c
.SH "Detailed Description"
.PP 
The purpose of this set of functions is to parse a Robots Exlusion Standard file into a data structure for easy access\&. 
.SH "Function Documentation"
.PP 
.SS "\fBwget_robots_t\fP* wget_robots_parse (const char * data, const char * client)"

.PP
\fBParameters\fP
.RS 4
\fIdata\fP Memory with robots\&.txt content (with trailing 0-byte) 
.br
\fIclient\fP Name of the client / user-agent 
.RE
.PP
\fBReturns\fP
.RS 4
Return an allocated ROBOTS structure or NULL on error
.RE
.PP
The function parses the robots\&.txt \fCdata\fP and returns a ROBOTS structure including a list of the disallowed paths and including a list of the sitemap files\&.
.PP
The ROBOTS structure has to be freed by calling \fBwget_robots_free()\fP\&. 
.SS "void wget_robots_free (\fBwget_robots_t\fP ** robots)"

.PP
\fBParameters\fP
.RS 4
\fIrobots\fP Pointer to Pointer to ROBOTS structure
.RE
.PP
\fBwget_robots_free()\fP free's the formerly allocated ROBOTS structure\&. 
.SH "Author"
.PP 
Generated automatically by Doxygen for wget2 from the source code\&.