PWSTAT(1) NetBSD General Commands Manual PWSTAT(1) NAME pwstat, pwstat3 -- generate a report from web logs SYNOPSIS pwstat [-htgrRouUve] [-y formula] [-l] [-s N] [-j N] [-J N] [-k pattern] [-K pattern] [-L] [-mM] [-b pattern] [-B pattern] [-f pattern] [-F pattern] [-d date] [-D date] [-q cufr] [logfile ...] [logfile.Z ...] [logfile.gz ...] pwstat3 [-htgrouUvWRyAjJkKL] [-y formula] [-l] [-P] [-s N] [-j N] [-J N] [-k pattern] [-K pattern] [-L] [-mM] [-b pattern] [-B pattern] [-f pattern] [-F pattern] [-d date] [-D date] [-q cufr] [logfile ...] [logfile.Z ...] DESCRIPTION These commands write to STDOUT an HTML-ized statistical extract of a Panix user's web log. pwstat requires the input files to be in the Panix Oldstyle Log Format (POLF), whereas pwstat3 accepts input in either POLF or Extended Common Log Format (XCLF). If no log file(s) are specified, the Panix command 'getlogs' (or pwstat (or pwstat3 ). Note that the information fetched by 'getlogs' and 'getclogs' is normally "reset" at the beginning of every month. The report generated by pwstat (or pwstat3 ) includes the following information: - Total traffic (in files and bytes delivered) from your site during the reporting period. - Analysis of traffic by date. - Analysis of traffic by hour of the day. - Analysis of traffic by archive name; i.e., filename. See the pwlog discussion for an explanation of the "(u)" and "(c)" sequences which precede the filenames. - Analysis of traffic by archive type; i.e., the filename extension. Exact archive type breakdown could be difficult to interpret if, for example, you are making significant use of CGI scripts to deliver your site's content, as such will be listed under "CGI". - Analysis of traffic by requesting top-level domain; e.g., gov, com, uk, etc. (Note: See the -r option.) - Analysis of traffic by requesting continent, as determined by exami- nation of the requesting domain. (Note: See the -r option.) The non- country top-level domains are determined as follows: com is treated as international and reported as "Commercial", edu although possibly international is treated as US and reported as "North America", gov is reported as "North America", mil is reported as "North America", net is treated as international and reported as "Network", and org is treated as international and reported as "Noncommercial Organiza- tion", - Analysis of traffic by requesting reversed sub-domain; e.g., requests from America On-Line users are reported as com.aol.*. (Note: See the -r option.) - List of URLs most frequently referring visitors to your site. - List of domains most frequently referring visitors to your site. Weblogs - as well as the outputs of 'getlogs' and 'getclogs' commands - use IP numbers, rather than hostnames, to identify the machines request- ing Web pages. When you use pwstat (or pwstat3 ) with the -r option, the program attempts to resolve those numbers to hostnames, and saves the resolved IP/hostname pairs in the pwstat , even without the -r flag, those hosts whose IPs already appear in the .pwhosts file will, by default, be identified by hostname rather than by IP. You can use the -R flag to turn this off and to get all hosts listed by IP number, as if the .pwhosts information didn't exist. When reporting by reversed subdomain, the default is to list resolved hostnames one level up from the machine name (e.g., gate- keeper.nytimes.com is included in "com.nytimes.*"), while all unresolved IP numbers are reported as "Unresolved". Use the -o option for more host- name detail and the -u or -U option for more detail on IP numbers. RUNNING PWSTAT If you have already run getlogs or pwlog , the procedure for obtaining webstats is to simply type: pwstat logfilename > statfilename If you do not specify an input file name, pwstat will automatically call getlogs for you. In other words, instead of typing getlogs > logfilename pwstat logfilename > statfilename you can just type pwstat > statfilename You can also create a statistical extract from more than one input log file by typing pwstat logfile1name logfile2name > statfilename OPTIONS The complete list of pwstat's options is included in its help message, which you can obtain by typing pwstat -h Notable among these options are: -b Include only requests from machines which include this pattern (a Perl regexp). Note: If the -r option is used, this test is made after the IP-to-hostname conversion is attempted, -B Omit requests from machines which include this pattern (a Perl regexp). Note: If you specify any combination of the -b, -B, -m and -M options, only one of them will be evaluated. Preference is in the order just given (i.e., -b always wins) -d Omit requests before the spcified date. The format of the date must be YYYY:MM:DD; for example, to obtain a report limited to requests on or after August 15, 1995, you would replace somedate with 1995:08:15. Note: Remember that the only requests which will be checked against the specified date are those from the log file(s) you've specified. -d Omit requests after the specified date. -f Include only requests for filenames which include this pattern (a Perl regexp). -F Omit requests for filenames which include this pattern (a Perl regexp). -g "Smash" the filenames of graphics, reducing any filename with extension bmp, gif, jpg, jpeg or png to (gfx) This is handy if you have directories full of GIFs and JPEGs that you don't want to see listed individually in your stats. -j In the list of URLs which most frequently referred visitors to your site, include only the N most frequent URLs. If this option is not specified, then the default is 25. If you do not want this section included in your pwstat report, then specify pwstat -j 0. -J In the list of domains which most frequently referred visitors to your site, include only the N most frequent domains. If this option is not specified, then the default is 25. If you do not want this section included in your pwstat report, then specify pwstat -J 0. -k In the list of URLs which most frequently referred visitors to your site, exclude URLS which match this pattern (a Perl regex). This option is most useful when you want to exclude referrals from within your own domain. For example, if your domain were www.skatecity.com, then you exclude self-referrals by specifying pwstat -k 'www.skatecity.com'. -K In the list of domains which most frequently referred visitors to your site, exclude domains which match this pattern (a Perl regex). -l Execute getlogs -o and use the result as input for pwstat. This results in pwstat output based on the previous getlogs reporting period. This option is ignored if you specify an input log file- name. -m Omit any request coming from any *.panix.com and *.access.net host. -M Omit any request coming from outside the *.panix.com and *.access.net domains. -o In the reversed sub-domain section of the report, the last por- tion of a computer name is normally lopped off; e.g., gate- keeper.nytimes.com would just be reported as com.nytimes.* as would all requests from everyone else in the nytimes.com domain. To force hostnames to be completely reported, invoke the -o option. -q Filter log entries by usage type, where "list" can be one or more of c, u, or f. If c, then we want corporate web hits included; if u, include personal web hits; and if f, include ftp transfers. Note: Most Panix users do not have both corporate and personal web traffic, but corporate users may want to use this option to generate separate reports for their web and ftp traffic. -r Turns on IP-to-hostname resolving. In the raw weblogs, the machines requesting your webpages are normally identified by IP number, and to turn that number into a computer name, a host lookup must be performed. Every time you use the -r option to pwstat, the newly found resolutions of IP numbers into hostnames are appended to a special file in your directory, and these results are used during subsequent runs of pwstat. See below for more information on why you should not use this option unless you really need to know the domains and subdomains of the computers visiting your site. -R Reports all the requesting hosts by IP number, even in cases where the IPs have been previously resolved to hostnames and the results are available in the .pwhosts file. -s Execute getlogs -sN and use the result as input for pwstat. The N is an integral value indicating the number of bytes at the begin- ning of the getlogs report to ignore/skip. This option is ignored if you specify an input log filename. -t Generate a text-only report. The default is an HTML report. -u Normally, unresolved IP numbers are listed in the domain and reversed sub-domain sections of the pwstat report as simply "Unresolved". To force all IP numbers to be individually reported in the reversed sub-domain section, invoke this option. -U The -u option will likely result in more data than you want, but perhaps you still want some sort of guess-timate of the number of different sites visiting your webpages. The -U option will force partial reporting of unresolved IP numbers, ignoring the last number in the four-number sequence. For example, the IP number 166.84.197.198 would be listed as 166.84.197.*, as would all other machines in the 166.84.197.* network that happened to visit your site. -y The pwstat output includes near the top a line that says "Approx. in the reversed sub-domain section, invoke this option. -U The -u option will likely result in more data than you want, but perhaps you still want some sort of guess-timate of the number of different sites visiting your webpages. The -U option will force partial reporting of unresolved IP numbers, ignoring the last number in the four-number sequence. For example, the IP number 166.84.197.198 would be listed as 166.84.197.*, as would all other machines in the 166.84.197.* network that happened to visit your site. -y The pwstat output includes near the top a line that says "Approx. Cost of External Transmissions $12.34". This cost is by default calculated using the formula for personal web service. However, the various levels of corporate web service have different cost formulas, but pwstat has no way of knowing which to use unless you tell it. Thus, you may specify one of the following schemes: personal, corporate, basic, standard or deluxe. (Note: Panix assesses monthly charges on your total traffic. If you have invoked any pwstat options which cause it to skip log entries, then the value calculated will not correspond with what you are actually charged.) SEE ALSO getlogs(1) getclogs(1) http://www.panix.com/web/faq/logs/ NetBSD 3.1 February 6, 2007 NetBSD 3.1