Readme for analog 6.0

Introduction

Although analog is free software, its distribution and modification are covered by the terms of the GNU General Public License. You are not required to accept this licence, but nothing else gives you permission to modify or distribute the program. Analog comes with no warranty.

Although analog is free, if you like it, please consider making a donation towards its development. Thank you.

This Readme describes analog 6.0. For the latest version of analog, see the analog home page. For examples of the output see

• sample statistics from the analog site
• a more graphical version (using Report Magic with analog)

This is a version of the Readme in one page. If you're reading it on line, you might prefer the version on several smaller pages. Beginners should start with the licence followed by the section on Starting to use analog. There is an index at the end of this document.

You might also find the How-To's helpful; these are descriptions by other authors of how to use analog for particular tasks.

Now you can go to

• Starting to use analog
• Customising analog
• What the results mean
• Errors and warnings
• Frequently asked questions (FAQ)
• Mailing lists
• Helper applications
• Acknowledgements
• What's new in this version?
• Quick reference (for experts)
• Index

Starting to use analog

If you log in to your ISP's machine from your home machine, you have two options. If you have the right permissions, you can run analog on your ISP's machine. Otherwise, you can download (e.g., ftp) the logfiles from their machine to yours, and then run analog on your machine.

Once you've downloaded the right version of analog for your computer from the analog home page (or a mirror site), you need to know how to set it up and run it. This is very easy, but the instructions are slightly different depending which platform you're using.

• Macintosh
• Windows 95 & later
• Everyone else (Unix, OpenVMS, OS/2, BeOS, Acorn, Windows 3.1 etc.)

If you can't manage to set up analog after reading the instructions, send a message to the analog-help mailing list.

Starting to use analog on a Mac

1. Edit analog.cfg
2. Run analog
3. Read Report.html

LOGFILE logfilename    # to set where your logfile lives

There are already some configuration commands to get you started in the configuration file, but there are lots of others available. You can find the most common ones in the section on basic commands later in the Readme, and you can read about all of them in the section on customising analog. There are also some sample configuration files in the examples folder.

There is another way to give options, via command line arguments. You'll see these mentioned in this Readme from time to time, but MacOS before MacOS X doesn't have a command line, so ignore these unless you've downloaded the Darwin version of analog.

If you want to compile your own version of analog (it's written in C), or just to read the source code, it's available from the analog home page. (It's the same source code for all versions).

Starting to use analog under Windows

Here is the really short summary:

1. Edit analog.cfg
2. Run analog (a DOS window flashes up).
3. Read Report.html

There's also a How-To written by Simon Handfield, which explains how to get started in more detail with lots of pictures.

(Some unzip programs are broken, and do not create folders when they should. If you don't have a folder called lang inside the analog folder, create one and put all the files called *.lng and *.tab into it.)

There are two ways of running analog. You can either run it from Windows (by single-clicking or double-clicking on its icon, depending on your setup), or you can run it from the DOS command prompt (under Start-Programs). If you run it from Windows, it will create a DOS window to run in. When it's finished, it will produce an output file called Report.html and some graphics; and a file called errors.txt which contains any errors there might have been. The first time you run it, this will all happen almost instantly. This is not a bug. For help in interpreting the output, see What the results mean.

LOGFILE logfilename    # to set where your logfile lives

There are already some configuration commands to get you started in the configuration file, but there are lots of others available. You can find the most common ones in the section on basic commands later in the Readme, and you can read about all of them in the section on customising analog. There are also some sample configuration files in the examples folder.

If you run analog from the DOS command prompt, there is another way to give options, via command line arguments, given on the command line after the program name. These are just shortcuts for configuration file commands. You can use the command line arguments if you run analog from a batch file too.

If you want to compile your own version of analog (it's written in C), or just to read the source code, it's available from the analog home page. (It's the same source code for all versions).

Starting to use analog on other platforms

1. Edit anlghead.h and compile, if necessary
2. Edit analog.cfg
3. Run analog

If you're not using one of the platforms for which a precompiled version is available, you'll have to compile your own version from the source. But don't worry -- it's written in standard C throughout, so it will compile out of the box on most platforms. (The source code is the same for all platforms.)

First, change to the src/ directory.

Then look at the file anlghead.h, and see if there's anything you want to edit.

When you have done that, you need to compile the program. How to do that depends on which operating system you're using.

make

(Experts can pass some arguments in on the make command line instead of by editing anlghead.h: e.g.

make DEFS='-DLANGDIR=\"/usr/etc/apache/analog/lang/\"'

If you haven't got gcc, you will need to change the compiler - try acc or cc instead.

Compiling under OpenVMS. You can find OpenVMS build scripts within the src/build directory. Unzip them within the src directory. Then to build Analog interactively from the command line, type

$ @ Build_Analog

$ Submit /NoPrint /Keep Batch.com

Compiling under Acorn RiscOS. The Makefile can be found in the src/build directly, although at this point it has not been updated for version 5 of analog. You will have to make directories called C, H and O, and move the sources files into the appropriate directories: e.g., alias.c must be renamed C.alias. And you will find that there are some filenames in the header file anlghead.h that you want to change to fit into the RiscOS directory structure.

Compiling under OS/2. To compile analog for OS/2, you will need the EMX package. You should edit the Makefile to have OS=OS2 and LIBS=-lsocket. Then after editing anlghead.h and running Make, you need to run the command

EMXBIND -b ANALOG

analog

You can configure analog by putting commands in the configuration file, which is called analog.cfg by default. Two commands you will need straight away are

LOGFILE logfilename      # to set where your logfile lives
OUTFILE outputfile.html  # to send the output to a file instead of the screen

There are already some configuration commands to get you started in the configuration file, but there are lots of others available. You can find the most common ones in the section on basic commands later in the Readme, and you can read about all of them in the section on customising analog. There are also some sample configuration files in the examples directory.

There is one other way to give options to analog, via command line arguments, given on the command line after the program name. These are just shortcuts for configuration file commands.

Customising analog

• basic commands

The following section is a technical (i.e., dull but important) one on the

• syntax of configuration commands.

Later there's an index of all the commands and topics, and also a quick reference containing the syntax of all the commands and examples.

• Choosing a logfile
• Specifying a log format
• Aliases
• Inclusions and exclusions
• Search arguments
• Configuring the output
• Time reports
• Other reports
• Hierarchical reports
• The domains file
• Computer-readable output
• Cache files
• DNS lookups
• Coping with low memory
• Debugging
• Form interface and CGI program

Basic commands

LOGFILE my_logfile
OUTFILE output.html

LOGFILE new1.log,old*.log
LOGFILE new2.log

HOSTNAME "Spam Widgets Inc."
HOSTURL http://www.spam-widgets.com/

If you have broken images in the output instead of graphs, you need to say in which directory on your server the images are stored. You do this by a command like

IMAGEDIR /analog/images/

MONTHLY ON       # one line for each month
WEEKLY ON        # one line for each week
DAILYREP ON      # one line for each day
DAILYSUM ON      # one line for each day of the week
HOURLYREP ON     # one line for each hour of the day
GENERAL ON       # the General Summary at the top
REQUEST ON       # which files were requested
FAILURE ON       # which files were not found
DIRECTORY ON     # Directory Report
HOST ON          # which computers requested files
ORGANISATION ON  # which organisations they were from
DOMAIN ON        # which countries they were in
REFERRER ON      # where people followed links from
FAILREF ON       # where people followed broken links from
SEARCHQUERY ON   # the phrases and words they used...
SEARCHWORD ON    # ...to find you from search engines
BROWSERSUM ON    # which browser types people were using
OSREP ON         # and which operating systems
FILETYPE ON      # types of file requested
SIZE ON          # sizes of files requested
STATUS ON        # number of each type of success and failure

You can configure lots of other things about each report, such as how many rows are listed, which columns are included, and how the reports are sorted. For example, the command

REQINCLUDE pages

REQFLOOR 10r

LANGUAGE FRENCH

The following languages were available for previous versions of analog, but have not yet been translated for version 5: BOSNIAN, CROATIAN, GREEK, ICELANDIC, LITHUANIAN and ROMANIAN. As and when they are translated, they will be added to the analog home page. See the section on Configuring the output for how to download, or even translate, new languages.

As I said, these are only a few of the commands available. To find out about all the commands, you'll have to read the remaining sections of the Readme, starting with a short section on the syntax of configuration commands.

Syntax of configuration commands

CONFIGFILE other.cfg

You can also include another configuration file from the command line by using a command like +gother.cfg. (Note that there is no space between +g and the filename; this is true of all command line arguments.) But note that reading an alternative configuration file does not stop the default configuration file (usually analog.cfg) being read as well. To do that you have to specify -G as well as the +g command. This is because if you want several different configurations, it's most convenient to put all the common options in analog.cfg, and options specific to each configuration in a separate file. Then the +g command line option will read both those files.

If the name of a configuration file given in a CONFIGFILE command doesn't include a directory, it will be looked for wherever analog expects to find its configuration files. (This location is a compile-time option.) For example, in the Windows version it would be in the same folder as the analog executable. This applies to the default and mandatory configuration files as well. But configuration files given with +g are relative to the current directory at the time you run the program.

In the Mac version, you can start up a program with a particular configuration file instead of the default one by dragging the configuration file onto the analog icon. The file must start with "# ".

You can also specify any configuration command on the command line even if it doesn't have a command line abbreviation, by use of the +C command. (NB The C must be upper case.) For example, +C"UNCOMPRESS *.gz gzcat" will include that command.

DAILYSUM   OFF   # We don't want a Daily Summary
DAILYREP  "ON"   # We want a full Daily Report instead 
HOSTNAME (Spam Widgets Inc.)  # Spaces, so quotes or brackets needed
LOGFILE logfile1.log,\
logfile2.log     # This line and the previous one are one command

analog -settings [other options]

analog -settings > file

Choosing a logfile

LOGFILE logfilename

A - sign or the word stdin is interpreted as standard input: this is useful on Unix systems for constructing pipes. There is also an optional second argument to the LOGFILE command which is explained below.

You can have several LOGFILE commands. You can include wildcards in the logfile name (but not necessarily in the directory name: this is system-dependent), and you can use a list of logfiles separated by commas (without spaces). So the following commands would tell analog to read logfile1, c:\logs\logfile2, and all files ending in .log:

LOGFILE logfile1,*.log
LOGFILE c:\logs\logfile2

LOGFILE "Hard Drive:Internet Applications:Analog:Logs:*"

LOGFILE none

If the name of a logfile in a LOGFILE command doesn't include a directory, it will be looked for wherever analog expects to find logfiles. (This location is built in when the program is compiled.) For example, on Windows it would be in the same folder as the analog executable. But logfile names given on the command line are within the current directory.

You can also include the date in the LOGFILE name, by using the following codes.

%D  date of month
%m  month name, in English
%M  month number
%y  two-digit year
%Y  four-digit year
%H  hour
%n  minute
%w  day of week, in English

LOGFILE access_log%Y%M.log

TO -00-0131                   # to end of last month
LOGFILE access_log%Y%M??.log  # finds access_log200108??.log in Sep 2001

The LOGFILE commands are cumulative, except that any logfiles on the command line or in configuration files specified on the command line override any in the default configuration file or configuration files loaded from there, and are themselves overridden by any in the mandatory configuration file or configuration files loaded from there. Usually you don't need to worry about this, and it will do what you expect! (Actually I should have said "logfiles or cache files" -- but we'll get on to that later).

If your logfile is not in one of the standard formats, you will probably still be OK, because it is possible to tell analog about other formats using a LOGFORMAT command. This is explained in the next section. But most users don't ever need to know about this because they have logfiles in a standard format. So the best thing to do is just to try analysing your logfile and see if analog will understand it. If it does, you don't need to worry about LOGFORMATs.

If analog can't understand your logfile, it will warn you that it can't detect the format, or possibly that it found a lot of corrupt lines. There are basically five reasons why this might happen:

1. Many people try and use a LOGFORMAT command when they don't need one. Always try without one first.
2. Some log formats are not very well designed and analog can't analyse them reliably. In this case it will give up, usually with a helpful message, rather than risk doing a bad job. For example, you might get "Logfile with ambiguous dates" or "Time without date." In this case you should read the notes on all the built-in formats below where some common problems with those formats are described.
3. Since analog tries to deduce the format based on the first line of the logfile, it could just be that the first line is corrupt. In this case, you could tell analog the format, or you could just fix the first line.
4. For the same reason, if the format changes midway through the log, analog will count the remaining lines as corrupt. In this case, you will find that your output page contains a partial analysis but with a large number of corrupt lines too. You will need to give analog two LOGFORMAT commands to tell it about the two different formats.
5. Finally, some logfiles really aren't in one of the standard formats. In this case you will need to read the next section and learn how to tell analog about your format.

LOGFILE mydomain.log http://www.mydomain.com

Note that because this actually changes the name of the file, any FILEINCLUDE, FILEEXCLUDE or FILEALIAS command will have to refer to the new name, including the prefix.

If you are using this command to combine logfiles from several different virtual hosts, then the Virtual Host Report doesn't tell you about the different virtual hosts. The virtual host name has just become part of the filename. So you want to look in the Directory Report instead. (And you will probably want to use the SUBDIR command as well.)

If the logfile contains the name of the virtual host on each line, then the argument can contain a %v, and the name of the virtual host will be inserted at that point. If %v is included and the logfile line doesn't have a virtual host, then that line will be marked as corrupt.

You need to supply the types of file that you want to uncompress in a comma-separated list, together with the name of a command that will uncompress the files to standard output (rather than to a file). For example, on Unix you might use

UNCOMPRESS *.Z "/usr/bin/uncompress -c"

UNCOMPRESS *.Z ("c:\Program Files\uncompress\uncompress" -c)

If analog determines that a logfile which it's uncompressing isn't wanted for the analysis, a "broken pipe" error can be reported. This is produced by the uncompressing command and is out of analog's control, but it's harmless.

(Hint: There's nothing to stop you using the UNCOMPRESS command for other types of preprocessing, for example DNS resolution.)

Logfile formats

The common logfile format is written by most servers. Its lines look like

jay.bird.com - fred [25/Dec/1998:17:45:35 +0000]
"GET /~sret1/ HTTP/1.0" 200 1243

jay.bird.com - fred [25/Dec/1998:17:45:35 +0000]
"GET /~sret1/ "HTTP/1.0" 200 1243

[25/Dec/1998:17:45:35] http://www.site.com/ -> /~sret1/

[25/Dec/1998:17:45:35] Mozilla/2.0 (X11; I; HP-UX A.09.05)

jay.bird.com - fred [25/Dec/1998:17:45:35 +0000] "GET /~sret1/ HTTP/1.0"
200 1243 "http://www.site.com/" "Mozilla/2.0 (X11; I; HP-UX A.09.05)"

LogFormat "%h %l %u %t \"%r\" %s %b \"%{Referer}i\" \"%{User-Agent}i\""

192.64.25.41, -, 25/12/98, 17:45:35, W3SVC1, HOST1, 192.16.225.10,
2178, 303, 1243, 200, 0, GET, /~sret1/, -,

There are also various third-party extensions to the Microsoft format to include, for example, the browser and referrer. But they all do it in different ways, so analog can't automatically diagnose them, and again, you need to write a LOGFORMAT command for them.

12/25/98 17:45:35  jay.bird.com  host1  Server  fred  GET  /~sret1/
http://www.site.com/    Mozilla/2.0 (X11; I; HP-UX A.09.05)  200  1243  2178

12/25/98  17:45:35   OK    jay.bird.com  /~sret1/  1243

If analog finds that the header line is corrupt, it will usually tell you what was wrong with it. The most common problem is that you're not allowed the time without the date or vice versa -- in particular, having the date just at the top of the logfile is not sufficient; you must have it on each line. By default, Microsoft servers produce extended logs with the date only at the top. But if the date changes during the logfile, the server doesn't then write a new date line. This means that missing days or corrupt entries can make analog get a day out in either direction, with no way to rescue or even recognise the situation!

For this reason analog knows that it can't analyse such logfiles safely, so instead it insists that the date should be on every line. There are some programs on the helper applications page to put the date on each line. If you already have such a logfile you might want to use one of these programs, but they have to assume that the date doesn't change during the logfile, so it would be much safer to tell your server to log the date on every line in future.

The extended log is described at http://www.w3.org/TR/WD-logfile.html. Its header line looks like

#Fields: date time cs-uri

There is also Microsoft's attempt at the extended format -- unfortunately they didn't read the spec., so they didn't enclose the browser and referrer in quotes, they replaced spaces in the browser name with +'s, and they put the time taken to serve the request in milliseconds instead of seconds. And there is WebSTAR's attempt which is very nearly right except that they erroneously used the CS-HOST field as the client hostname instead of the server hostname. Analog will understand all of these versions.

Extended logs always record the time in GMT, so you will probably need to use a LOGTIMEOFFSET command to convert to your local timezone.

The WebSTAR format is described at http://www.starnine.com/webstar/docs/ws4manual.3f.html. It has a header line like

!!LOG_FORMAT DATE TIME RESULT URL BYTES_SENT HOSTNAME

Finally, the Netscape header line looks like

format=%Ses->client.ip% [%SYSDATE%] "%Req->reqpb.clf-request%"
%Req->srvhdrs.clf-status% %Req->srvhdrs.content-length%

Log formats

The basic command to specify a log format looks like

LOGFORMAT format

APACHELOGFORMAT apacheformat

The LOGFORMAT and APACHELOGFORMAT commands only apply to logfiles specified with a LOGFILE command later in the same configuration file. So you must put the LOGFORMAT above the LOGFILE to which it refers. If you declare your logfiles on the command line, or drag them onto the app on the Mac, you must use DEFAULTLOGFORMAT or APACHEDEFAULTLOGFORMAT instead. This is so that different logfiles can have different formats, like this:

LOGFILE log0
LOGFORMAT format1
LOGFILE log1
LOGFORMAT format2
LOGFILE log2
LOGFILE log3

LogFormat "%h %l %u %t %v \"%r\" %>s %b" myformat
CustomLog /var/log/apache/access.log myformat

APACHELOGFORMAT (%h %l %u %t %v \"%r\" %>s %b)
LOGFILE /var/log/apache/access.log

There are format words for all the built-in formats analog knows about. You might need one of these words if your logfile is in a standard format, but analog can't detect which format it's in for some reason; for example, maybe the first line is corrupt; or maybe analog can't tell whether you're using North American or international dates. So for example

LOGFORMAT COMMON

If your logfile is not in one of the recognised formats, you can tell analog about your format using a log format string. You only ever need this if your logfile has lines which are not in one of the standard formats. (And even if it isn't in a standard format, if you're using the Apache web server, you will find APACHELOGFORMAT easier.)

The format string consists of a template for the logfile line, with the various fields and special characters replaced by codes as follows. Please note that these codes are case sensitive -- for example, %b is completely different from %B!

%S

host (the client hostname, or address of the computer making the request)

%s

numerical IP address of client (if recorded in a separate field; used when %S is empty)

%r

file requested

%q

query string (part of filename after ?, if recorded in a separate field)

%B

browser

%A

browser with +'s instead of spaces

%f

referrer

%u

user (tip: a cookie or session id can usefully be defined as %u too)

%v

virtual host (the server hostname, also called the virtual domain)

%d

day of the month

%m

month in digits

%M

month, three letter English abbreviation

%y

year, last two digits

%Y

year, four digits

%Z

year, two or four digits (less efficient)

%h

hour of the day

%n

minute of the hour

%a

a or A for am, or p or P for pm, if %h is in the 12-hour clock. (So to match "am" you need %am and to match "AM" you need %aM)

%U

"Unix time" (seconds since beginning of 1970, GMT). If it includes decimals, use %U.%j

%b

number of bytes transferred

%t

processing time in seconds

%T

processing time in milliseconds

%D

processing time in microseconds

%c

HTTP status code

%C

code words used instead of HTTP status code in some servers -- only used internally

%j

junk: ignore this field (field can be empty too)

%w

white space: spaces or tabs

%W

optional white space

%%

% sign

\n

new line

\t

tab stop

\\

single backslash

jay.bird.com - fred [25/Dec/1998:17:45:35 +0000]
"GET /~sret1/ HTTP/1.0" 200 1243

LOGFORMAT (%S - %u [%d/%M/%Y:%h:%n:%j %j] "%j %r %j" %c %b)

Fri 25/12/98 5:45pm, /~sret1/, jay.bird.com, 200, 1243,
http://www.site.com, Mozilla/2.0 (X11; I; HP-UX A.09.05)

LOGFORMAT (%j %d/%m/%y %h:%n%am, %r, %S, %c, %b, %f, %B)

LOGFORMAT COMMON
LOGFORMAT COMBINED
LOGFILE log1
LOGFORMAT (%j %d/%m/%y %h:%n%am, %r, %S, %c, %b, %f, %B)
LOGFILE log2
LOGFILE log3

If you specify several formats, analog tries to match each line to the first format first, then if that fails the next, and so on, so the order of the formats is important. Usually you want to specify the most common one first, to minimise the time spent trying to match lines to inappropriate formats.

So let's go back to the first example:

LOGFILE log0
LOGFORMAT format1
LOGFILE log1
LOGFORMAT format2
LOGFILE log2
LOGFILE log3

The times you need to use the DEFAULTLOGFORMAT instead of the LOGFORMAT are if you want to change the format of logfiles which aren't given in a LOGFILE command -- for example, ones specified on the command line, or dragged onto the program icon on a Mac, or compiled in.

The "Unix time", %U, is always recorded in GMT. So you will probably need to use a LOGTIMEOFFSET command to convert to your local timezone. Also, it's just the integer part of the time, so if you have decimals you will have to use %U.%j .

The log formats which analog can handle are those which are known as instantaneously decipherable: in practice, this means that the character which terminates a string can never occur in the string. So for example, in common format, which looks like

LOGFORMAT (%S - %u [%d/%M/%Y:%h:%n:%j %j] "%j %r %j" %c %b)

Sometimes you need to read one of the fields in a logfile, but not analyse it. For example, if you have a separate common log and referrer log, the referrer log might look like

http://guide-p.infoseek.com/Titles -> /~sret1/analog/

LOGFORMAT (%f -> %*r)

A tip: sometimes it is more efficient to specify two or more adjacent fields to ignore with a single %j, as long as the whole group ends with a recognisable character. So common format is more efficiently specified as

LOGFORMAT (%S - %u [%d/%M/%Y:%h:%n:%j] "%j %r %j" %c %b)

Another tip: %j can also be used to ignore whole lines, rather than just fields analog doesn't use. For example, the extended log format ignores lines beginning with # by using

LOGFORMAT #%j

LOGFORMAT (%*S, %*u, %m/%d/%y, %h:%n:%j, %j)

Common format, LOGFORMAT COMMON

jay.bird.com - fred [25/Dec/1998:17:45:35 +0000]
      "GET /~sret1/ HTTP/1.0" 200 1243
LOGFORMAT (%S %j %u [%d/%M/%Y:%h:%n:%j] "%j%w%r%wHTTP%j" %c %b)
LOGFORMAT (%S %j %u [%d/%M/%Y:%h:%n:%j] "%j%w%r" %c %b)
LOGFORMAT (%S %j %u [%d/%M/%Y:%h:%n:%j] "%r" %c %b)

Microsoft common format, LOGFORMAT MS-COMMON

jay.bird.com - fred [25/Dec/1998:17:45:35 +0000]
      "GET /~sret1/ "HTTP/1.0" 200 1243
LOGFORMAT (%S %j %u [%d/%M/%Y:%h:%n:%j] "%j%w%r%w"HTTP%j" %c %b)
LOGFORMAT (%S %j %u [%d/%M/%Y:%h:%n:%j] "%j%w%r" %c %b)
LOGFORMAT (%S %j %u [%d/%M/%Y:%h:%n:%j] "%r" %c %b)

Combined log, LOGFORMAT COMBINED

jay.bird.com - fred [25/Dec/1998:17:45:35 +0000] "GET /~sret1/ HTTP/1.0" 200
      1243 "http://www.site.com/" "Mozilla/2.0 (X11; I; HP-UX A.09.05)"
LOGFORMAT (%S %j %u [%d/%M/%Y:%h:%n:%j] "%j%w%r%wHTTP%j" %c %b "%f" "%B")
LOGFORMAT (%S %j %u [%d/%M/%Y:%h:%n:%j] "%j%w%r" %c %b "%f" "%B")
LOGFORMAT (%S %j %u [%d/%M/%Y:%h:%n:%j] "%r" %c %b "%f" "%B")

Referrer log, LOGFORMAT REFERRER

[25/Dec/1998:17:45:35] http://www.site.com/ -> /~sret1/
or http://www.site.com/ -> /~sret1/
LOGFORMAT ([%d/%M/%Y:%h:%n:%j] %f -> %*r)
LOGFORMAT (%f -> %*r)

Browser log, LOGFORMAT BROWSER

[25/Dec/1998:17:45:35] Mozilla/2.0 (X11; I; HP-UX A.09.05)
LOGFORMAT ([%d/%M/%Y:%h:%n:%j] %B)

Microsoft log, North American dates, LOGFORMAT MICROSOFT-NA

192.64.25.41, -, 12/25/98, 17:45:35, W3SVC1, HOST1, 192.16.225.10,
      2178, 303, 1243, 200, 0, GET, /~sret1/, -,
192.64.25.41, -, 12/25/2001, 17:45:35, W3SVC1, HOST1, 192.16.225.10,
      2178, 303, 1243, 200, 0, GET, /~sret1/, -,
LOGFORMAT (%S, %u, %m/%d/%Z, %h:%n:%j, W3SVC%j, %j, %v,
      %T, %j, %b, %c, %j, %j, %r, %q,)
LOGFORMAT (%*S, %*u, %m/%d/%Z, %h:%n:%j, %j)

Microsoft log, international dates, LOGFORMAT MICROSOFT-INT

192.64.25.41, -, 25/12/98, 17:45:35, W3SVC1, HOST1, 192.16.225.10,
      2178, 303, 1243, 200, 0, GET, /~sret1/, -,
192.64.25.41, -, 25/12/2001, 17:45:35, W3SVC1, HOST1, 192.16.225.10,
      2178, 303, 1243, 200, 0, GET, /~sret1/, -,
LOGFORMAT (%S, %u, %d/%m/%Z, %h:%n:%j, W3SVC%j, %j, %v,
      %T, %j, %b, %c, %j, %j, %r, %q,)
LOGFORMAT (%*S, %*u, %d/%m/%Z, %h:%n:%j, %j)

WebSite log, North American dates, LOGFORMAT WEBSITE-NA

12/25/98 17:45:35  jay.bird.com  host1  Server  fred  GET  /~sret1/
   http://www.site.com/    Mozilla/2.0 (X11; I; HP-UX A.09.05)  200  1243  2178
LOGFORMAT (%m/%d/%y %h:%n:%j\t%S\t%v\t%j\t%u\t%j\t%r\t%f\t%j\t%B\t%c\t%b\t%T)

WebSite log, international dates, LOGFORMAT WEBSITE-INT

25/12/98 17:45:35  jay.bird.com  host1  Server  fred  GET  /~sret1/
   http://www.site.com/    Mozilla/2.0 (X11; I; HP-UX A.09.05)  200  1243  2178
LOGFORMAT (%d/%m/%y %h:%n:%j\t%S\t%v\t%j\t%u\t%j\t%r\t%f\t%j\t%B\t%c\t%b\t%T)

MacHTTP format, LOGFORMAT MACHTTP

12/25/98  17:45:35   OK    jay.bird.com  /~sret1/  1243
LOGFORMAT (%m/%d/%y\t%h:%n:%j \t%C%w%S\t%r\t%b)

Aliases

CASE INSENSITIVE
CASE SENSITIVE

USERCASE SENSITIVE

DIRSUFFIX default.htm

FILEALIAS /football.html /soccer.html
HOSTALIAS lion lion.statslab.cam.ac.uk

The alias commands for the other items are called BROWALIAS, REFALIAS, USERALIAS and VHOSTALIAS. Only one alias is ever applied to any item. So after

FILEALIAS /football.html /soccer.html
FILEALIAS /soccer.html /brazil.html

You can also use wildcards in ALIAS commands: ? matches any one character and * matches any number of characters (including none). And on the right-hand side, you can use $1, $2 etc. to represent the parts of the original name matched by the *'s. As a special abbreviation, if there is exactly one * on the left-hand side, then a * on the right-hand side can be used to represent $1. So, for example,

FILEALIAS /*/football/* /soccer/

FILEALIAS /*/football/* /$1/soccer/$2         # or
FILEALIAS /sport/football/* /sport/soccer/*

You can use $$ to get an actual $ on the right-hand side. Or you can prefix the right-hand side with "PLAIN:" to treat any $'s and *'s on the right-hand side literally. For example

FILEALIAS /*/football/* PLAIN:/$1/soccer/$2

Analog's *'s are un-greedy: if there are two possible ways of matching, the part of the expression on the left matches as little as possible. This is more often what you want. But it contrasts with Perl's regular expressions, for example. (Oh, two consecutive *'s are completely useless, but if you try it they are collapsed into one before counting the $1, $2, etc.)

The behaviour of FILEALIAS and REFALIAS can be slightly unintuitive if the file has search arguments.

A warning to Unix users: if you put an ALIAS command on the command line with +C, the shell may try and expand $1 etc., which is not what you want. To stop the shell doing this, put the command in single quotes instead of double quotes.

TYPEALIAS .txt ".txt (Plain text files)"

There can be some confusion between some normal alias and output alias commands. For example, what is the difference between FILEALIAS and REQALIAS? In fact, there are several differences because of the different things the aliases are doing. FILEALIAS applies to the files themselves, but REQALIAS only applies to the lines in the Request Report. This means that FILEALIAS also affects the other reports which use the filenames, such as the Directory Report, whereas REQALIAS only affects the Request Report.

Another difference is that REQALIAS applies separately to each line of the Request Report. This means that if two separate files translate to the same thing in a FILEALIAS command, they will become one file for all the reports. But if you were to use the same REQALIAS command, they would still be two files, and would still be listed on separate lines in the Request Report, but with the same name.

So in summary, when should you use each command? FILEALIAS should be used if a single file has two different names; i.e., if your web server returns the same file for two different URLs. REQALIAS, on the other hand, would typically be used to annotate or clarify the Request Report. Sometimes it's useful to use both; first combine some files with FILEALIAS, and then annotate them in the Request Report with REQALIAS.

The full list of output aliases is REQALIAS, REDIRALIAS, FAILALIAS, TYPEALIAS, DIRALIAS, HOSTREPALIAS, REDIRHOSTALIAS, FAILHOSTALIAS, DOMALIAS, ORGALIAS, REFREPALIAS, REFSITEALIAS, REDIRREFALIAS, FAILREFALIAS, BROWREPALIAS, BROWSUMALIAS, OSALIAS, VHOSTREPALIAS, REDIRVHOSTREPALIAS, FAILVHOSTREPALIAS, USERREPALIAS, REDIRUSERALIAS and FAILUSERALIAS.

There is one known bug with the output aliases. The report is sorted before the alias is applied. This means that if the SORTBY for the report is set to ALPHABETICAL, then the report will not be sorted correctly.

You include regular expressions in an ALIAS command by prefixing the left-hand side of the alias with "REGEXP:". Or you can specify a case-insensitive match, like Perl m//i or Unix egrep -i, by using "REGEXPI:". (It's automatically case-insensitive for many items, such as hostnames, or filenames if you have specified CASE INSENSITIVE.)

On the right-hand side of the alias you can use $1, $2 etc. to represent the first, second etc. bracketed expression on the left-hand side, counting in order of the left brackets. (Again, you can't put $1, $2 etc. on the command line unless you put them in single quotes.)

Regular expressions match if they match just part of the string. If you want them to have to match the whole of the string, you have to anchor them to the ends of the string with ^ and $.

For example,

REQALIAS REGEXP:^(/~(.+?)/.*) "[$2] $1"

/~sret1/backgammon/rules.html

[sret1] /~sret1/backgammon/rules.html

HOSTALIAS REGEXP:^([^.]*)$ $1.mycompany.com

Regular expressions are greedy: if there are two possible ways of matching, the part of the expression on the left matches as much as possible.

Inclusions and exclusions

HOSTEXCLUDE mycomputer.myisp.com

The rule for determining whether an item is included or excluded is as follows. All the INCLUDE and EXCLUDE commands for that item are considered one by one in order, and the item is included or excluded according to the last command it matched. Items which don't match any of the INCLUDE or EXCLUDE commands are included if the first command was an exclusion, and excluded if the first command was an inclusion. For example, the configuration

FILEINCLUDE /~sret1/*
FILEEXCLUDE /~sret1/backgammon/*,/~sret1/analog/*
FILEINCLUDE /~sret1/backgammon/*.gif

FILEEXCLUDE /~sret1/*/img/*

The full list of these commands is HOSTINCLUDE and HOSTEXCLUDE; FILEINCLUDE and FILEEXCLUDE; BROWINCLUDE and BROWEXCLUDE; REFINCLUDE and REFEXCLUDE; USERINCLUDE and USEREXCLUDE; VHOSTINCLUDE and VHOSTEXCLUDE; and STATUSINCLUDE and STATUSEXCLUDE.

Because the inclusions and exclusions take place after the aliasing, the name you must use is the aliased name. (In the absence of output alias commands, this is the name of the item in the output.)

Sometimes a line doesn't contain a particular sort of item, either because there is no field reserved for it on the line, or because the browser didn't send it for that request, or because it was present but corrupt. You can include or exclude these lines by making a special blank entry in the INCLUDE or EXCLUDE command. For example,

USERINCLUDE jim
USERINCLUDE ""

The behaviour of REQINCLUDE and REFINCLUDE can be slightly unintuitive if the file has search arguments.

You can also use regular expressions for the inclusions and exclusions by prefixing the expression with "REGEXP:" or "REGEXPI:". I've already described this at length in the context of aliases, so you can look there for all the details. A regular expression must be on a line on its own, not within a comma-separated list.

HOSTINCLUDE 131.111.20.18      # simple IP address
HOSTINCLUDE 131.111.20.*       # wildcard
HOSTINCLUDE 131.111.20         # the same meaning
HOSTINCLUDE 131.111.20-23      # a range of class C addresses
HOSTINCLUDE 131.111.20.18/23   # subnet mask

STATUSINCLUDE 200-206,304,500-

Some people want to exclude status code 304 (Not Modified) to stop those requests appearing in the Request Report. But there is a better solution. By default, analog counts code 304 as a successful request, because it assumes that the cached version of the document is then presented to the user. But you can count it as a redirected request with the command

304ISSUCCESS OFF

FROM 990701
TO   000615:1300

FROM -01-00+01   # from tomorrow last year
TO -00-0131  # to the end of last month (OK even if last month
             # didn't have 31 days)
FROM -00-00-112
TO   -00-00-01  # statistics for the last 16 weeks
FROM -00-00-00:-06+01  # statistics for the last 6 hours

So, for example, the command

REFREPEXCLUDE http://your.site.com/*

The full list of these commands is REQINCLUDE and REQEXCLUDE; REDIRINCLUDE and REDIREXCLUDE; FAILINCLUDE and FAILEXCLUDE; TYPEINCLUDE and TYPEEXCLUDE; DIRINCLUDE and DIREXCLUDE; HOSTREPINCLUDE and HOSTREPEXCLUDE; REDIRHOSTINCLUDE and REDIRHOSTEXCLUDE; FAILHOSTINCLUDE and FAILHOSTEXCLUDE; DOMINCLUDE and DOMEXCLUDE; ORGINCLUDE and ORGEXCLUDE; REFREPINCLUDE and REFREPEXCLUDE; REFSITEINCLUDE and REFSITEEXCLUDE; SEARCHQUERYINCLUDE and SEARCHQUERYEXCLUDE; SEARCHWORDINCLUDE and SEARCHWORDEXCLUDE; INTSEARCHQUERYINCLUDE and INTSEARCHQUERYEXCLUDE; INTSEARCHWORDINCLUDE and INTSEARCHWORDEXCLUDE; REDIRREFINCLUDE and REDIRREFEXCLUDE; FAILREFINCLUDE and FAILREFEXCLUDE; BROWSUMINCLUDE and BROWSUMEXCLUDE; BROWREPINCLUDE and BROWREPEXCLUDE; OSINCLUDE and OSEXCLUDE; VHOSTREPINCLUDE and VHOSTREPEXCLUDE; REDIRVHOSTREPINCLUDE and REDIRVHOSTREPEXCLUDE; FAILVHOSTREPINCLUDE and FAILVHOSTREPEXCLUDE; USERREPINCLUDE and USERREPEXCLUDE; REDIRUSERREPINCLUDE and REDIRUSERREPEXCLUDE; and FAILUSERINCLUDE and FAILUSEREXCLUDE.

The inclusion or exclusion applies to the unaliased name, if you are doing any output aliases. (This contrasts with the behaviour of normal INCLUDE and EXCLUDE commands, which apply to the aliased name.)

All directory names end in slashes, so DIRINCLUDE and DIREXCLUDE, and REFSITEINCLUDE and REFSITEEXCLUDE, implicitly add a trailing slash even if you don't give one. This sometimes catches people out in the following situation.

REFSITEEXCLUDE http://my.host.com/*     # probably not what you want

REFSITEEXCLUDE http://my.host.com/

You can also use the symbolic word pages in suitable INCLUDE and EXCLUDE commands; one very common command is

REQINCLUDE pages

PAGEINCLUDE *.asp
PAGEEXCLUDE /sret1.html

REQLINKINCLUDE pages,*.pdf

ROBOTINCLUDE Googlebot/*

Search arguments

/cgi-bin/script.pl?x=1&y=2

You can tell analog either to read or to ignore the arguments using the commands ARGSINCLUDE and ARGSEXCLUDE which we'll discuss in a minute. But by default, all arguments are read, and as this is usually what you want, you don't usually need those commands.

You don't always see the arguments in the reports, even if they're being read, because analog doesn't show them if there aren't enough of them. In order to see them, you have to set the corresponding ARGSFLOOR parameter low enough.

Also note that within a report, the search arguments are listed immediately under the file to which they refer. This temporarily interrupts the normal order of the files. It may be clearer if you turn the N column on.

The reason is that, for example, the command

FILEINCLUDE /cgi-bin/script.pl

FILEINCLUDE /cgi-bin/script.pl*

FILEALIAS /cgi-bin/script.pl /script.pl

FILEALIAS /cgi-bin/script.pl?* /script.pl?$1

Conversely, because in the Request Report files with arguments are only included if their parent file is included, you can't just

REQINCLUDE /cgi-bin/script.pl?*x=1*

REQINCLUDE /cgi-bin/script.pl

ARGSEXCLUDE /cgi-bin/script.pl

ARGSINCLUDE /cgi-bin/script.pl

Technical note: the check for whether the arguments should be included happens before the filename has been subject to either built-in or user-specified aliases. So you have to use the unaliased name, exactly as it occurs in the logfile. For example, ARGSINCLUDE /~sret1/script.pl won't match /%7Esret1/script.pl even though they are really the same file. It also means that you can't use "pages" in the ARGSINCLUDE or ARGSEXCLUDE command, because we don't know whether a file is a page until after it's been aliased.

http://www.altavista.com/cgi-bin/query?pg=q&kl=XX&q=carrot+cake

SEARCHENGINE http://www.altavista.com/cgi-bin/query q

SEARCHENGINE http://*altavista.*/* q

The command INTSEARCHENGINE is the same for search engines, or other scripts which take arguments, within your site. For example, you might have requests for files like

/cgi-bin/search?trm=chocolate+cake

INTSEARCHENGINE /cgi-bin/search trm

Sometimes a search engine has two or more possible fields for the search term. In that case you can list all of them separated by commas, like this:

SEARCHENGINE http://*webcrawler.*/* search,searchText

I said previously that %7E in a URL is automatically converted to ~, etc. In fact this is only done to the ASCII-printable characters %20-%7E, because these are the only characters that are the same in every character set. (In fact, even that isn't true. Experts might want to know that ?, &, ; and = aren't converted either, to distinguish them from query-string delimiters: an encoded ?, &, ; or = is one that is not intended to be a delimiter. Also % isn't converted, to avoid confusing %25nm with %nm.)

But in the Search Query Report and Search Word Report it is useful to be able to convert non-ASCII characters too, so that you can see the actual words people typed, rather than get the %nm codes in place of all accented letters. So in these reports analog also converts characters %A0-%FF (if you are using an ISO-8859-* character set) or %80-%FF (for most other character sets).

However, there are reasons why you might not want this feature, and you can turn it off with the command

SEARCHCHARCONVERT OFF

1. The character set in which the query was submitted to the search engine may not be the same as that in which the page reached was written, or that in which the analog output page is being written. So converting to the character set of the analog output page may give garbage anyway. This is particularly a problem with languages, such as Russian, which have two or more characters sets in common use. It is also a problem for sites which host resources in many languages.
2. Not all of the character positions correspond to printable characters in every character set. Analog knows that %80-%9F are non-printable in the ISO-8859-* character sets, but apart from that it converts everything in %80-%FF. So you may end up with non-printable characters in your output.

Configuring the output

XHTML is the default. It produces web pages in XHTML 1.0. HTML produces web pages in HTML 2.0.

PLAIN produces plain text files, and ASCII is the same as PLAIN except that it uses all ASCII characters (no accents etc.) if possible. (This is because some applications don't understand accented characters).

LATEX produces LaTeX code which can be turned into PDF if you have the pdflatex command installed. (If you want to use the ordinary latex command, specify PDFLATEX OFF.) It's only available with certain European languages (US-ASCII, ISO-8859-1 and ISO-8859-2 character sets). Yes, I know it gives overfull hboxes sometimes.

COMPUTER is a special format suitable for reading by a computer (useful for reading into a spreadsheet, or post-processing with a graphics package, for example). There is a separate section about this format later.

XML produces an XML output which is an alternative format for post-processing. The DTD for the XML output is distributed with the program. You can find more information about the XML style, and an example of a post-processing program, at http://timian.jessen.ch/.

As well as a command like

OUTPUT PLAIN

You can also specify OUTPUT NONE for no output, if you are producing a cache file.

LANGUAGE FRENCH

The following languages were available for previous versions of analog, but have not yet been translated for version 5: BOSNIAN, CROATIAN, GREEK, ICELANDIC, LITHUANIAN and ROMANIAN. As and when they are translated, they will be added to the analog home page. If you want to translate any of them (or any other language), I would be delighted! See below.

The other way to specify a language is to use the LANGFILE command. This is useful if you want to download a new language from the analog home page, or if you want to translate one yourself, or even if you want to change some words or phrases or the way the dates and times are formatted in the output. The LANGFILE command tells analog in which file to find the various words and phrases for a new language. For example, the command

LANGFILE guarani.lng   # or
LANGFILE /usr/etc/httpd/analog/lang/guarani.lng

Some languages also have domains files or report descriptions files available. These are normally selected automatically by the LANGUAGE command. But you can tell analog to use different ones with the DOMAINSFILE and DESCFILE commands. Also, some languages have translations of the form interface or configuration file.

If you want to translate another language, I would be delighted! Do contact me first to make sure that no-one else is already translating the same language. The file README.txt in the language directory, and the English language file, contain some brief instructions for translating new languages.

Equally, if you find any mistakes in the output in different languages, please do let me know because I'm not able to check them all myself!

OUTFILE stats.htm

OUTFILE /usr/bin/httpd/htdocs/stats.html  # Unix
OUTFILE "Hard Disk:Server Apps:WebSTAR:Analog:Report.html" # Mac

You can include date codes in the OUTFILE in exactly the same way as for the LOGFILE. So for example,

OUTFILE stats%y%M%D.html

x  GENERAL         General Summary
1  YEARLY          Yearly Report
Q  QUARTERLY       Quarterly Report
m  MONTHLY         Monthly Report
W  WEEKLY          Weekly Report
D  DAILYREP        Daily Report
d  DAILYSUM        Daily Summary
H  HOURLYREP       Hourly Report
h  HOURLYSUM       Hourly Summary
w  WEEKHOUR        Hour of the Week Summary
4  QUARTERREP      Quarter-Hour Report
6  QUARTERSUM      Quarter-Hour Summary
5  FIVEREP         Five-Minute Report
7  FIVESUM         Five-Minute Summary
S  HOST            Host Report
l  REDIRHOST       Host Redirection Report
L  FAILHOST        Host Failure Report
Z  ORGANISATION    Organisation Report
o  DOMAIN          Domain Report
r  REQUEST         Request Report
i  DIRECTORY       Directory Report
t  FILETYPE        File Type Report
z  SIZE            File Size Report
P  PROCTIME        Processing Time Report
E  REDIR           Redirection Report
I  FAILURE         Failure Report
f  REFERRER        Referrer Report
s  REFSITE         Referring Site Report
N  SEARCHQUERY     Search Query Report
n  SEARCHWORD      Search Word Report
Y  INTSEARCHQUERY  Internal Search Query Report
y  INTSEARCHWORD   Internal Search Word Report
k  REDIRREF        Redirected Referrer Report
K  FAILREF         Failed Referrer Report
B  BROWSERREP      Browser Report
b  BROWSERSUM      Browser Summary
p  OSREP           Operating System Report
v  VHOST           Virtual Host Report
R  REDIRVHOST      Virtual Host Redirection Report
M  FAILVHOST       Virtual Host Failure Report
u  USER            User Report
j  REDIRUSER       User Redirection Report
J  FAILUSER        User Failure Report
c  STATUS          Status Code Report

You can turn each report on or off with configuration commands like

FIVEREP OFF
REFSITE ON

DESCRIPTIONS OFF

DESCFILE descriptions.txt

You can turn the "Go To" lines in the output off with the command

GOTOS OFF

You can turn off the "Program started at" line at the top of the output, and the "Running Time" line at the bottom, with the command

RUNTIME OFF

The figures in parentheses in the General Summary are for the last seven days: either the seven days before the TO time, or if no TO time is given, the seven days before the time of the program start. The figures for the last seven days are normally included if some, but not all, of the requests fall in those seven days; but you can turn them off by means of the command

LASTSEVEN OFF

You can change the order of the reports by means of the REPORTORDER command. You should list the code letters for all possible reports in the order you want them. Non-alphanumeric characters are ignored and so can be used as separators. For example,

REPORTORDER x-1QmdDhHw4567W-cPz-ritEIYy-SlLZo-sNnfKk-ujJ-vMR-bBp

GENSUMLINES ALL

GENSUMLINES -KL

GENSUMLINES +K

GENSUMLINES CDFGHM

 

Successful requests (always listed)

B

Average successful requests per day

C

Logfile lines without status code

D

Successful requests for pages

E

Average successful requests for pages per day

F

Failed requests

G

Redirected requests

H

Requests with informational status code

I

Distinct files requested

J

Distinct hosts served

K

Corrupt logfile lines

L

Unwanted logfile entries

M

Data transferred

N

Average data transferred per day

IMAGEDIR img/   # relative URL: within the same directory as the output
IMAGEDIR /img/  # off the root directory of your server
IMAGEDIR http://www.myother.server.com/img/  # on another server

You can use gif images instead of png's for the bar charts by specifying

PNGIMAGES OFF

LOGO picture.gif  # for this file
LOGO /images/picture2.gif  # a different file
LOGO none         # for no logo

The LOGOURL command specifies a URL to link the logo to. If you change the LOGO, you probably want to change the LOGOURL as well. For example,

LOGOURL http://www.mycompany.com/
LOGOURL none   # for no link

There are commands HOSTNAME and HOSTURL which affect the name and link at the end of the title line. For example, I might specify

HOSTNAME "Stephen Turner"
HOSTURL  http://homepage.ntlworld.com/adelie/stephen/

HOSTNAME "M\üller & S\öhne"

HEADERFILE none

STYLESHEET /housestyle.css
STYLESHEET none   # to cancel it

There is a command CSSPREFIX to add a prefix to all the CSS class names used in the XHTML output style. This is useful to avoid clashes with other style sheets: the disadvantage is that it will make your output longer. For example,

CSSPREFIX anlg
CSSPREFIX none    # to cancel it

SEPCHAR " "
REPSEPCHAR none
DECPOINT ,

There is a command called RAWBYTES. Specify RAWBYTES ON if you want the exact number of bytes to be listed, or RAWBYTES OFF if you want the number of kilobytes or Megabytes as appropriate to be listed instead.

If RAWBYTES is OFF (which is the default), then you can use the BYTESDP command to specify how many decimal places you want the bytes rounded to. The default is 2, which will display numbers like "91.26 kilobytes".

You have to be careful using this command. Because of daylight savings time in operation in different parts of the world at different times, analog cannot attempt to convert between different timezones. So it's your responsibility to set the right offset for different times of year. For example, if you were in Chicago, but your server was recording time in GMT, you would need to specify two different time offsets, one of minus five hours for summer and one of minus six hours for winter. You would need to split your logfiles in the right places and then run commands like

LOGTIMEOFFSET -300
LOGFILE summer*.log
LOGTIMEOFFSET -360
LOGFILE winter*.log

There is also a related command called TIMEOFFSET. This tells analog how much to offset the time of the computer on which it is running (rather than the computer running the server), to get your local time.

Time reports

By the way, in the following lists, don't get confused between the commands for the Quarterly Report (which begin with QUARTERLY) and those for the Quarter-Hour Report and Quarter-Hour Summary (with begin with QUARTERREP and QUARTERSUM respectively).

R

Number of requests

r

Percentage of the requests

P

Number of page requests

p

Percentage of the page requests

B

Number of bytes transferred

b

Percentage of the bytes

HOURSUMCOLS Pb

DAYREPGRAPH P

DAYREPGRAPH b

BARSTYLE a  
BARSTYLE b  
BARSTYLE c  
BARSTYLE d  
BARSTYLE e  
BARSTYLE f  
BARSTYLE g  
BARSTYLE h  
BARSTYLE i  
BARSTYLE j  

MONTHBACK ON  # Monthly Report backwards
WEEKBACK OFF  # Weekly Report forwards

QUARTERREPROWS 96  # only the last day's worth
MONTHROWS 0        # 0 means no restriction: show all time

MARKCHAR =

There is a parameter called MINGRAPHWIDTH which sets the minimum nominal size of the graphs. For example, if you set

MINGRAPHWIDTH 10

There is one more command which affects the time reports. You can specify which day should be counted as the first day of the week. This affects the layout of the Daily Report, Daily Summary, Weekly Report and Hour of the Week Summary. For example, our local student newspaper publishes a new edition on the web every Friday, so they like to specify WEEKBEGINSON FRIDAY for their reports.

In the next section, we'll look at commands relating to the non-time reports.

Other reports

First, these reports have COLS commands, just like the time reports. (See the section on Time reports for how to use these commands.) But for these reports, several additional columns are available. Here is the full list of columns for the non-time reports

R

Number of requests

r

Percentage of the requests

S

Number of requests in the last 7 days

s

Percentage of the requests in the last 7 days

P

Number of page requests

p

Percentage of the page requests

Q

Number of page requests in the last 7 days

q

Percentage of the page requests in the last 7 days

B

Number of bytes transferred

b

Percentage of the bytes

C

Number of bytes transferred in the last 7 days

c

Percentage of the bytes in the last 7 days

d

Date of last access

D

Date and time of last access

e

Date of first access

E

Date and time of first access

N

The number of the item in the list

REQCOLS NRSD

REQUESTS

total number of requests

REQUESTS7

requests within the last 7 days

PAGES

total requests for pages

PAGES7

requests for pages within the last 7 days

BYTES

total bytes transferred

BYTES7

bytes transferred within the last 7 days

FIRSTDATE

time of first request

DATE

time of most recent request

ALPHABETICAL

alphabetically

RANDOM

unsorted, sometimes useful for speed in very long reports

HOSTSORTBY ALPHABETICAL

There is one known bug concerned with SORTBY ALPHABETICAL. The report is sorted before any output alias is applied. This means that if an output alias has been specified for the report, then the report may appear not to be sorted correctly.

DOMFLOOR 1000r       # all domains with at least 1000 requests
DOMFLOOR 100s        # at least 100 requests within the last 7 days
DOMFLOOR 1000p       # at least 1000 requests for pages
DOMFLOOR 100q        # at least 100 requests for pages within the last 7 days
DOMFLOOR 1000000b    # at least 1,000,000 bytes transferred
DOMFLOOR 1kb         # at least 1 kilobyte (1024 bytes)
DOMFLOOR 10.5Mc      # at least 10.5Mb within the last 7 days
DOMFLOOR 0.5%r       # 0.5% of the total requests in the Domain Report
                     # (ditto %s, %p etc.)
DOMFLOOR 0.5:r       # 0.5% of the maximum number of requests for any domain
                     # (ditto :s, :p etc.)
DOMFLOOR 970701d     # last access since 1st July 1997
DOMFLOOR 970701e     # first access since 1st July 1997
DOMFLOOR -00-01-00d  # last access in last month (see
                     # documentation on FROM and TO commands)
DOMFLOOR -100r       # domains with top 100 number of requests
                     # (ditto -100s, p, q, b, c, d, or e)

r

REQUESTS

s

REQUESTS7

p

PAGES

q

PAGES7

b

BYTES

c

BYTES7

d

DATE

e

FIRSTDATE

a

ALPHABETICAL

x

RANDOM

+fp

means turn the Referrer Report on and sort it by page requests, but says nothing about the floor;

+f100s

means list all referrers with at least 100 requests in the last 7 days, but says nothing about the sort method;

+fb10000

means list all referrers with at least 10,000 bytes, sorted by bytes;

+fa-000101d

means list all referrers with accesses this year, sorted alphabetically.

REQCHART OFF

REQCHART ON

REQUESTS

total number of requests

REQUESTS7

requests within the last 7 days

PAGES

total requests for pages

PAGES7

requests for pages within the last 7 days

BYTES

total bytes transferred

BYTES7

bytes transferred within the last 7 days

The full list of CHART commands is HOSTCHART, REDIRHOSTCHART, FAILHOSTCHART, ORGCHART, DOMCHART, REQCHART, DIRCHART, TYPECHART, REDIRCHART, FAILCHART, REFCHART, REFSITECHART, SEARCHQUERYCHART, SEARCHWORDCHART, INTSEARCHQUERYCHART, INTSEARCHWORDCHART, REDIRREFCHART, FAILREFCHART, BROWREPCHART, BROWSUMCHART, OSCHART, VHOSTCHART, REDIRVHOSTCHART, FAILVHOSTCHART, USERCHART, REDIRUSERCHART, FAILUSERCHART, STATUSCHART, SIZECHART and PROCTIMECHART. Again, not every chart method is available for every report. You can also use

ALLCHART ON
ALLCHART OFF

The pie charts are normally written to the same directory as the OUTFILE. But you can specify a different location using the CHARTDIR and LOCALCHARTDIR commands. If the OUTFILE is standard output, you must use these commands, or you will not get any pie charts. Also, if you are writing two output files to the same directory, you must use these commands, or one set of images will overwrite the other. And you also need to use these commands if you are using the form interface, but there are additional issues in that case.

You have to use both of the commands before they have any effect. The CHARTDIR is the location of the pie chart directory on your server, similar to the IMAGEDIR; it's used for putting in the <img> tag to include the image. The LOCALCHARTDIR is the location on your local disk; it's where the image is written to. For example, you might have

CHARTDIR /images/
LOCALCHARTDIR /usr/local/apache/htdocs/images/

LOCALCHARTDIR "\webimages\analog\"

CHARTDIR /stats/rep1
LOCALCHARTDIR /usr/local/htdocs/stats/rep1

If you compiled your own version of analog, and you used your version of libgd instead of mine (see the Makefile for how to do this), then you can use the command

JPEGCHARTS ON

Here are the exact rules for which wedges are plotted in the pie chart. Up to ten wedges, plus "Other", are drawn, but wedges are only drawn if they are large enough. Also, wedges are only drawn if the item is listed in the main table for the report. And the whole chart will not be plotted if it would contain only one wedge.

If you want to plot more detail on the pie charts, for example Windows 98 & Windows NT instead of just Windows on the Operating System Report chart, see the next section.

REPORTSPAN ON

REPORTSPANTHRESHOLD 60

REPORTSPANTHRESHOLD 0

There's one other command which affects the links in the Request Report, Redirection Report and Failure Report. The command BASEURL prepends an additional string to the URLs in the target of the link. For example, after the command

BASEURL http://www.statslab.cam.ac.uk

In the next section, we'll look at commands for generating hierarchical reports, which are closely related to the commands in this section.

Hierarchical reports

First, you need to be able to control what gets listed in the reports. For this you need to use the SUB family of commands. So, for example, the command SUBDIR /~sret1/* would ensure that the Directory Report would not only contain an entry for the sum of my files, but also one for each of my subdirectories, something like this:

29,111: /~sret1/
10,234:   /~sret1/analog/
 5,179:   /~sret1/backgammon/
11,908: /~steve/

SUBDOMAIN *.*

If you specify a SUB command, all the intermediate levels are included automatically. So, for example, after

SUBDOMAIN statslab.cam.ac.uk

Here are examples of the other four SUB commands:

SUBTYPE *.gz         # in the File Type Report
SUBBROW */*          # e.g. Netscape/4 in the Browser Summary
SUBBROW Netscape/*.*  # add minor version numbers for Netscape
REFDIR http://search.yahoo.com/*   # Referring Site Report
SUBORG *.aol.com     # Organisation Report
SUBORG *.*.com       # Break down all .com's

The SUBDOMAIN command (but none of the others) can include a second argument describing the subdomain. For example

SUBDOMAIN cam.ac.uk 'University of Cambridge'

SUBDOMAIN 131.111 'University of Cambridge'

One other use for the SUBDIR command is if you have used the second argument to the LOGFILE command. Suppose you have translated files like /index.html into http://www.mycompany.com/index.html. Then the command

SUBDIR http://*    # or
SUBDIR http://*/*

A sub-item is listed in a hierarchical report only if it is above the sub-FLOOR, and it is included with a SUB command, and it is not excluded because of an INCLUDE or EXCLUDE command, and its immediate parent is listed. For example, specifying

SUBDIR /*/*/
SUBDIRFLOOR -3r
SUBDIRSORTBY REQUESTS

The three file reports (Request Report, Redirection Report and Failure Report) and the three referrer reports (Referrer Report, Redirected Referrer Report and Failed Referrer Report) are not fully hierarchical, but they do list search arguments together under the file to which they refer (provided that the arguments have been read in: see the ARGSINCLUDE command). So they have similar sub-FLOOR and sub-SORTBY commands, namely REQARGSFLOOR, REDIRARGSFLOOR, FAILARGSFLOOR, REFARGSFLOOR, REDIRREFARGSFLOOR and FAILREFARGSFLOOR; and REQARGSSORTBY, REDIRARGSSORTBY, FAILARGSSORTBY, REFARGSSORTBY, REDIRREFARGSSORTBY and FAILREFARGSSORTBY. The same applies to the Operating System Report with its subdivisions of operating systems: it has SUBOSFLOOR and SUBOSSORTBY.

OSCHARTEXPAND Windows

Just as for top-level items, sub-items will only be included if they are listed in the main table for the report. And so if an item does not have any sub-items listed, it will not be expanded, even if you asked for it to be. This means that you may need to use the appropriate SUB commands or SUBFLOOR commands to make sure that the sub-items do get listed. For example

SUBDOMAIN *.com
DOMCHARTEXPAND .com

Also, sub-items will only appear on the pie chart if they are large enough to get on in their own right. Note that if an item has any sub-items listed, and is expanded, then the item itself will no longer appear on the chart, only any sub-items which are large enough.

The domains file

DOMAINSFILE mydomains.tab

You should have got a domains file with the program, but if you've lost it, you can download one from http://www.analog.cx/ukdom.tab. It should contain on each line a domain code, followed by a number, followed by its location, like this:

ad  2  Andorra
ae  3  United Arab Emirates
[...]

There are some problems with this. A few countries have organisations at both levels 2 and 3 (for example asaspace.at and univie.ac.at). In those cases I've favoured false negatives over false positives by using the bigger number. (Also there is a correction which will make most of them right again: the first component is always removed from a hostname of three or more components.) For other countries, I don't have enough information to tell what the level should be. I've just given those a 1. Do let me know if you have any more information, or corrections, for the numbers.

For numerical addresses, the organisation is considered to be at level 2 if the first component is 24, 61-68, 80-81 or 128-255; and otherwise at level 1. Again this is only an approximation -- for example, many organisations use two adjacent blocks of numbers, or subdivided blocks -- but it's the best we can do without looking up every address we come across. (Note that you can always see more detail using the SUBORG command).

Lines starting with a hash (#) in the domains file are considered to be comments.

Computer-readable output style

OUTPUT COMPUTER

Each line in the output is separated into fields by means of a special string. You can specify this string by means of the COMPSEP command; for example, you could use

COMPSEP ,

COMPSEP \t

After that, there follows a field indicating the remaining columns in the report (using the letters RrSsPpQqBbCcDdEeN as usual). In hierarchical reports (including the reports which can show search arguments) there is an additional column l at the beginning, indicating the level in the hierarchy.

Finally there are the numerical data for each column and then the name of the item. Times actually take up several fields: year, month, date, hour & minute, or as many of those as are necessary to identify the time. (Year and quarter in the case of the Quarterly Report).

So here is an example line from the Domain Report, showing the third-level domain cam.ac.uk with 43 requests and 3.516% of the bytes.

o     lRb     3     43     3.516     cam.ac.uk

The first lines of non-time reports can also contain overall information about the report. First, if the REPORTSPAN for the report is wanted, it will be listed in lines with *FR and *LR instead of the normal column letters. Then there is a line listing the floor and sortby for the report. It has *f instead of the normal column letters, followed by the floor in the form it would be written for a FLOOR command, followed by the SORTBY using the code letters

r

REQUESTS

s

REQUESTS

p

PAGES

q

PAGES

b

BYTES

c

BYTES

d

DATE

a

ALPHABETICAL

x

RANDOM

VE

Version of analog

HN

HOSTNAME

HU

HOSTURL

PS

Program start time

FR

Time of first request

LR

Time of last request

E7

Time last 7 days ends

SR

Total successful requests

S7

Total successful requests in last 7 days

PR

Total successful requests for pages

P7

Total successful requests for pages in last 7 days

FL

Total failed requests

F7

Total failed requests in last 7 days

RR

Total redirected requests

R7

Total redirected requests in last 7 days

NC

Logfile lines without status code

C7

Lines without status code in last 7 days

NF

Number of distinct files requested

N7

Number of distinct files requested in last 7 days

NH

Number of distinct hosts served

H7

Number of distinct hosts served in last 7 days

CL

Number of corrupt lines in the logfile

UL

Number of unwanted lines in the logfile

BT

Total number of bytes transferred

B7

Total number of bytes transferred in last 7 days

Cache files

For most people, the cache file will not be needed: compressing the logfile using a standard compression utility such as gzip will be sufficient. Compressing a logfile is very efficient owing to the large number of repeated strings: I find about 12 times compression in practice. That in itself may solve your filespace problems, without needing to throw away any information.

The cache file is also not the best format for post-processing the data or feeding it into a spreadsheet. For that you should use the computer-readable output style.

Many people have trouble using the cache file, and end up accidentally recording corrupt data. You do need to understand what you're doing before you throw away your logfiles. See the discussion on Procedures below.

If you are going to use the cache file feature, it is also very important that you understand what is and what is not recorded. The summary is that all INCLUDE and EXCLUDE commands, including FROM and TO, and any ALIASes and LOGTIMEOFFSETs, must be applied when you create the cache file, not when you read it later. If you want different sets of options, you must create several cache files from the same logfile.

The reason for this is that it is not possible to reconstruct everything of interest in the logfile from the cache file. The cache file does contain information about the total number of requests for each host and each file, but not about, for example, which files were read by which hosts. (To do so would take up as much disk space as the compressed logfile.) So you cannot later look at only one file and see which hosts read that file.

Another way to look at this: if you do, for example, a HOSTEXCLUDE when reading the cache file, you are not doing a genuine HOSTEXCLUDE because files that that host read will still be included. You are only excluding those hosts from the Host Report, Organisation Report and Domain Report. This is why you must do all the inclusions and exclusions you want when you create the cache file.

When analog reads in a cache file, it does not apply any more aliases to the items. This is to avoid double-aliasing. So you must do any aliases you want at the time you create the cache file. Similarly, it does not obey the LOGTIMEOFFSET variable, to avoid double-offsetting, so any offset you want must be applied at cache-creation time too.

Also, the cache file does not contain data about the number of requests for each item in the last seven days: it can't, because the figures will be different by the time they are wanted.

Finally, times are only recorded to five-minute resolution.

CACHEOUTFILE none

You can read in a previously-made cache file with the CACHEFILE command, or with the +U command line option. This works exactly the same as the LOGFILE command, so you can use commas and wildcards to read in several cache files, and read compressed cache files using the UNCOMPRESS mechanism.

If the name of the CACHEFILE or the CACHEOUTFILE doesn't include a directory, it will be looked for, or written to, wherever analog expects to find its cache files. (This location is built in when the program is compiled.) For example, on Windows it would be in the same folder as the analog executable. But a cache file specified in a +U command line option is within the current directory.

It is possible (and useful) to analyse some CACHEFILEs and some LOGFILEs together. LOGFILE and CACHEFILE commands are basically cumulative, except that any logfiles and cache files in the mandatory configuration file or configuration files loaded from there override any on the command line or in configuration files specified on the command line, which themselves override any in the default configuration file or configuration files loaded from there, which in turn override compile-time options. Usually you don't need to worry about this, and it will do what you expect.

Sometimes you don't want to record all the types of item in the cache file. You might want to forget about which hosts had accessed your web site, for example, and only remember how many times each file was requested. You can choose not to include one type of item in the cache file by setting its LOWMEM to 3; for example, specify

HOSTLOWMEM 3

Procedures

Here is one way to use the cache files correctly. It's not the only correct way, but I think it's conceptually the simplest. The idea is that whenever you start a new logfile, you make a cache file out of the old logfile. So each cache file contains all the data from one, and only one, logfile. You never use old cache files to make new ones: so you never have a CACHEFILE and a CACHEOUTFILE command in the same configuration file.

Here is the procedure.

1. Rotate your logs: that means, archive the old logfile, and restart the server with a fresh logfile. (There are several standard tools to do this: or see your server documentation.)
2. Make both a cache file and an ordinary report from the old logfile. You can do this simultaneously by using one LOGFILE command, one OUTFILE command, and one CACHEOUTFILE command.
3. Make a test report from the cache file (using CACHEFILE and OUTFILE but no LOGFILE) and compare it against the report from the logfile to check it works. (This step really is worth doing!)
4. Now you can throw away the old logfile, if you've really understood what data you're losing by doing so. (But please remember that I can take no responsibility if something goes wrong: see the licence.)
5. When you want to make the main report, you can now use all your cache files and the current (not-yet-cached) logfile.

DNS lookups

First a note. Because analog's DNS lookups use only standard, platform-independent code, they are slower than lookups optimised for particular platforms. (This is especially true on platforms without the DNSTIMEOUT command described below.) So if you are doing DNS lookups regularly, you will probably be better off using one of the many specialist DNS lookup programs which you can find on the helper applications page.

DNSFILE dnscache

There are four possible levels of DNS activity. If you specify DNS NONE, no numerical addresses will be resolved. If you specify DNS READ, then analog will read the DNS file for old lookups, but no new lookups will take place. This mode is suitable if you are running analog while not connected to the internet. The third level is DNS WRITE. This reads the old file, looks up new addresses, and adds them to the file. (The first time you use DNS WRITE, you will get a missing-file warning as it tries to read the old file, but it will exist the next time.) The final level is DNS LOOKUP. This reads the old file and looks up new addresses, but doesn't add the new addresses to the file, so that they will not be remembered for next time. This is not normally a level that the user wants to specify, but analog will switch to this the behaviour if DNS WRITE fails for some reason.

If you are using a HOSTEXCLUDE command, you need to exclude the numerical IP address if it can't be resolved, or the name if it can. In other words, exclude whatever the host is known as in the Host Report.

DNSLOCKFILE filename

If analog crashes, it may not clear up the lock file, so in that case you may have to delete it yourself. (Disclaimer: on some systems, race conditions may occasionally thwart this mechanism, but this is very unlikely.)

Analog never deletes anything from the DNS file: this means that the DNS file will grow, and can become quite large. You should delete the top of it every so often. There is a program on the helper applications page to help you do this more systematically.

There are two parameters which say how long to trust old lookups for. If you set

DNSGOODHOURS 672

On some platforms (maybe only Unix) you can set a parameter called DNSTIMEOUT. If the DNS server still hasn't returned a reply within this many seconds, then the lookup will be aborted. However, this feature is off by default, because it breaks DNS lookups on many platforms. (The symptom is that the first few lookups succeed, but after a while, all lookups fail). If you try it, monitor it, and turn it off again if it causes problems.

Finally, there is a debugging command, DEBUG +D to show all the DNS lookups that analog is making.

timestamp IP_address name

Coping with low memory

Recall what happens to an item when it has been read in. First it is aliased. Secondly, it is checked to see whether it is included or excluded. Then finally, if all the items are wanted, one request is added to its score.

Normally the name of the item is saved before the aliasing takes place. This avoids analog having to do the aliasing again next time the same item is encountered. But this can take up more memory than necessary. So there is a family of LOWMEM commands provided, which tell analog to record the name at a later stage, or even not at all. If you use these commands, analog will have to do a bit more work than normal, but it will use less memory. On most sites, the hosts take up most of the memory, so I'll use the HOSTLOWMEM command as an example.

The command

HOSTLOWMEM 0

HOSTLOWMEM 1

HOSTLOWMEM 2

HOSTLOWMEM 3

Debugging

First, remember the option we mentioned before, to list the current settings of all of analog's variables. To get this, just put -settings on the command line, or SETTINGS ON in one of your configuration files, along with your other commands. Then analog will produce the list of settings instead of running in the normal way.

DEBUG ON

C

list all corrupt logfile lines

D

information about DNS lookups

F

information about file opening and closing

S

summary information about each logfile when it's closed

U

list unknown domains

V

list hosts without a domain (i.e., without a dot)

DEBUG FS

DEBUG +CD

DEBUG -CD

There is also a command line abbreviation for this command. Use +V (for ON), -V (for OFF), +VFS (to select exactly options FS), +V+FS (to add those options), and +V-FS (to remove them).

The C messages actually come on two lines. The first line gives the logfile line which was corrupt. The second line indicates where analog first noticed a problem. (This is usually, but not always, close to where the problem actually was!) In fact, each "line" of the message may spread over more than one line on your screen, and you have to be careful to take that into account when trying to find out where the logfile line was corrupt.

C

invalid configuration specified

D

dubious configuration specified

E

ERRFILE command used (see below)

F

files missing or corrupt

L

apparent problems in logfiles

M

possible problems in logfiles

R

turning off empty reports

Warnings range from the probably harmless to the usually serious. So it is generally not a good idea to turn all warnings off, or you might miss some important information. If you want to ignore a particular warning, turn just its category off.

There is also a command line version of the WARNINGS command, looking like +q, -q, +q<options>, +q+<options> or +q-<options>. (The syntax is the same as the +V command above.)

ERRFILE newfile

ERRFILE stderr

ERRLINELENGTH 0

PROGRESSFREQ 20000  # say

Form interface and CGI program

Important: For security reasons, you must not attempt to run analog itself as a CGI program, or even leave it in the directory or folder with your web files or CGI programs. When the form interface runs analog for you, it checks that analog isn't given any dangerous options. Without this check, your system could be vulnerable to attack.

The form interface is suitable for ordinary users to use, but it needs to be set up by a system administrator or other expert. In order to set it up, you have to be running a web server. You need to know what CGI programs are, where they live on your server, and how to set up their permissions properly. You also need to know how to write HTML forms. I shall assume this level of background knowledge for the rest of this section. And you have to be running Perl 5.001 or later: see Technical details below for other system requirements. (Actually, if you're on Windows and don't have Perl, you can download an executable version of the form interface from the helper applications page.)

Please don't try and set up the form until analog has been set up and is running properly on its own. It just adds another level of complexity to troubleshoot. And unlike analog itself, the form interface will not run "out of the box". You have to read the whole of this section to find out how to set it up safely.

Warning: CGI programs can contain security loopholes which allow an unscrupulous user to harm your system. (If you don't know about this, you shouldn't be running CGI programs at all. Read and understand the World Wide Web Security FAQ and the CGI Security FAQ first.) I have tried to make this form interface safe, but I cannot guarantee it. Even the most carefully-designed CGI programs can accidentally have serious security bugs. And I take no responsibility if anything goes wrong: you use it at your own risk. (See the licence.) Furthermore, you should be aware that unless you take special measures like password protection or limiting anlgform.pl to specific hostnames, setting up the form interface implies making analog executable, and your logfiles analysable, by anyone on the internet. It's usually a bad idea to allow this, because it has obvious negative implications both for privacy and for the load on your system: an attacker can run multiple copies of analog causing a denial-of-service attack. There are more notes on security design in this program towards the end of this section.

The form interface consists of two parts: a form (called anlgform.html) to choose the options, and a cgi program (called anlgform.pl) to pass them to the analog program. Both anlgform.html and anlgform.pl must be configured to your system before they will work at all. There are instructions at the top of both files explaining how to do this.

The form which is distributed with the program should only be regarded as an example form. You can find forms in languages other than English in the lang directory. Or you can write your own if you prefer. In fact you don't actually need the form at all: if you want just to create a link to the cgi program, with the arguments passed after a question mark in the URL in the usual way, then that's fine.

Logfile name: <input type=text name="LOGFILE">

<select name=LOGFILE size=1>
  <option value="/var/log/apache/fred"> Fred's logfile
  <option value="/var/log/apache/jane"> Jane's logfile
</select>

There are a few commands which you can't specify on the form for security or performance reasons. The full list is *LOGFORMAT, LANGFILE, DESCFILE, HEADERFILE, FOOTERFILE, UNCOMPRESS, OUTFILE, CACHEOUTFILE, ERRFILE, LOCALCHARTDIR, DNS and SETTINGS; and the person setting up the form can add more. See the security notes below for the reasons for these exclusions, and for some more commands you might want to add to the forbidden list. You can, if you prefer, specify the commands which are allowed, rather than those which are forbidden.

Alias this file: <input type=text name="FILEALIAS1">
To this one: <input type=text name="FILEALIAS2">

Then there are FLOOR commands. To avoid users of the form having to know the syntax of these commands, you can if you want specify them in two halves, FLOORA and FLOORB, and they will be stuck together. For example, the form distributed with the program specifies

<br>Include all domains with at least
<input type=TEXT name="DOMFLOORA" maxlength=6 size=6>
<select name="DOMFLOORB">
  <option value=r>requests
  <option value=p>requests for pages
  <option value=b selected>bytes
</select>

Secondly, you can specify other configuration files to be included at specific times. When analog is called by the CGI program, it first processes the default configuration file as usual. Then it processes any configuration file specified by an option with name cg. Then it processes all the other commands which the CGI program specifies. After that, it processes any configuration file specified by an option with name cm. Finally, it processes the mandatory configuration file as usual. (You may therefore want two copies of analog, one for form use and one for non-form use, with different configuration files compiled in.) Note that the commands in the default and mandatory configuration files will contribute to the configuration: some of them may even override options specified on the form. For example, if the default configuration file contains an INCLUDE command, this may cause INCLUDE and EXCLUDE commands specified on the form to behave unexpectedly.

There are a couple of commands which the form always sets. These may override what you have set elsewhere. First, it sets either DNS READ (if a DNSFILE is set on the form) or DNS NONE (otherwise). Do not attempt to override this -- not only will you get timeout problems, but an attacker can then write to any file by setting DNSFILE.

The second command which the form always sets is WARNINGS FL, so that the less important warnings don't fill up your server's error log. You can override this by sending an explicit WARNINGS command from the form. And thirdly, it sets DEBUG -C to avoid filling up the error log if the LOGFORMAT is incorrectly configured: this can't be overridden from the form, only from the mandatory configuration file.

You won't get pie charts on the form unless you set a CHARTDIR and LOCALCHARTDIR in your default configuration file (LOCALCHARTDIR is disabled from the form for security reasons). And even if you do this, there will be a problem if two users try and run the form interface at the same time, because they will be trying to write the same images, so they may see broken images or each other's charts.

There is one small point about compressed logfiles. For security reasons, when using the form interface you need to specify the full pathname to the uncompression command in the UNCOMPRESS command in your configuration file.

Again for security reasons, analog checks the input from configuration commands more carefully when using the form interface before outputting it. One side-effect of this is that the JAPANESE-JIS character set won't work. Use one of the other Japanese character sets instead.

Form troubleshooting

First, does analog run properly on its own without anlgform?

Next, you can run anlgform.pl from the (DOS or Unix) command line. This is good enough to debug most problems. You can specify options in pairs like this:

anlgform.pl qv=1 LOGFILE=/some/log REQINCLUDE=pages

If it still doesn't work, check the following points:

1. Have you edited anlgform.pl and anlgform.html as instructed at the top of those files?
2. Do other CGI programs work on your server? Is anlgform.pl in the right place to be recognised as a CGI program by the server?
3. Look in the server's error log for clues. You might want to set WARNINGS ON before you do this, because by default only warnings in categories F and L are reported.
4. Sometimes it's helpful to set the ERRFILE in your analog configuration file (it won't work from the form) to catch any errors and warnings which may be getting lost. This is especially true on IIS which incorrectly sends errors to the browser instead of to an error log. If you are using Internet Explorer you will probably also need to disable the "friendly" error messages so that you can see the actual error message.
5. Are all relevant files (analog itself, logfiles, configuration files, auxiliary files such as domain files...) executable/readable by your web server?
6. If some form options don't seem to take effect, then check whether they are being overridden by a command in a configuration file. (Although SETTINGS is forbidden from the form, you might find it useful to set it in your default configuration file.)
7. If you get a long wait, then no data returned, the server is probably timing out the request before analog has finished. The remedy is to increase the timeout interval.
8. As explained above, the form always sets DNS READ or DNS NONE, WARNINGS FL, and DEBUG -C, overriding your default configuration file.
9. Again as explained above, pie charts may not appear or may appear wrongly.
10. Again as explained above, uncompressing of compressed logfiles doesn't work unless you use the full pathname in the UNCOMPRESS command.
11. And once again as explained above, the JAPANESE-JIS character set won't work from the form interface.

Security notes

First, you should think about who can run the form interface. Unless you take special measures like password protection or limiting anlgform.pl to specific hostnames, adding the form interface to your site implies making analog executable, and your logfiles analysable, by anyone on the internet, as often as they want. It's usually a bad idea to allow this, because of the obvious concerns both about privacy and about the load on your system. Unless you limit the total CPU available to any analog processes, it is easy for an attacker to run multiple copies of analog, causing a denial-of-service attack.

Certain commands are ignored by anlgform.pl and not passed to analog. The list of them can be found at the top of anlgform.pl. Here are the reasons for them. HEADERFILE and FOOTERFILE would place any file on your system within the output. The *LOGFORMAT commands would also allow any file to be read, because someone could designate each line to be a single filename and then just list the filenames. OUTFILE, CACHEOUTFILE, ERRFILE and LOCALCHARTDIR would allow people to write to your filespace; ERRFILE would also divert warnings away from your server's error log. UNCOMPRESS would allow a user to execute any command. DNS is forbidden because setting it higher than READ would normally cause the process to time out, and also because with DNS WRITE, the DNSFILE would be a file to write, not just a file to read. CGI would allow the user to generate syntactically incorrect output. PROGRESSFREQ would allow a user to conduct a denial-of-service attack by filling up your error log really, really fast (and DEBUG C is also disabled for the same reason.)

None of the above should be deleted (unless you are really, really sure that it's completely impossible for anyone other than yourself to run anlgform.pl). There are three other commands which are forbidden by default but which you could consider removing from the forbidden list. SETTINGS is included because it will give away the locations of some files on your system. But it is useful for diagnostic purposes, and you could consider removing it temporarily if you have trouble setting up the form. The other commands which are included are LANGFILE and DESCFILE. They are included because it is possible that another file could be exactly the right number of lines long to be accepted as a language file or report descriptions file, and then parts of it would get into the output. But it would have to be exactly the right number of lines long first. These commands shouldn't really be needed if your copy of analog is installed correctly, because the LANGUAGE command should find the right files. But if you want them, and you're prepared to take the risk described above, you can remove LANGFILE and/or DESCFILE from the list.

There are other commands which you might consider adding to the list. For example, it is theoretically possible (though rather unlikely), that another file on your system could conform sufficiently closely to one of the predefined log formats that analog could be persuaded to analyse it and so reveal some of its contents. If you're worried about this, or even if you want to force only one particular logfile to be analysed from the form, you can add the LOGFILE command to the list of forbidden commands. And you could add DOMAINSFILE for similar reasons. Or if you wanted to stop a user having control over which analog warnings were written to the error log, you could add WARNINGS to the list. (Possible attempted security violations detected by anlgform will always be written.)

You can of course add any command you like to the list. For example, a user can use any configuration file on your system unless you add CONFIGFILE. If you add a command, you must also add any aliases for it. Have a look in the source file globals.c for the same command under different names -- some commands have legacy names which I don't admit to in the documentation.

For more certainty, you can, if you prefer, configure anlgform so that you specify the commands which are allowed, rather than those which are forbidden. See the top of anlgform.pl for how to do this.

The arguments to LOGFILE and CACHEFILE commands are checked for containing only certain allowed characters (specifically, letters, digits, /\.:_*? space, and - between two {letter, digit, underscore}'s). This is because they could match an UNCOMPRESS command and thus be passed to the shell when the uncompress command is popen()'ed.

Apart from that, command names are checked for containing only letters and the digits 1 and 2; and the arguments to commands are checked for not containing control characters (actually characters 0-32 and 127-159; in particular newline characters are prohibited). The length of the commands isn't checked by anlgform.pl, but buffer overflow shouldn't be an issue as configuration commands are checked for length by analog.

By the way, the reason that I advise that analog itself shouldn't be used as a CGI program is that some servers, notably Microsoft IIS, allow users to pass command line arguments into a CGI program. And even if the program doesn't return the proper CGI headers, the output can be sent back to the user. This means that all the above checking of arguments is then thwarted. Of course, on servers on which you can't pass command line arguments to a CGI program, there are not the same security concerns, but then analog isn't very useful as a CGI program because if you can't pass any arguments, you can only get the default output.

Technical details

On Windows, you have to associate the .pl extension with the Perl executable so that Perl scripts are executed by Perl.

anlgform.pl will understand the GET or POST methods of form submission. The HTML spec says that GET should be used when, as in this case, running the program has no side effects. However, section 15.1.3 of the HTTP spec says that POST should be used if some of the options being passed might be confidential. Also, very long URLs, formed by specifying lots of options, can cause trouble to some older servers. So anlgform.html uses the POST method by default. However, the GET method will also work. For example, you could make a normal link to anlgform.pl with options specified after a question mark in the usual GET way.

What the results mean

• How the web works. This section discusses what happens when somebody connects to your web site, and what you can and can't find out about them. If you think that you can get statistics on how many people have visited your web site (or want to know why you can't), then this section is for you.
• Analog's reports. This section gives a summary of analog's reports, what they contain, and which commands influence each one.
• Analog's definitions. This section gives precise details on all of analog's terminology, exactly what is counted in each report, and so on.

How the web works

This section is fairly long, but it's worth reading carefully. If you understand the basics of how the web works, you will understand what your web statistics are really telling you.

So, what do you know about it? First, I make one request for your front page. You know the date and time of the request and which page I asked for (of course), and the internet address of my computer (my host). I also usually tell you which page referred me to your site, and the make and model of my browser. I do not tell you my username or my email address.

Next, I look at the page (or rather my browser does) to see if it's got any graphics on it. If so, and if I've got image loading turned on in my browser, I make a separate connection to retrieve each of these graphics. I never log into your site: I just make a sequence of requests, one for each new file I want to download. The referring page for each of these graphics is your front page. Maybe there are 10 graphics on your front page. Then so far I've made 11 requests to your server.

After that, I go and visit some of your other pages, making a new request for each page and graphic that I want. Finally, I follow a link out of your site. You never know about that at all. I just connect to the next site without telling you.

The other sort of cache is on a larger scale. Almost all ISPs now have their own cache. This means that if I try to look at one of your pages and anyone else from the same ISP has looked at that page recently, the cache will have saved it, and will give it out to me without ever telling you about it. (This applies whatever my browser settings.) So hundreds of people could read your pages, even though you'd only sent it out once.

You can also know what people told you their browsers were, and what the referring pages were. You should be aware, though, that many browsers lie deliberately about what sort of browser they are, or even let users configure the browser name. And some people use "anonymizers" which deliberately send false browsers and referrers.

1. You can't tell the identity of your readers. Unless you explicitly require users to provide a password, you don't know who connected or what their email addresses are.
2. You can't tell how many visitors you've had. You can guess by looking at the number of distinct hosts that have requested things from you. Indeed this is what many programs mean when they report "visitors". But this is not always a good estimate for three reasons. First, if users get your pages from a local cache server, you will never know about it. Secondly, sometimes many users appear to connect from the same host: either users from the same company or ISP, or users using the same cache server. Finally, sometimes one user appears to connect from many different hosts. AOL now allocates users a different hostname for every request. So if your home page has 10 graphics on, and an AOL user visits it, most programs will count that as 11 different visitors!
3. You can't tell how many visits you've had. Many programs, under pressure from advertisers' organisations, define a "visit" (or "session") as a sequence of requests from the same host until there is a half-hour gap. This is an unsound method for several reasons. First, it assumes that each host corresponds to a separate person and vice versa. This is simply not true in the real world, as discussed in the last paragraph. Secondly, it assumes that there is never a half-hour gap in a genuine visit. This is also untrue. I quite often follow a link out of a site, then step back in my browser and continue with the first site from where I left off. Should it really matter whether I do this 29 or 31 minutes later? Finally, to make the computation tractable, such programs also need to assume that your logfile is in chronological order: it isn't always, and analog will produce the same results however you jumble the lines up.
4. Cookies don't solve these problems. Some sites try to count their visitors by using cookies. This reduces the errors. But it can't solve the problem unless you refuse to let people read your pages who can't or won't take a cookie. And you still have to assume that your visitors will use the same cookie for their next request.
5. You can't follow a person's path through your site. Even if you assume that each person corresponds one-to-one to a host, you don't know their path through your site. It's very common for people to go back to pages they've downloaded before. You never know about these subsequent visits to that page, because their browser has cached them. So you can't track their path through your site accurately.
6. You often can't tell where they entered your site, or where they found out about you from. If they are using a cache server, they will often be able to retrieve your home page from their cache, but not all of the subsequent pages they want to read. Then the first page you know about them requesting will be one in the middle of their true visit.
7. You can't tell how they left your site, or where they went next. They never tell you about their connection to another site, so there's no way for you to know about it.
8. You can't tell how long people spent reading each page. Once again, you can't tell which pages they are reading between successive requests for pages. They might be reading some pages they downloaded earlier. They might have followed a link out of your site, and then come back later. They might have interrupted their reading for a quick game of Minesweeper. You just don't know.
9. You can't tell how long people spent on your site. Apart from the problems in the previous point, there is one other complete show-stopper. Programs which report the time on the site count the time between the first and the last request. But they don't count the time spent on the final page, and this is often the majority of the whole visit.

Defenders of counting visits etc. claim that these are just small approximations. I disagree. For example, almost everyone is now accessing the web through a cache. If the proportion of requests retrieved from the cache is 50% (a not unrealistic figure) then half of the users' requests aren't being seen by the servers.

Other defenders of these methods claim that they're still useful because they measure something which you can use to compare sites. But this assumes that the approximations involved are comparable for different sites, and there's no reason to suppose that this is true. Pirolli & Pitkow's results show that the figures you get depend very much on how you count them, as well as on your server configuration. And even once you've agreed on methodology, different users on different sites have different patterns of behaviour, which affect the approximations in different ways: for example, Pirolli & Pitkow found different characteristics of weekday and weekend users at their site.

Still other people say that at least the trend over time of these numbers tells you something. But even that may not be true, because you may not be comparing like with like. Consider what would happen if a large ISP decided to change its proxy server configuration. It could substantially change your apparent number of visits, even if there was no actual change in the traffic levels at your site.

I've presented a somewhat negative view here, emphasising what you can't find out. Web statistics are still informative: it's just important not to slip from "this page has received 30,000 requests" to "30,000 people have read this page." In some sense these problems are not really new to the web -- they are present just as much in print media too. For example, you only know how many magazines you've sold, not how many people have read them. In print media we have learnt to live with these issues, using the data which are available, and it would be better if we did on the web too, rather than making up spurious numbers.

Another, extremely well-written document on these ideas is Measuring Web Site Usage: Log File Analysis by Susan Haigh and Janette Megarity. Being on a Canadian government site, it's available in both English and French. Or for an even more negative point of view, you could read Why Web Usage Statistics are (Worse Than) Meaningless by Jeff Goldberg.

Analog's reports

You can get descriptions of each report within the output by using the DESCRIPTIONS and DESCFILE commands.

Note: The examples below are in the HTML output style. Most people are using the XHTML output style, but I've used the HTML output style for the documentation to make sure that it works on all browsers.

Top lines

General Summary

The figures in parentheses represent the seven days given at the top of this report: it's the seven days before the TO time if there was a TO command, or if not the seven days before the program was run.

You can't find out the number of visitors or visits you've had, and don't believe any program which tells you that you can. See the section on How the web works for a discussion of this.

You can turn this report on or off with the GENERAL command. You can control which lines are included using the GENSUMLINES command. You can include or exclude the figures for the last seven days with the LASTSEVEN command. You may get slightly different lines to those above, depending on exactly what's in your logfile.

Time reports

week beg.: #reqs: pages: 
---------: -----: -----: 
13/Sep/98: 69614: 25277: 
20/Sep/98: 10032:  5861: 

The timezone is whatever your server records time in -- usually your server's local time, or sometimes GMT. You can adjust it to another timezone with the LOGTIMEOFFSET command.

You can control whether each report is included or not with the appropriate ON or OFF command. You can control which columns are listed by the COLS commands. You can control which measurement to use for the bar charts and the "busiest" line by the GRAPH commands. You can determine how many rows are displayed with the ROWS commands. You can display the lines backwards or forwards in time by the BACK commands. You can change the graphic used for the bar charts with the BARSTYLE command.

Time summaries

day: #reqs: pages: 
---: -----: -----: 
Sun:  2031:  1193: 
Mon:  8001:  4668: 
Tue:     0:     0: 
Wed: 13934:  5915: 
[etc.]

You can control whether each report is included or not with the appropriate ON or OFF command. You can control which columns are listed by the COLS commands. You can control which measurement to use for the bar charts by the GRAPH commands. You can change the graphic used for the bar charts with the BARSTYLE command.

Other reports

#reqs: %bytes:       last date: file
-----: ------: ---------------: ----
 4123:  2.29%: 21/Sep/98 01:57: /~sret1/analog/
 3064:  0.15%: 21/Sep/98 01:54: /~sret1/analog/analogo.gif
 1737:  0.01%: 21/Sep/98 01:53: /~sret1/images/bar1.gif
 1692:  0.01%: 21/Sep/98 01:53: /~sret1/images/bar16.gif
 1685:  0.01%: 21/Sep/98 01:53: /~sret1/images/bar8.gif
67345: 97.54%: 21/Sep/98 02:04: [not listed: 8,175 files]

• The Host Report lists all computers which downloaded files from you.
• The Domain Report lists which countries those computers came from. (If you only get "unresolved numerical addresses", see the FAQ.)
• The Organisation Report attempts to list the organisations (companies, institutions, ISPs etc.) which the computer was registered under.
• The Host Redirection Report and Host Failure Report list all computers which encountered redirections or errors.
• The Request Report (the example above) lists which files were downloaded.
• The Directory Report lists which directories those files came from.
• The File Type Report lists the file types (actually, extensions) of those files.
• The File Size Report breaks them down by size.
• The Processing Time Report shows the time taken to serve each file.
• The Redirection Report lists the filenames which resulted in redirections: mainly directories without the final slash, and "click-thru"'s.
• The Failure Report lists the filenames which caused errors.
• The Referrer Report lists which pages linked to your files (and also pages which included your images).
• The Referring Site Report lists the servers those referrers were on.
• The Search Query Report and the Search Word Report list which search terms people used to find your site (provided you've used the appropriate SEARCHENGINE commands).
• The Internal Search Query Report and Internal Search Word Report list the search terms people used on scripts within your site (provided you've used the appropriate INTSEARCHENGINE commands).
• The Redirected Referrer Report lists the referrers which led to redirections.
• The Failed Referrer Report is essentially a broken link report.
• The Browser Report lists the detailed versions of browsers used, and the Browser Summary collects them by vendor. You should be aware that browsers can lie about what sort of browser they are.
• The Operating System Report lists the operating systems of the visitors whose browser types you know (as far as possible: it's not always possible to distinguish accurately between different Windows versions, for example, because the same browser can run on more than one Windows version). Which browsers count as robots is controlled by the ROBOTINCLUDE and ROBOTEXCLUDE commands.
• The Virtual Host Report lists the activity of your various virtual domains.
• The Virtual Host Redirection Report and Virtual Host Failure Report give the number of redirections and errors on each of those domains.
• The User Report lists your visitors if your server requires authentication; or perhaps the visitors' cookies or session ids.
• The User Redirection Report and User Failure Report list the users who encountered redirections or errors.
• The Status Code Report lists the number of each HTTP status code that you had.

There are lots of commands which control these reports. As usual, you can control whether each report is included or not with the appropriate ON or OFF command. You can control which columns are listed with the COLS commands. You can change how the reports are sorted with the SORTBY commands. You can control how many items are listed with the FLOOR commands. You can control whether and how the pie charts are plotted with the CHART commands. You can list the time period covered by each report with the REPORTSPAN command. You can include or exclude individual items with the output INCLUDE and EXCLUDE commands. You can change the names of items in the reports with the output alias commands. Which files are linked to in the reports is controlled by the LINKINCLUDE and LINKEXCLUDE commands. The links are also affected by the BASEURL command.

The "not listed" line at the bottom counts those items which didn't get enough traffic to get above the FLOOR for the report. (It doesn't include items which you've explicitly excluded.)

Most of these reports have a hierarchical structure, like this example for the Domain Report:

no.: #reqs: %bytes: domain
---: -----: ------: ------
  1: 13243: 16.23%: .com (Commercial)
   :  1262:  1.26%:   aol.com
  2: 11783: 25.64%: .jp (Japan)
   :  9592: 22.19%:   ad.jp
   :  1043:  1.97%:   co.jp
  3: 10073: 11.62%: .net (Network)
   :  1926:  1.71%:   uu.net
  4:  9657: 13.31%: [unresolved numerical addresses]
  5:  7388:  8.04%: .uk (United Kingdom)
   :  5792:  5.74%:   ac.uk
   :  1510:  1.99%:   co.uk
   : 18502: 25.16%: [not listed: 82 domains]

You can control which items are listed on the lower levels by the SUB family of commands. There are also separate sub-SORTBY and sub-FLOOR commands for the lower levels. (These commands are called ARGSSORTBY and ARGSFLOOR for some reports, such as the Request Report.) You can plot the lower levels instead of the top-level items on the pie charts using the CHARTEXPAND commands.

Bottom lines

Analog's definitions

We start with some basic definitions. The host is the computer which has asked you for a file (often called the "client"). The file might be a page (i.e., an HTML document) or it might be something else, such as an image. By default filenames ending in (case insensitive) .html, .htm, or / count as pages, but you can tell analog to count any file as a page with the PAGEINCLUDE command.

The total requests counts all the files which have been requested, including pages, graphics, etc. (Some people call this the number of hits, but that word is also used in other ways by other people, so I avoid it). The requests for pages obviously only counts pages. One user can generate many requests by requesting lots of different files, or the same file many times.

The referrer for a request is the place that the user (or his computer) heard about your file from. If he followed a link to reach a page, it will be the previous page. In the case of a graphic on a page, the referrer will be the page containing the graphic.

Analog's kilobytes are 1024 bytes. (If you prefer to call these kibibytes, you can do so by editing your language file.)

First, successful requests are those with HTTP status codes in the 200's (where the document was returned) or with code 304 (where the document was requested but was not needed because it had not been recently modified and the user could use a cached copy). (Actually, you can configure code 304 to be a redirected request instead of a successful request with the 304ISSUCCESS command.) Successful requests for pages refers to those lines on which the file requested was named and was a page.

Redirected requests are those with other codes in the 300's, indicating that the user was directed to a different file instead. The most common cause of these requests is that the user has incorrectly requested a directory name without the trailing slash. The server replies with a redirection ("you probably mean the following") and the user then makes a second connection to get the correct document (although usually the browser does it automatically without the user's intervention or knowledge). The other common cause of redirected requests is their use as "click-thru" advertising banners.

Failed requests are those with codes in the 400's (error in request) or 500's (server error). They come about for a variety of reasons, but the most common are when the requested file is not found or is read-protected.

Finally, requests returning informational status code are those with status codes in the 100's. These are very rare at the moment.

There are a few other types of logfile lines listed in the General Summary. Lines without status code refers to those logfile lines without a status code, and the successful requests in the General Summary only counts the ones with a status code: except if the line contains the name of the file requested, and the filename is being counted (not starred in the LOGFORMAT), then it's counted as a success. Unwanted logfile entries are ones which you have explicitly excluded. Finally, corrupt logfile lines are those which analog didn't manage to parse. (The number given is the number of unparseable lines in the whole logfile, even if the rest of the analysis is restricted to a small part of the logfile, because analog doesn't know whether a line would have been wanted if it couldn't parse it! You can list all the corrupt lines by turning debugging on.)

The "not listed" line at the bottom of each of the non-time reports represents those items which were not listed because they were below the floor for the report. (It doesn't include items which you've explicitly excluded.)

The figures in parentheses in the General Summary are for the last seven days: either the seven days before the TO time, or if no TO time is given, the seven days before the time of the program start. (It would be nicer to use the seven days before the last time in the logfile, but we don't know when this is until we've read the whole logfile, and by then it's too late.) The figures for the last seven days are not included if all, or none, of the requests fall in the last seven days.

In the Domain Report, "domain not given" means that the hostname did not contain a dot. "Unknown domain" means that it did contain a dot, but that the domain name was not in the domains file (or that the domains file could not be read). The hosts and domains concerned can be listed by turning debugging on.

In the Operating System Report, which browsers count as robots is controlled by the ROBOTINCLUDE and ROBOTEXCLUDE commands.

Errors and warnings

First, you should understand the difference between a crash, an error, a warning, and a debugging message. First, a crash is when analog exits prematurely, without producing the whole output file. The system might give a message, but analog will not give one of its own messages. Analog should never crash. If it does crash, please tell me about it.

An error is something which stops analog finishing its job. Whenever an error is detected, analog gives a message starting something like analog: Fatal error: and will then tell you what type of thing went wrong before quitting.

A warning is a problem which is not fatal to analog: it will keep on with its processing. These vary from the possibly serious, such as files which could not be found, to purely informational. They produce a message starting analog: Warning. You can turn warnings off using the WARNINGS command.

Finally, a debugging message gives information on the state of the program. They just begin with a single code letter followed by a colon. You don't get any debugging messages unless you've asked for them.

If you want to send these messages to a file instead of to the screen, you can use the ERRFILE command. To tell analog the width of your screen for these messages, you can use the ERRLINELENGTH command.

Now I shall describe all the possible errors and warnings in detail.

Errors

Ran out of memory: cannot continue

Analog ran out of memory. Try increasing the memory available to the process, if your operating system will allow it, or using the LOWMEM commands.

Cannot ignore mandatory configuration file

See the section in the Readme on the mandatory configuration file.

Can't find language file
Language file too short
Language file too long
Language file contains excessively long lines

Analog can't run without a well-formed language file. See the documentation on language files.

Attempted to read more than 50 configuration files

The most likely explanation for this is that you have accidentally created a loop using the CONFIGFILE command, for example if a configuration file includes itself.

Incorrect default given in anlghead.h
Default given in anlghead.h too short

If you've compiled your own version, and you've specified an incorrect configuration in the file anlghead.h, analog gives up to allow you to fix it.

Failed to open output file for writing

Analog couldn't create, or couldn't write to, the output file you specified.

Cache output file already exists: won't overwrite

Analog won't overwrite an old cache file. You must move or delete it yourself first.

OUTFILE and CACHEOUTFILE are the same

OUTFILE and CACHEOUTFILE both set to stdout

This can't be what you wanted, because one would overwrite the other.

OUTPUT NONE and CACHEOUTFILE none selected

You requested no output.

OUTPUT LATEX only available with US-ASCII, ISO-8859-1 and ISO-8859-2 character sets

The LaTeX output style only works with certain European languages because the standard LaTeX distribution doesn't contain the characters for other languages.

Warnings

C

invalid configuration specified

D

dubious configuration specified

E

ERRFILE command used

F

files missing or corrupt

G

corrupt lines in support files

L

apparent problems in logfiles

M

possible problems in logfiles

R

turning off empty reports

Category C

Category D

LOGFORMAT with no subsequent logfile

You have specified a LOGFORMAT command, but no subsequent logfile to which it could be applied. Most likely you put the LOGFORMAT after the LOGFILE command. You must put the LOGFORMAT before the LOGFILE command or use DEFAULTLOGFORMAT instead. See the section on Specifying a log format for more details.

Offset not a multiple of 30
Offset more than 25 hours

The time offsets are meant to be for correcting between differences in time zones. These differences are usually multiples of 30 minutes between -25 and +25 hours. Maybe you specified the offset in hours instead of minutes by mistake, or something like that.

FROM time is later than the present

Usually this will mean that no entries are counted. Analog doesn't try and correct it in case the clock on your computer or your server is wrong -- but you would be better using TIMEOFFSET or LOGTIMEOFFSET to correct those clocks.

SORTBY doesn't match FLOOR
SORTBY doesn't match SUBSORTBY (or FLOOR/SUBFLOOR)
SORTBY (or FLOOR or GRAPH) isn't included in COLS

Within one report, it's helpful to your readers to have the sort methods and the floors compatible, and all included in the COLS. (See the section on Non-time reports).

Column N with SORTBY ALPHABETICAL/RANDOM

Numbering off the items when they're not in order of busyness is probably a mistake.

Time reports have not all got same value of BACK

It's usually helpful to have all the time reports running in the same direction.

Report contains no COLS

You've got an empty COLS list for one report, so you'll just get a list of names, not any information about them.

LOWMEM 3 prevents that item being cached

You're making a cache file, but one item is not being recorded because of a LOWMEM command, and will therefore not be saved in the cache file.

Category E

Redirecting future diagnostic messages

You've used an ERRFILE command to change the destination of errors, warnings, debugging and PROGRESSFREQ diagnostics. This is just warning you so that you don't miss any messages.

Category F

Can't auto-detect format of logfile

The LOGFORMAT is set to automatic detection, but the first line of the logfile is not in any of the standard formats. This error can often be generated when you try and specify your own LOGFORMAT but put it after the LOGFILE command so that it is not in effect for that logfile.

Logfile with ambiguous dates

Some servers, notably IIS and WebSite, record dates in their logfiles according to local conventions. Then if analog encounters 2/1/99, for example, it doesn't know whether it's the 2nd January or 1st February. This problem, and what to do about it, is described in more detail in the section on Choosing a logfile.

Logfile seems to be in Microsoft format

The most common cause of this message is that your IIS logfile uses local conventions for listing dates which analog doesn't know about -- e.g. 25.12.98 instead of 25/12/98. Again, see the section on Choosing a logfile for help.

Ignoring corrupt format line in logfile

The format line within a W3 extended log, Netscape log or WebSTAR log is invalid in some way. Analog will always tell you what's wrong with it. The most common problem is that the date only appears at the top of the logfile, not on every line, which is not allowed. The reason for this, and what to do about it, are in the section on Choosing a logfile.

Failed to open domains file

In this case, all domains will be counted as "unknown domains".

Failed to open DNS input file

The first time you use DNS lookups, you don't have a DNS cache file, so you get this warning. Assuming you are using DNS WRITE, the message will go away next time you run analog.

DNS lock file already exists

To stop two copies of analog trying to write the DNS file at the same time, an empty "lock file" is created, which tells the second copy of analog to use DNS LOOKUP instead of DNS WRITE. If analog crashes, it may not delete its lock file. So if you get the "already exists" message even though no other copy of analog is running, you may need to delete the lock file yourself.

Category G

Category L

Large number of corrupt lines

This could indicate a problem with the logfile, or with the LOGFORMAT specification. The possible causes are described in the section on Choosing a logfile. If you specify DEBUG ON, analog will report where each line was corrupt.

Logfiles overlap: possible double counting

This means that two logfiles which were counting the same type of item overlapped in time. Because it's only based on the time period of the logfiles, not the actual entries, this may or may not indicate a genuine problem. It is a problem if you read the same logfile twice. Or maybe you used the cache file feature incorrectly. Or maybe your web server produces several logfiles, and your LOGFORMAT specification should have told analog to ignore some of the items in some of the logfiles. It is not a problem if the logfiles are in fact completely disjoint; for example if you analyse logfiles from two different virtual hosts. In this case, the statistics produced will still be correct.

Category M

Logfile contains lines with no [whatevers], which are being filtered

This is usually harmless. It is perhaps best explained by example. Suppose you are excluding certain files from the analysis, but that you are also analysing a browser log which just contains information about the browsers used, not which files they read. Then we can't exclude the browsers which read the excluded files, because we don't know which they were, so all browsers will be included.

Logfile contains lines with no file names (or bytes): page (or byte) counts may be low

If a logfile line doesn't contain a file name, analog will assume that the request wasn't for a page. Similarly, if it doesn't give the number of bytes transferred, analog will make the bytes zero. So the number of page requests or bytes credited to the other items on that line will then be too low.

Old-style cache file doesn't contain data on first-request times of items; so these may be overestimated

Cache files now contain the first-request time of each item. But if you read a cache file from an older version of analog, this data will not have been recorded, and so the last-request time will be used instead.

Cache file doesn't contain last-seven-day statistics

It is impossible for cache files to record the number of requests in the last 7 days, because the data would be wrong at the time the cache file was read.

Category R

Broken Pipe

Frequently asked questions

1. Getting Started
2. Basic Configuration
3. Understanding the Output
4. Advanced Usage
5. Form Interface
6. Design Decisions

List of Questions

1. Getting Started
   See also Starting to use analog.
   1. Analog doesn't have a setup.exe.
   2. Analog just flashes up a DOS window and then quits.
   3. When I try and edit analog.cfg, Windows asks me which program I want to use to open that file.
   4. When I try and compile analog, it gives me an error (e.g. on SunOS 5).
   5. Analog didn't write the logfile when I ran it.
   6. Analog won't read extended logfiles generated by IIS.
      or What does "time without date" mean?
   7. What does "Logfile with ambiguous dates" mean?
   8. I tried to analyse a logfile, but I just got "Large number of corrupt lines."
   9. Analog won't read logfiles bigger than 2GB.
   10. What does this error message mean?
   11. I tried to run analog from my browser, but it didn't work.
   
   
2. Basic Configuration
   1. I want to analyse several logfiles together.
   2. I want to make several different statistics pages. Do I have to install several copies of analog?
   3. My analog.cfg included lots of CONFIGFILE commands, but only one output page was produced.
   4. Why does the Daily Report only show the last six weeks?
   5. Why do the time reports all list 0 requests?
      or Why are there no bar charts in the time reports?
   6. How do I get the Request Report to list files with fewer than 20 requests?
   7. How do I ignore accesses from my site?
   8. How do I ignore internal referrers in the Referrer Report?
   9. How do I get information on just my pages, not everybody's?
   10. How do I list subdirectories not just top-level directories in the Directory Report?
   11. How do I list minor browser versions in the Browser Summary?
   12. I want to see the full hostnames in the Organisation Report.
   13. I used the command "DIREXCLUDE /mydir/", but files in that directory were still listed.
   14. I used the command "FILEEXCLUDE /cgi-bin/script.pl", but that file was still listed in the Request Report.
   15. I used the command "IMAGEDIR C:\analog\images\", but I only got broken images.
   16. I want to put several output pages in the same directory, but the pie charts overwrite each other.
   17. I want a configuration file with all of the possible configuration commands in it.
   18. I want to see your configuration file.
   19. Does the order of the commands matter in the configuration file?
   20. Why are my browser and referrer reports empty?
   21. Why isn't the Referrer Report sorted properly?
   22. I want to list (or not to list) referrers with their search arguments in the Referrer Report.
   23. Why are my click-thru's (or CGI scripts) not listed in the Request Report?
   24. I can't find /script.pl?q=1 in the Request Report.
   25. Why can't I have P in the REQCOLS, REQSORTBY or REQFLOOR?
   26. Can I find out which files each referrer pointed to?
       or Can I find out which files each host has read?
       or Can I find out which hosts have read each file?
       or Can I find out the number of hosts visiting on each day?
       or lots of similar questions.
   27. Can SETTINGS ON produce a configuration file instead of an English list of settings?
   28. I get the message "logfiles overlap" even though the two logfiles contain completely separate requests.
   29. Can I count the individual visitors, or visits, to my site?
       or Can I see how long visitors spend on my site?
   30. Can I change the way dates are formatted in the output?
       or Can I change some of the phrases in the output?
   31. How can I change the background colour of my output?
       or How can I make the output prettier?
   
   
3. Understanding the Output
   See also What the results mean.
   1. How do I find out the number of hits from your data?
   2. Why are there so many referrers from my own site?
   3. The analysis covers exactly a week, but the figures for the last seven days don't agree with the totals.
   4. I only have 240 requests in total. Why does analog think there are 840 requests per week?
   5. The pie charts don't agree with the figures in the tables.
   6. Why doesn't analog agree with the counter on my page?
   7. Why doesn't analog agree with grepping the logfile?
   8. Why doesn't analog agree with my other logfile analysis program?
   9. Why do I only get "unresolved numerical addresses" in the Domain Report?
   10. Why are directories listed in the Request Report?
   11. When someone reads one of my PDF files, it scores dozens of hits.
   12. Kilobytes should be 1000 bytes, not 1024 bytes.
   13. The Organisation Report doesn't identify organisations correctly.
   14. "Organization" isn't spelled correctly.
   
   
4. Advanced Usage
   1. How can I do such-and-such with a command line option?
   2. I want a list of all command line arguments.
   3. How do I list all numerical subdomains to depth 2 in the Domain Report?
   4. I want to be able to count requests with status code 301 and 302 as successes, so that they appear in the Request Report.
   5. I want to report on a field analog doesn't know about.
   6. Can analog analyse Squid proxy logfiles?
   7. Can analog analyse FTP logfiles?
      or Can analog analyse streaming media logfiles?
   8. Can analog analyse other logfiles, such as mail logs, or the syslog?
   9. How can I run analog automatically every day?
   10. How can I automatically email the results to myself or someone else?
   11. I'm setting up IIS. Which logfile format should I use?
   12. I host lots of virtual domains. How should I set up analog?
   13. Can I make several output pages with just one run of analog?
   14. I ran out of memory when trying to run analog. What can I do?
   15. You're processing 20,000,000 requests in under 10 minutes. Why is mine much slower?
       or Analog appears to stall.
   16. How do I make a link on my page that runs analog?
   17. Do I have to save all my old logfiles?
       or Can analog make statistics from old reports instead of reading the whole logfile again?
   18. Can analog write to a database or spreadsheet?
   
   
5. Form Interface
   See also Form troubleshooting.
   1. I couldn't make the form run.
   2. How can I specify different logfiles from the form interface?
   3. My browser showed me anlgform.pl, rather than running it.
   4. Why does the form interface give "Document Returned no Data"?
   5. The images don't appear when running analog from the form interface.
   6. Why do I get some reports that weren't requested on the form?
   7. How do I make a link to anlgform.pl without using anlgform.html?
   8. Is there a form interface not using Perl (e.g. ASP or .exe)?
   
   
6. Design Decisions
   1. Why doesn't the HEADERFILE replace the whole <head> of the output file?
   2. Why don't you just use one image, and scale it with the width and height attributes?
   3. In the XHTML output, why not just put a class= on each <COL> instead of on each cell?
   4. Why doesn't the OUTFILE command automatically create the directory for the output file?
   5. Why not automatically spot robots by whether they request /robots.txt?
   6. Why not just do DNS resolution of the hosts that actually make it into the Host Report?
   7. Couldn't you do the DNS lookups faster with threads?
   8. Why doesn't analog analyse the error_log?
   9. My server lists local names in the logfile. Can you put a common suffix on them automatically?
   10. Can you extrapolate from the current month's partial data to produce a prediction for the whole month, based on the rate so far?
   11. Can you extend the Domain Report to say which US states people visited from?
   12. Please distinguish between the different BSDs in the Operating System Report.
   13. Why not use language codes instead of country codes for the names of the language files?
   14. Why doesn't analog produce statistics on "visits"?
   15. Why don't you sell analog?

A. Getting Started

1. Analog doesn't have a setup.exe.
   No, and it doesn't need one. It's already ready to run! See Starting to use analog under Windows.
2. Analog just flashes up a DOS window and then quits.
   This is the correct behaviour. It should have created an output page called Report.html. See Starting to use analog under Windows.
3. When I try and edit analog.cfg, Windows asks me which program I want to use to open that file.
   Use Notepad, or any other plain text editor.
4. When I try and compile analog, it gives me an error (e.g. on SunOS 5).
   Maybe you need to edit the Makefile. There are some platform-specific notes in the section Starting to use analog on other platforms, and in the Makefile itself.
5. Analog didn't write the logfile when I ran it.
   Analog doesn't write the logfiles. Your web server writes the logfiles, and analog just reads them. See Starting to use analog.
6. Analog won't read extended logfiles generated by IIS.
   or What does "time without date" mean?
   By default, IIS writes the date only at the top of the logfile, not on every line. But it doesn't write a new date if the date changes during the logfile, so analog can't tell which date later entries in the log occurred on. More details, and what to do about it, are in the section on Choosing a logfile.
7. What does "Logfile with ambiguous dates" mean?
   See the section on Errors and warnings.
8. I tried to analyse a logfile, but I just got "Large number of corrupt lines."
   There are lots of possible reasons for this. You can find them described in the section on Choosing a logfile.
9. Analog won't read logfiles bigger than 2GB.
   Analog doesn't have any limitation, but there may be a limitation in your OS and/or compiler. Try adding CFLAGS=-D_FILE_OFFSET_BITS=64 to the Makefile before compiling.
10. What does this error message mean?
    Again, see the section on Errors and warnings.
11. I tried to run analog from my browser, but it didn't work.
    Analog should not be run as a CGI program, or even put in the folder with your CGI programs, for security reasons. You should use the special CGI program instead.

B. Basic Configuration

1. I want to analyse several logfiles together.
   Just use several LOGFILE commands, or wildcards in the logfile name.
2. I want to make several different statistics pages. Do I have to install several copies of analog?
   No. Just install it once, and run it with different configuration files. (You do have to run it once per output page though.)
3. My analog.cfg included lots of CONFIGFILE commands, but only one output page was produced.
   Analog can only produce one output page per run. To produce several reports, you have to run it several times.
4. Why does the Daily Report only show the last six weeks?
   This is controlled by the DAYREPROWS command.
5. Why do the time reports all list 0 requests?
   or Why are there no bar charts in the time reports?
   They probably only list 0 requests for pages. Maybe you need to use PAGEINCLUDE to count more files as pages.
6. How do I get the Request Report to list files with fewer than 20 requests?
   Use the REQFLOOR command, e.g., REQFLOOR 10r to list down to 10 requests. Also, if you want to list all the files not just pages, you may need to use the command REQINCLUDE *
7. How do I ignore accesses from my site?
   Use the HOSTEXCLUDE command.
8. How do I ignore internal referrers in the Referrer Report?
   Use the REFREPEXCLUDE command.
9. How do I get information on just my pages, not everybody's?
   Use the FILEINCLUDE command.
10. How do I list subdirectories not just top-level directories in the Directory Report?
    SUBDIR */*
11. How do I list minor browser versions in the Browser Summary?
    Use SUBBROW */*.* (Note that they then appear in the Browser Summary, not the Browser Report).
12. I want to see the full hostnames in the Organisation Report.
    The full hostnames are listed in the Host Report.
13. I used the command "DIREXCLUDE /mydir/", but files in that directory were still listed.
    DIREXCLUDE only affects the Directory Report, not the other reports. You want "FILEEXCLUDE /mydir/*" instead.
14. I used the command "FILEEXCLUDE /cgi-bin/script.pl", but that file was still listed in the Request Report.
    If the file has search arguments, you have to be a bit careful with FILEEXCLUDE. This is described in the section about search arguments.
15. I used the command "IMAGEDIR C:\analog\images\", but I only got broken images.
    The IMAGEDIR command has to be a URL, not a directory on your disk. (It's just inserted into the <img> tags in the output: have a look at the output and you'll see.) Also this means that the images have to be put in the part of your filespace that has your web files.
16. I want to put several output pages in the same directory, but the pie charts overwrite each other.
    You have to set the CHARTDIR and LOCALCHARTDIR to be different for each output. (You can still have all the charts in the same directory if the CHARTDIR and LOCALCHARTDIR don't end with slashes.)
17. I want a configuration file with all of the possible configuration commands in it.
    One is already distributed with the program, in the examples folder.
18. I want to see your configuration file.
    This is also included in the examples folder in the distribution.
19. Does the order of the commands matter in the configuration file?
    Only occasionally. If you have two of one command, the later one will generally override the earlier one. Apart from that, commands can come in any order, except that LOGFORMAT and LOGTIMEOFFSET commands must come before the LOGFILE to which they refer.
20. Why are my browser and referrer reports empty?
    Maybe your logfile doesn't contain any browser and referrer information?
21. Why isn't the Referrer Report sorted properly?
    It is sorted properly. But search arguments are also listed under the file they belong to, and this interrupts the ordering. If you set the REFARGSFLOOR high enough you won't see the search arguments. Or you can include the N column to make the ordering more obvious.
22. I want to list (or not to list) referrers with their search arguments in the Referrer Report.
    To see the search arguments you may need to set the REFARGSFLOOR lower. To avoid seeing them, you could set the REFARGSFLOOR higher, or alternatively use the REFARGSEXCLUDE command to ignore them either for all files or just for particular files.
23. Why are my click-thru's (or CGI scripts) not listed in the Request Report?
    If they cause a redirection to another page, they will be listed in the Redirection Report, rather than the Request Report.
24. I can't find /script.pl?q=1 in the Request Report.
    If it causes a redirection, it will be in the Redirection Report not the Request Report. But also, you may need to set the REQARGSFLOOR or REDIRARGSFLOOR lower to actually see it.
25. Why can't I have P in the REQCOLS, REQSORTBY or REQFLOOR?
    The number of page requests doesn't make sense in the Request Report because it's either the same as the number of requests (if the file is a page) or zero (if it isn't). If you want to list only pages in this report, use REQINCLUDE pages instead.
26. Can I find out which files each referrer pointed to?
    or Can I find out which files each host has read?
    or Can I find out which hosts have read each file?
    or Can I find out the number of hosts visiting on each day?
    or lots of similar questions.
    There are lots of questions like this. They all want analog to cross-reference two sorts of item (e.g. files and referrers in the first example above, or hosts and dates in the last).
    The solution is to use the *INCLUDE commands. For example, to find out which files a particular referrer pointed to, restrict the analysis to just the referrer you're interested in with the REFINCLUDE command. The Request Report will then just list the files reached from that referrer. Similarly, if you're interested in a particular time period, you can focus on just that time by using the FROM and TO commands.
    You do have to run analog once for each report generated. So for the example of referrers, you would have to run it once for each referrer you're interested in. Of course, it would be useful to be able to cross-refererence all the files and referrers at once. But it is fundamental to analog's speed and minimal memory requirement that it only records statistics for each type of item individually, and doesn't record enough information to cross-reference them afterwards.
27. Can SETTINGS ON produce a configuration file instead of an English list of settings?
    No. But it does tell you which configuration files it read, so you can just get the commands out of them. Or if you want a list of all configuration commands, there is one in the examples directory.
28. I get the message "logfiles overlap" even though the two logfiles contain completely separate requests.
    This message is based only on the dates of the files, not the contents. If you're sure there is no problem, you can turn it off with the command WARNINGS -L.
29. Can I count the individual visitors, or visits, to my site?
    or Can I see how long visitors spend on my site?
    No, it's not technically possible, and don't believe any program which tells you it is. See the section on How the web works for details.
30. Can I change the way dates are formatted in the output?
    or Can I change some of the phrases in the output?
    Yes, by editing the language file.
31. How can I change the background colour of my output?
    or How can I make the output prettier?
    You can change almost any aspect of the design using a style sheet. There are also some programs on the helper applications page to make completely different output from the analog data.

C. Understanding the Output

1. How do I find out the number of hits from your data?
   I don't use the word hits, because people use it in different ways, so it's misleading. I use requests for the number of transfers of any type of file (text, graphics, ...), and page requests for the number of transfers of HTML pages. See the section on Analog's definitions for more information.
2. Why are there so many referrers from my own site?
   These come from all the internal links on your site, and all the graphics on your pages. See the section on How the web works for more information. If you don't want to see them, you can use REFREPEXCLUDE to exclude them.
3. The analysis covers exactly a week, but the figures for the last seven days don't agree with the totals.
   The figures in parentheses are for the seven days before the time the program was run, unless there is a TO command. They are never for the seven days before the end of the logfile. (Although if you know that the logfile only contains entries up to a certain time, you may want to include a TO command for that time to get the last seven days' data right.)
4. I only have 240 requests in total. Why does analog think there are 840 requests per week?
   If you have 240 requests in two days, that's a rate of 840 requests per week. Just like if you drove 28 miles in 20 minutes, you'd have driven at 84 miles per hour.
5. The pie charts don't agree with the figures in the tables.
   Possibly you are looking at out-of-date images. Make sure to reload the images as well as the text. Also, if you are running analog several times, make sure to use CHARTDIR and LOCALCHARTDIR to stop the images for the different runs overwriting each other.
6. Why doesn't analog agree with the counter on my page?
   There are lots of possible reasons. Do they both start from the same date? Are you just looking at requests for that one page with analog, not for all your other pages and graphics? Also, analog will record all requests to that page; if it's a graphic, your counter will only measure requests from people on graphical browsers that reached that place on the page.
7. Why doesn't analog agree with grepping the logfile?
   Have you understood what analog includes in its counts? In particular, most reports only list "successful" requests (HTTP status codes 200-209 & 304). A naïve grep would count failures too.
8. Why doesn't analog agree with my other logfile analysis program?
   Small differences can be put down to different parsing. But if you are seeing large differences, you have to understand what analog counts, and what the other program counts. For example, some programs count HTTP status codes 301 & 302 as successes, whereas I think that to do so gives extremely misleading results.
9. Why do I only get "unresolved numerical addresses" in the Domain Report?
   Your server only records the numerical IP address of the hosts that contact you, not their names. Read the section about DNS lookups, or turn DNS resolution on in your server.
10. Why are directories listed in the Request Report?
    They are not directories, they are pages with the same name as the directory. For example, I have both a directory called /analog/ and a page called /analog/ (which happens to be the same as /analog/index.html).
11. When someone reads one of my PDF files, it scores dozens of hits.
    PDF reading software often downloads a file in small parts, and each part counts as a separate request. This is unavoidable: analog has no way of knowing how many downloads constituted a single document in the reader's mind. The parts may be pages, or some larger amount, and sometimes the software may download the whole document at once. In addition, the user may only want to read part of the document, and so never download the whole thing. As usual, we can only reliably report how many requests there were at the server, not guess what the user did with the file later.
12. Kilobytes should be 1000 bytes, not 1024 bytes.
    Personally I think that whatever 1024 bytes should have been called originally, it's stupid to try and change half a century of established usage now. But we don't need to argue about it. Analog's kilobytes are 1024 bytes, but if you prefer to call them kibibytes, you can do so by editing your language file.
13. The Organisation Report doesn't identify organisations correctly.
    The rules I use are described in the section on The domains file. I admit they aren't perfect, but this is because in domains in which organisations aren't all at the same level in the domain hierarchy, there is no way to identify them perfectly without long lists.
14. "Organization" isn't spelled correctly.
    Yes it is. If you want American spellings, you have to specify

    LANGUAGE US-ENGLISH

    in your configuration file.

D. Advanced Usage

1. How can I do such-and-such with a command line option?
   Use the +C option to put any configuration command on the command line.
2. I want a list of all command line arguments.
   There is a list in the index.
3. How do I list all numerical subdomains to depth 2 in the Domain Report?
   SUBDOMAIN *.* deliberately only lists the top-level numerical subdomains to avoid cluttering the output. SUBDOMAIN *.*.* will work but will list everything else to depth 3. So the best solution is

   SUBDOMAIN 1*.*,2*.*,3*.*,...

4. I want to be able to count requests with status code 301 and 302 as successes, so that they appear in the Request Report.
   No, you really don't, because that would lead to double counting when a request for /dir (code 301) is redirected to /dir/ (code 200). For CGI scripts etc. look in the Redirection Report instead of the Request Report.
5. I want to report on a field analog doesn't know about.
   Use the following kludge. Write a LOGFORMAT to declare the field to be a virtual host or a user (whichever you aren't already using). Then edit your language file so that the right text is output.
6. Can analog analyse Squid proxy logfiles?
   It can analyse Squid's common log format, although Squid uses some extra HTTP status codes which will be rejected as corrupt by analog. But really you want to know different statistics from a proxy log, such as percentage of requests retrieved from cache, and you might be better to use Squid's native format and a tool specifically designed to analyse it such as Calamaris.
7. Can analog analyse FTP logfiles?
   or Can analog analyse streaming media logfiles?
   Yes. There are some configuration files in the examples directory giving sample LOGFORMAT commands. If you have a server which isn't covered there, you will have to write your own LOGFORMAT.
8. Can analog analyse other logfiles, such as mail logs, or the syslog?
   Yes and no. Analog is primarily designed as an HTTP logfile analyser. For mail logs, there is a program on the helper applications page to help you. For other logs, you can get some results out by writing your own LOGFORMAT. But analog does make some assumptions about the sort of information it expects on a logfile line, and the further these assumptions are from being met, the harder it will be!
9. How can I run analog automatically every day?
   This depends on your particular machine. On Unix, you need to run analog as a cron job (see "man cron"). This is my cron command to run it at 1:50am every day:
   50 1 * * * $HOME/bin/analog
   On Windows NT you can do the same with the at command. (It's probably easiest to put it in a batch job; also only an administrator can run at.) On Windows 98, it should be possible with the Task Scheduler, although I haven't tried it. On Windows 95 it's not possible as far as I know.
   On Mac, there are programs called Cron or CronoTask to do this.
10. How can I automatically email the results to myself or someone else?
    Again, this depends on your operating system. On Unix, it's easy:

    analog +a +O- | mailx -s"Subject" someuser@somewhere.com

    I don't know about other operating systems, but at the worst, you can write the output to a temporary file, and then mail that file.
11. I'm setting up IIS. Which logfile format should I use?
    The W3C format is probably best. You can turn fields on and off in this format. And it contains all the possible fields which can be logged, which the other formats do not. However, it is important to turn the date field on (it's off by default), not just to log the date once at the top: see the section on problems with logfile formats for why.
12. I host lots of virtual domains. How should I set up analog?
    There's a How-To which discusses this issue. There's also a file in the examples directory.
13. Can I make several output pages with just one run of analog?
    Not at the moment. I want to do this in a future version, but it will require some considerable work. However, depending on your which options you want to vary, you may be able to avoid having to read the logfile several times by using cache files. (This is likely to be faster, but more complicated.)
14. I ran out of memory when trying to run analog. What can I do?
    See the section on Coping with low memory.
15. You're processing 20,000,000 requests in under 10 minutes. Why is mine much slower?
    or Analog appears to stall.
    If you have DNS lookups on, they are very slow. Otherwise, it probably depends on the speed of your computer and disks, and what other programs are running at the same time. You can use the PROGRESSFREQ command to see if it's really stalled or whether it's just being slow. If you are running out of memory, you might find analog's LOWMEM commands helpful.
16. How do I make a link on my page that runs analog?
    Link to the anlgform program, with the desired options. But be careful about the load on your server.
17. Do I have to save all my old logfiles?
    or Can analog make statistics from old reports instead of reading the whole logfile again?
    These questions are answered in the section about Cache files.
18. Can analog write to a database or spreadsheet?
    Use the computer-readable output style, which can export to CSV. Or if what you really want to do is to run analog again without re-reading the logfiles, read the section about Cache files.

E. Form Interface

1. I couldn't make the form run.
   Have you made analog work without the form? Have you run anlgform.pl from the command line as explained in the section on troubleshooting?
2. How can I specify different logfiles from the form interface?
   Just add a new field to the form with name=LOGFILE
3. My browser showed me anlgform.pl, rather than running it.
   You have to tell the server to execute the CGI program, not just send it out like it would for a normal file. Often this is done by putting it in a special /cgi-bin/ directory.
4. Why does the form interface give "Document Returned no Data"?
   If it doesn't happen for a while, then probably the server is giving up before the analog process has finished running. Increase the timeout interval on the server.
5. The images don't appear when running analog from the form interface.
   For the bar charts, you probably need to set the IMAGEDIR, because if the images are in your /cgi-bin/ directory, the server will normally try to execute them instead of just sending them out. Pie charts don't appear unless you configure them specially.
6. Why do I get some reports that weren't requested on the form?
   If a report is neither included nor excluded on the form, the system default will be used. This will depend on your configuration files and on compile-time settings.
7. How do I make a link to anlgform.pl without using anlgform.html?
   anlgform.pl accepts the GET or POST methods of form submission. So you can make a link with the arguments passed after a question mark in the usual GET way.
8. Is there a form interface not using Perl (e.g. ASP or .exe)?
   There is a Windows executable version of the Perl script on the analog helpers page. At the time of writing, I don't know of any ASP version of the anlgform program, but if someone writes one, I'll put it on the analog helpers page too. Warning: Potential authors must understand CGI security issues in general, and the extra issues about what the analog form interface must disallow, or they will open security holes on their system.

F. Design Decisions

1. Why doesn't the HEADERFILE replace the whole <head> of the output file?
   Because you almost never get valid HTML or XHTML that way. Use a style sheet instead.
2. Why don't you just use one image, and scale it with the width and height attributes?
   It doesn't work with the BARSTYLE command.
3. In the XHTML output, why not just put a class= on each <COL> instead of on each cell?
   Because most properties can't be set on columns. For example, you can't set the text colour for a whole column. See the CSS spec.
4. Why doesn't the OUTFILE command automatically create the directory for the output file?
   Actually, most programs don't do this. It's too easy to create directories in unrelated parts of the filesystem if you make a typo.
5. Why not automatically spot robots by whether they request /robots.txt?
   It's not reliable. Not all robots request /robots.txt, and not everything that requests /robots.txt is a robot. (Consider a webmaster checking his own /robots.txt, for example.)
6. Why not just do DNS resolution of the hosts that actually make it into the Host Report?
   There is one theoretical and one practical problem. Theoretically, the problem is that which hosts do make it into the Host Report can change when the DNS lookups have been done. And practically, this wouldn't help identify the busiest countries or organisations, which is usually what you really want to know. However, there is a Perl script on the helper applications page to do this.
7. Couldn't you do the DNS lookups faster with threads?
   The problem is, the standard commands for DNS lookups are not thread-safe on many platforms, so it would involve a lot of platform-specific code. Also it's technically difficult to coordinate the lookup threads and the main logfile-reading thread. Again, there are programs for specific platforms on the helper applications page.
8. Why doesn't analog analyse the error_log?
   The error log is intended for humans rather than computers to read. So there is no consistent format: even different versions of the same server have different formats. And there is not much need to analyse it because analog's various failure reports are good enough for almost all purposes.
9. My server lists local names in the logfile. Can you put a common suffix on them automatically?
   This wouldn't be a good idea by default, because things like "unknown" would get the suffix. You can always add them using HOSTALIAS. (There is an example to accomplish this using regular expressions in the section about aliases.)
10. Can you extrapolate from the current month's partial data to produce a prediction for the whole month, based on the rate so far?
    No. There are too many problems in trying to produce anything sensible, especially near the beginning of the month. Different days of the week and different times of day cause lots of problems. I would prefer to produce accurate raw data than suspect derived data.
11. Can you extend the Domain Report to say which US states people visited from?
    No. Some programs pretend to do this, but you can actually only tell which state the computer the person was using is in, which may be quite different from where the user was for ISP's or other large organisations.
12. Please distinguish between the different BSDs in the Operating System Report.
    Sorry, I know they're different operating systems, but I don't want to introduce any finer granularity. At least, not in the main distribution -- the prepackaged versions for BSD do add this granularity, and this is good.
13. Why not use language codes instead of country codes for the names of the language files?
    People are more familiar with the country codes, and not all of my languages have language codes anyway. Anyway, the filenames are normally invisible to the user.
14. Why doesn't analog produce statistics on "visits"?
    See How the Web Works.
15. Why don't you sell analog?
    I didn't write analog for the money, and I'm happy just to see people use it. Also, by making it open source, lots of people send me ideas and code to include in future versions. How do you think I got all those languages? (Of course, if you want to send me money, or gifts in kind, or even just postcards...).

Mailing lists

If your statistics are provided by someone else, for example your web hosting company, you should normally get support from them. They're the only ones who can configure analog to your specifications.

There are three mailing lists for analog.

analog-announce

Announcements about analog. I post to this when there are new versions, for example. Only gets a few messages a year. You can join this list by filling in the form at http://lists.meer.net/mailman/listinfo/analog-announce

analog-help

Getting help with analog from experienced users. This is the place to go if you have trouble setting up or configuring the program. Usually you will get a swift reply. You have to subscribe to the list before you can send a message. This is necessary to deter spammers. You can subscribe at http://lists.meer.net/mailman/listinfo/analog-help

You can also read this list as a forum (but not post) or as a newsgroup.

analog-author

This just goes to me. Use for private comments, or other things that would not be suitable for the analog-help list. I'm pretty bad at answering my mail, so don't expect a swift reply. Don't use this address for user support questions: they will be ignored.

1. Read the FAQ. Maybe I've answered your question already. If I have, I'll just direct you to the FAQ, not answer it again.
2. If analog produced some error messages when it ran, they probably indicate what went wrong. Read the section on Errors and warnings.
3. If your question is "Will this command have that effect?", why not try it and see!
4. If you think you've found a bug, read the list of known bugs at my site, to see if your bug is already known about.
5. Read the other relevant pages of the Readme, particularly the sections on Starting to use analog and Customising analog. You may also find the index useful. We don't appreciate people who are too lazy to read the documentation. (If the documentation is unclear, or the relevant paragraph is too well hidden, then that's a different matter. Of course I want to know about that.)
6. Have a look in the web archives of the mailing list to see if your question has already been answered there.
7. If analog isn't doing what you thought you asked it to, then run it with the SETTINGS ON configuration command, and see what options it thinks it's meant to be using.

If you still need help, subscribe to the to the analog-help mailing list using the form at http://lists.meer.net/mailman/listinfo/analog-help, and then send your mail to analog-help@lists.meer.net. Please do the following when you send mail to the list.

1. Describe exactly what you did, what you expected, and what the computer did. Include the exact text of any error messages, not a précis.
2. Mention which version of analog you are using, on which operating system.
3. Although you should obey the first two points, you should also try and send a short message. Most people don't bother to read long messages.
4. Give your mail a subject line which indicates immediately what aspect of analog it is about. (This is useful for the archives).
5. Do not send long files or attachments unless you're asked to. We do not want to see your configuration file, your header file, your output file, or any logfile over 10 lines long. They are almost always useless to us. And anyway, excessively long messages will be rejected by the mailing list server.

• http://lists.meer.net/pipermail/analog-help/
• http://news.gmane.org/thread.php?group=gmane.comp.web.analog.general
• http://www.mail-archive.com/analog-help@lists.meer.net/

• http://lists.meer.net/pipermail/analog-announce/
• http://news.gmane.org/thread.php?group=gmane.comp.web.analog.announce

• news://news.gmane.org/gmane.comp.web.analog.general
• news://news.gmane.org/gmane.comp.web.analog.announce

Helper applications

These helper applications are all listed at the analog site. The list is constantly changing, so I'm not distributing it with the program. But I strongly recommend you go to the analog helper applications page and check it out.

There are also some example configuration files in the examples directory or folder distributed with the program.

Acknowledgements

Thanks are due to the author of getstats, Kevin Hughes. In the days before analog there were only three serious logfile analysis programs, and only one of them, getstats, had attractive output. I wrote analog when getstats stopped being able to cope with the size of our logfile, but my output was based on his, and still shows its parentage.

Thanks are also due to all those who helped in the early stages of writing this program, and gave me the encouragement to continue with analog and to release it publicly. Those who made helpful suggestions during the first few weeks of the program are numerous, but I must mention particularly Dan Anderson, Martyn Johnson, Joe Ramey, Chris Ritson, Quentin Stafford-Fraser and Dave Stanworth. Above all Gareth McCaughan gave me lots of programming advice. The early versions of the program would have run much more slowly without him.

My employer, ClickTracks, kindly let me continue to develop analog while I am working for them. More than that -- they even let me incorporate some code developed for ClickTracks into analog!

For six years, analog's home page was at my previous employer, the University of Cambridge Statistical Laboratory. Now it is provided by SourceForge. niccx.com donated the address analog.cx, and Takayuki Matsuki of Tokyo Kasei University and Peter Gradwell of gradwell dot com ltd provide DNS services.

meer.net very kindly provide the mailing lists for analog, as well as a mailbox for the analog-author mail. Thank you too to Aengus for administering the mailing lists, GMane for turning them into newsgroups, and all the services who provide list archives.

Thanks to Ian Jackson for providing official FTP and rsync sites for analog, and for hosting the old versions of the documentation. Many other people have provided mirror sites for analog, starting with Dave Stanworth (again!). The full list of mirror sites is listed elsewhere; thanks to all of them.

Mark Roedel first suggested porting analog to different platforms, and made the original DOS port. Shortly afterwards, Jason Linhart made the Mac port, and has continued to contribute lots of extra code for that platform and for the program in general. The Mac version also includes code contributed by Stephan Somogyi and Nigel Perry. Later ports were made by Dave Jones, Martin Zinser & Rick Dyson (OpenVMS), Magnus Hagander (Win32), Ivan Martinez (OS/2), Nick Smith & Stefan Bellon (RISC OS), Scott Tadman & Rob Judd (BeOS), Thomas Engel (NeXTSTEP), Martin Kraemer & Holger Schranz (BS2000/OSD, including EBCDIC support), and Hideyuki Yahagi (AS/400). Thank you also to the people who make precompiled versions available for various platforms.

The regular expression parsing is taken from Philip Hazel's PCRE library. The graphics use Thomas Boutell's gd library, the libpng library, and the zlib library by Jean-loup Gailly & Mark Adler. Logfile decompression uses the zlib library, Gilles Vollant's unzip library and Julian Seward's bzip2 library. Each of these libraries is subject to its own copyright and licensing conditions: PCRE licence, gd licence, libpng licence, zlib licence, unzip licence, bzip2 licence. If NEED_MEMMOVE is defined at compile time, then this product includes software developed by the University of California, Berkeley and its contributors (licence).

The XML output style was written by Per Jessen. The form interface is based on an idea by James Dean Palmer. The code to expand wildcards in directory names under Unix is by Owen Cliffe. Thanks to all the other people who have contributed bits of code too: I apologise for not having room to name all of them.

Thanks also to those who have written helper applications, for making analog more usable, and to those who have written How-To's, for sharing their experience of how to use analog in practical situations. And thanks to Alexander Toth for the Unix man page.

Many people have volunteered to translate analog into their own languages. Many thanks to the following who have contributed in this way over the years: Tigran Nazarian (Armenian), Jon Otegi (Basque), Emir Alikadic (Bosnian), Luchezar Georgiev (Bulgarian), Francesc Rocher, M. Mercè Llauge, Francesc Burrull i Mestres & Jordi Vidal (Catalan), Yang Meng & Che Dong (Simplified Chinese), Andrew Choi & Tzu-hsien Yu (Traditional Chinese), Tomo Sombolac (Croatian), Jan Simek & Karel Fajkus (Czech), Adrian Price (Danish), Ferry van het Groenewoud, Joost Baaij, Dimitry Smagghe, Bert Hiddink & Frank Fesevur (Dutch), Henrik Huhtinen, Steve Kelly, Andrew Staples, Mikko Silvonen, Juha Ojaniemi & Markus Peuhkuri (Finnish), Patrice Lafont, Lucien Vieira, Jean-Marc Coursimault, Lionel Delaude & Gordon Macpherson (French), Mario Ellebrecht, Martin Kraemer, Holger Schranz, Thomas Jacob, Thomas Frings, Georg Schwarz, Ralf Döring & Gustaf Mossakowski (German), Dimitris Xenakis (Greek), Laszlo Nemeth & Andras Kemeny (Hungarian), Gustaf Gustafsson & Valberg Larusson (Icelandic), Haris Hasanudin (Indonesian), Furio Ercolessi, Luca Andreucci, Alessio Bragadini & Marco Bernardini (Italian), Takayuki Matsuki, Stephen Obenski, Motonobu Takahashi, Kaori Chikenji & Kazuto Ishigaki (Japanese), Byungkwan Kim & InChang Oh (Korean), Jurijs Turjanskis & Anda Bimbere (Latvian), Ingrid (Lithuanian), Jan-Aage Bruvoll, Espen Bjarnø & Pål Løberg (Norwegian Bokmål), Magni Onsøien & Trond Øksendal (Norwegian Nynorsk), Wlodek Lapot, Tomek Wozniak & Marcin Sochacki (Polish), Ivan Martinez, Paulino Michelazzo & Ronan Lucio Pereira (Brazilian Portuguese), Jaime Carvalho e Silva (European Portuguese), Alex Mihaila (Romanian), San Sanych Timofeev, Boris Litvinenko, Vyacheslav Nikitich, Oleg Philon & Denis Zhukov (Russian), Mile Peric (Serbian), Stefan Billik & Dusan Zervan (Slovak), Andrej Zizmond & Dalibor Cvijetinoviè (Slovene), Javier Solis, Alexander Velasquez, Alfredo Sola, Martin Perez, Nelson Tactuk, Javier Kohan & J. L. García (Spanish), Björn Malmberg, Frank Osterberg, Wesley Schaal & Christian Rose (Swedish), Nezih Erkman & Dikran Diragormacioglu (Turkish), and Yaroslav Boychuk (Ukrainian).

Finally, thanks to all of you for using the program!

What's new in this version?

6.0 (19-Dec-04)

Added Palm OS and Symbian OS to the Operating System Report.

ISO 8601 extended date format available in language files.

Another style sheet from James Reeves added to the examples directory.

5.92beta1 (14-Nov-04)

Analog is now distributed under the GNU General Public License.

Fix problems building unzip.c on some platforms.

Fix invalid XHTML output in non-European languages.

The DNSTIMEOUT command is now off by default, because it breaks DNS lookups on many platforms.

Recognises Firebird and Firefox as browsers.

msnbot recognised as a robot in the default configuration.

Patches for Mac.

New Makefiles for Windows and RISC OS.

Corrections to Finnish and Japanese language files.

5.91beta1 (23-Aug-03)

Reads zip and bzip2 logfiles without the need for an UNCOMPRESS command.

Automatically strips ;params section from URLs (for example, jsessionid's).

Recognises Windows Server 2003 in the Operating System Report.

First user-contributed style sheets added to examples/css directory.

Computer-readable output style now forces English output.

More corrections to XML output style.

Patches for OpenVMS and RISC OS.

Basque language files.

Country code .cs, formerly Czechoslovakia, is now Serbia and Montenegro. (Changed in English, French and German domains files, and removed from other domains files).

Corrections to Swedish and Ukrainian language files, and to German domains files.

5.90beta2 (10-Apr-03)

Internal Search reports now work properly on case-insensitive file systems.

Various bug fixes and improvements to XHTML and XML output styles.

New command CSSPREFIX to add a prefix to the CSS class names used in the XHTML output.

XML DTD distributed with the program.

Language files for Simplified Chinese.

5.90beta1 (30-Mar-03)

This is the first beta test for version 6.

Output code completely rewritten, to permit more output formats.

New output formats XHTML and XML. (Thanks to Per Jessen for XML, and to Jeremy Wadsack for help with XHTML).

XHTML is now the default output style.

The barcharts are now made from png's not gif's by default.

New command LOGOURL.

What was new in version 5?

What was new in version 4?

What was new in version 3?

What was new in version 2?

What was new in version 1?

Upgrading from earlier versions

If you are upgrading from a previous version of analog, you should keep your old analog.cfg, and your old anlghead.h if you are compiling your own copy of analog rather than using one of the precompiled versions.

Upgrading from 5.91beta1 and earlier

• The DNSTIMEOUT command is now off by default. On platforms where it's available, you have to specify it explicitly if you want it.

Upgrading from 5.90beta2 and earlier

• The computer-readable output style now forces English language output. The idea is to have a consistent output style, and let the post-processor do any necessary localisation. But this may break some existing applications.

Upgrading from 5.32 and earlier

• The default output style is now XHTML, not HTML. Also the command-line argument "-a" selects XHTML, not HTML.
• If you are using a style sheet for HTML output, it probably won't work well with XHTML output, because it replaces the default style sheet. So you will probably need to write a new one for XHTML.
• The default for PNGIMAGES is now ON, so bar charts are made from png's not gif's by default. Some users may need to copy the png versions of the bar charts into their images directory.

Upgrading from 5.31 and earlier

• More browsers are recognised, which will also reduce the Mozilla numbers.

Upgrading from 5.24 and earlier

• In the Operating System Report, Macintosh is no longer split into PPC and 68k. (68k is now so rare that it's not worth it. Unfortunately, it's not possible to distinguish OS 9 from OS X, because the browsers don't all report it.)
• The language files have changed, so don't try and use the old ones.
• Some dead search engines have been removed from the default configuration file, and some new ones added.
• Kilobytes, megabytes etc. are now listed to two not three decimal places by default; but you can configure the number of decimal places with the new BYTESDP command.

Upgrading from 5.23 and earlier

• There will be small changes to the browser and operating system reports thanks to better parsing of recent Netscape and Mozilla browsers.
• In version 5.23 and earlier, earlier UNCOMPRESS commands took precedence over later ones. This has been corrected in 5.24.

Upgrading from 5.21 and earlier

• As a side-effect of the security fix in version 5.22, the JAPANESE-JIS character set no longer works from the form interface.

Upgrading from 5.1 and earlier

• The language files have all changed, so don't try and use the old ones.
• The rules for generating "organisations" from numerical addresses have changed, which will change the Organisation Report.
• Filenames given on the command line, as opposed to in configuration commands, are now relative to the current directory, not the analog directory. This will probably only affect Unix users.

Upgrading from 5.03 and earlier

• The code to generate the Browser Summary has been completely rewritten. Apart from now recognising Konqueror and Galeon, and distinguishing Mozilla and Netscape, this may result in other small changes to the reported numbers. Also, if you are using a configuration file from an earlier version, remove the line that reads

  BROWOUTPUTALIAS Mozilla Netscape

• Similarly, the code to generate the Operating System Report has been slightly modified, in addition to recognising Windows XP.
• Numerical addresses are now included in the Organisation Report, which will change that report.
• Some warnings have moved from category C to category F or to the new category G.
• SEARCHCHARCONVERT is now always turned off in multibyte character sets.
• The default plotting method for the pie charts will now take the FLOOR into account as well as the SORTBY.
• The LATEX output style now produces LaTeX suitable for converting to PDF with the pdflatex command. If you want to use the ordinary latex command, specify PDFLATEX OFF.

Upgrading from 5.02 and earlier

• The order of the lines in the report descriptions files has changed, so don't try to use the old versions.

Upgrading from 5.01 and earlier

• LANGUAGE JAPANESE now selects the JIS character set instead of EUC. (Four Japanese character sets can be chosen explicitly through the LANGUAGE command.)

Upgrading from 5.0 and earlier

• The language files have all changed, so don't try and use the old ones.
• Organisations in Belgium are now considered to be at depth 3, which will change the Organisation Report.
• There are some new images in the images/ directory, which you will have to copy into your IMAGEDIR.

Upgrading from 4.91beta1 and earlier

• LOCALCHARTDIR and DESCFILE can no longer be specified from the form interface, for security reasons.

Upgrading from 4.90beta4 and earlier

• Previously, when an output INCLUDE or EXCLUDE command was in operation, the excluded items were still included in the "not listed" line at the bottom of the report. This is no longer the case. Also, percentages of the requests and bytes are now calculated ignoring the excluded items, which will make those numbers larger, make their wedges in the pie charts larger, and change what is included by floors specified in terms of percentages.

Upgrading from 4.90beta3 and earlier

• The Operating System Report, Browser Report & Browser Summary are now sorted by page requests instead of raw requests by default.

Upgrading from 4.90beta1 and earlier

• There are some new files (with names beginning with sq) in the images/ directory. These will need to be copied into your IMAGEDIR.
• *.asp is no longer included in the default definition of "pages" (which it was from 4.11 to 4.90beta1) because there are too many other equally well qualified candidates. But you can easily re-enable it in analog.cfg.
• The format of the "busiest time" and floor/sortby lines has changed in the computer-readable output.
• The default DNSGOODHOURS and DNSBADHOURS have been increased. In particular, the default DNSGOODHOURS is now as near infinity as makes no difference!

Upgrading from 4.16 and earlier

• Some languages have still not been translated for version 5 yet: see the list of available languages in the What's new? section.
• All files are now looked for in sensible directories, specified at compile time, if no other directory is specified. On platforms where these directories are not known at compile time, analog formerly looked in the current working directory, but now looks in (its guess at) the directory of the analog binary. However, filenames specified in command line arguments are within the current working directory since version 5.2.
• The cache file now includes data on the first-request time of each item. But if you read a cache file from an older version of analog, this data will not have been recorded, and so the last-request time will be used instead. Analog will warn you about this.
• If you specify a CACHEFILE command but no LOGFILE command, analog won't read the default (compile-time) logfile. This is much more intuitive behaviour, but some users may have been relying on the old behaviour. The actual rule is given in the documentation on Cache files.
• Some browsers will be rediagnosed as robots in the Operating System Report. This will mainly reduce the "OS unknown" total, but may also reduce other categories.
• US English now uses the 12-hour clock by default. If you want to continue to use the 24-hour clock, use the language file us24.lng instead. (Either use a LANGFILE command, or rename us24.lng to us.lng). Conversely, you can get British English with the 12-hour clock by using uk12.lng.
• Because of the twelve new reports, if you use a REPORTORDER command, you should include the corresponding new letters: 1Q76wLlMRjyY.
• Computer readable output now has an extra line reporting the busiest time period for the time reports.
• The date codes in the OUTFILE and CACHEOUTFILE commands now always produce dates in English.
• The REFLINKINCLUDE command now only controls links in the Referrer Report. Use REDIRREFLINKINCLUDE and FAILREFLINKINCLUDE for the Redirected and Failed Referrer Reports.
• When doing a negative floor, items are no longer included if they have 0 of the criterion in question, even if there aren't enough items otherwise. For example FLOOR -25p will list fewer than 25 items if there aren't 25 items with requests for pages, even if there are other items with 0 requests for pages.
• When a logfile line contains bytes but no filename, analog previously ignored the bytes. It now counts them for other items on the line, but doesn't put them in the File Size Report or General Summary (to avoid double-counting).
• There are no longer any HTML language files containing HTML entities. So, for example, the HTML output will always contain a literal é instead of the code &eacute;. This should make no difference to the reader, but please do tell me about any problems.
• The source files have moved to the src/ directory. And there are new source files in subdirectories of the src/ directory. Also the header files have different variables in them. This means that automatic build scripts will have to be rewritten.

Upgrading from 4.13 and earlier

• Regular expressions in an INCLUDE or EXCLUDE command must now occur on a line on their own, not within a comma-separated list.
• The search terms reported in the Search Word and Search Query Reports are no longer converted to lower case if you are using a multibyte character set.
• Unprintable characters in the output are now replaced by '?', except for multibyte character sets.

Upgrading from 4.11 and earlier

• *INCLUDE "" and *EXCLUDE "" (see documentation) now apply to items which were present but corrupt. This may have the effect of including or excluding some new lines.
• There has been a tiny change in computer-readable output style. Previously if a time was blank, it took up only one column. Now it takes up as many columns as if it had been present.

Upgrading from 4.1 and earlier

• The default definition of "pages" is now case insensitive: so it includes *.HTM and even *.hTmL etc. This will serve to increase the page count.

Upgrading from 4.04 and earlier

• There is better parsing of extended format and WebSTAR format logs, which may cause differences in some cases.
• All referrers now count as "pages" irrespective of any PAGEINCLUDE and PAGEEXCLUDE commands. Consequently, for example, "REFLINKINCLUDE pages" is now the same as "REFLINKINCLUDE *". You can recreate the previous behaviour with "REFLINKINCLUDE *.html,*.htm,*/".
• The default REPORTORDER has changed.
• New anchor names are used internally to the output page.

Upgrading from 4.03 and earlier

• There may be slight differences in the results in this version owing to stripping anchors off filenames.

Upgrading from 4.0 and earlier

• Some of the default paths have changed in anlghead.h.

Upgrading from 3.90beta1 and earlier

• It is now recommended that you don't run analog as a CGI program, or put it in the directory with your CGI programs, for security reasons.
• Each browser in the Browser Summary is now sorted by major version number then minor version number. So SUBBROW */* will now only show the major versions. To get all the minor versions, you need SUBBROW */*.*
• PAGEWIDTH has been replaced by HTMLPAGEWIDTH and ASCIIPAGEWIDTH.
• PRINTVARS has been renamed SETTINGS.

Upgrading from 3.32 and earlier

• The form interface has been completely rewritten, and old versions of anlgform.html will not work with this version.
• The Browser Summary now diagnoses MSIE, Opera and WebTV browsers better. This will cause differences in output from previous versions.
• With RAWBYTES OFF, bytes are now listed as, for example, 47.68 Mbytes instead of 48,832 kbytes. This should be less confusing.
• The DNS file has a new time encoding. It's only a few hours different, so I haven't made any special provision for it. The effect is that the DNSGOODHOURS and DNSBADHOURS may be a few hours out for existing entries (but not for new ones).

Upgrading from 3.3 and earlier

• There is a new set of graphics in the images directory, which you will have to move to your web directory.
• In the Mac version, if a configuration file is dragged onto the analog icon, it is used instead of, not as well as, the default configuration file.

Upgrading from 3.2 and earlier

• In the computer-readable output style, the line L7, the time the last seven days begins after, has been replaced by E7, the time the last seven days ends. This is for consistency with the other output styles.
• Also in the computer-readable output, there is a new line reporting the floor and the SORTBY for the report. In 3.11 and earlier, this didn't exist, and in 3.2 it only reported the floor, not the SORTBY.
• %R (Mac-style filename) has been abolished in the LOGFORMAT. Just use plain %r instead.
• It is no longer allowed to set the CACHEOUTFILE to be the same as a previous cache file.
• The definition of the common log format and related formats changed between 3.11 & 3.2, and again between 3.2 & 3.3. This could cause differences in output, but they are likely to be only very minor.

Upgrading from 3.11 and earlier

• Lines without a particular item now work properly with INCLUDE and EXCLUDE commands. For example, if you do an INCLUDE to look at only certain lines, then lines without that type of item at all will not now be included, whereas previously they would have been. This can make the results lower than in these earlier versions.
• I have reluctantly removed support for NetPresenz logs. This hasn't worked for some time, and I have already been advising NetPresenz users not to use newer versions of analog because they could get wrong results. Unfortunately, fixing it would require a complete rewrite of the entire parsing code, which isn't going to happen any time soon. So my advice remains the same: continue to use version 2.11 or (even better) pre-process your logfiles into a form which analog can handle safely.
• The English domains file has changed name from domains.tab to ukdom.tab.

Upgrading from 3.0, Win32 form interface

• If using the form interface on Windows, it is now necessary to put the analog executable at \analog\analog.exe instead of \Program Files\analog\analog.exe

Upgrading from 2.90beta1

• LOGFORMAT MICROSOFT has been replaced by LOGFORMAT MICROSOFT-NA and LOGFORMAT MICROSOFT-INT; and similarly for LOGFORMAT NETPRESENZ.

Upgrading from 2.11 and earlier

• It is possible that there may be small discrepancies between the results from previous versions and the results from this version because the parsing code has changed, but any such differences should be minor. However...
• If you used to use REFEXCLUDE or BROWEXCLUDE, you most likely now want REFREPEXCLUDE or BROWREPEXCLUDE instead, or you will exclude lots of lines that were previously included.
• It is possible that this version may not automatically parse a logfile that previous versions could parse, because it checks more carefully that the logfile is in the format claimed. If so, you will have to use a LOGFORMAT command.
• Approximate host counting has been abolished, unless there's a significant demand for it.
• Count of number of new hosts in last seven days abolished. It was too confusing because it depended on which old logfiles you analysed.
• The Error Report has been abolished (together with the configuration commands ERROR, ERRLOG and ERRMINOCCS). See the What's new? page.
• The BROWLOG and REFLOG commands have also been abolished: just use LOGFILE instead.
• The HASHSIZE commands have been abolished: analog now chooses the size of the hash tables itself.
• The MINREQS and similar options have been replaced by the FLOOR commands.
• Only one * is now allowed on the left-hand side of aliases, to avoid ambiguities.
• Automatic detection of log type is now on a per-file rather than a per-line basis.
• ISPAGE is now called PAGEINCLUDE.
• WITHARGS and REFWITHARGS are now called ARGSINCLUDE and REFARGSINCLUDE.
• MONTHLYBACK is now called MONTHBACK.
• FULLHOSTS is now just called HOST.
• The old LOGOURL command is now called LOGO. LOGOURL now has a new function, namely where to link the logo to.
• The UNIT commands have been abolished. They weren't very useful, and they didn't make much sense with the different ways of displaying the time report bar charts. The unit is now always chosen automatically.
• DIRLEVEL has been abolished, because the SUBDIR command is more general. Use SUBDIR */* or whatever instead.
• Comments aren't allowed in the domains file. I don't think this should cause a problem.
• GRAPHICAL is abolished. Instead, use lower case letters with the GRAPH commands.
• NUMLOOKUP has been replaced by DNS, and DNSFRESHHOURS by the commands DNSGOODHOURS and DNSBADHOURS.
• DNS cache files from previous versions are not compatible with this version.
• You can't use PAGES in the columns or SORTBY or FLOOR for the Request Report. Use REQINCLUDE pages instead.
• - as a synonym for none has been abolished in some places, e.g., HOSTURL.
• The following command line arguments have been abolished from earlier versions, many of the letters getting new meanings: 7, l, n, p, s, u, v, w. (-v has moved to -settings.) Others have been changed since version 1.2 as well.

Upgrading from 2.0, Win32 users

• Filenames for logfiles etc. should now be given DOS-style, with backslashes, rather than Unix-style with forward slashes.

Upgrading from 1.92 and earlier, Mac users

• There is no longer an automatic progress report. Use the PROGRESSFREQ command instead.

Upgrading from 1.9beta's

• Use REQINCLUDE and LINKINCLUDE instead of REQTYPE and PAGELINKS.

Upgrading from 1.2's and earlier

• Use *INCLUDE and *EXCLUDE instead of *ONLY and *IGNORE.
• The syntax of the *FLOOR commands has changed.
• Use *SORTBY REQUESTS or BYTES instead of *SORTBY BYREQUESTS or BYBYTES.

What was new in version 5?

What's new in version 6?

5.32 (23-Mar-03)

Recognises Safari and Camino browsers.

New LOGFORMAT specification %D (processing time in microseconds).

Understands %A and %D in APACHELOGFORMAT.

5.31 (11-Jan-03)

Recognises Phoenix and Chimera browsers.

Fixes for Mac, OpenVMS and RISC OS ports.

Various other small bug fixes.

5.30 (30-Nov-02)

Reads gzipped logfiles without the need for an UNCOMPRESS command.

Host inclusions and exclusions can now use IP address ranges and subnet masks.

New command BYTESDP command to list kilobytes etc. to a chosen number of decimal places. The default number of decimal places has changed from three to two.

New code for RISC OS port, including wildcards in filenames. (Thanks to Stefan Bellon for this code).

Various bug fixes.

Indonesian and Slovak language files.

New How-To entitled Getting started under Windows. (Thanks to Simon Handfield).

5.24 (25-Jun-02)

Recognises Netscape 7 browser. Also better diagnosis of Windows operating systems for Netscape and Mozilla browsers.

New version of IIS How-To.

Later UNCOMPRESS commands now correctly override earlier ones.

SEARCHCHARCONVERT can now be turned on for multibyte character sets, though it's not recommended.

French report descriptions files.

New versions of Polish and alternative Swedish language files.

East Timor country code is now .tl.

5.23 (14-May-02)

Added PROGRESSFREQ to the list of forbidden commands from the form interface, because it allows a denial-of-service attack.

Also emphasised in the documentation that allowing untrusted users to run analog through the form interface is inherently open to a denial-of-service attack.

Fixed two bugs which caused occasional crashes while outputting.

Russian language files included (KOI8-R and Windows-1251 character sets).

Added new domains to the French domains files.

New How-To on configuring IIS.

5.22 (20-Mar-02)

Security fix for cross-site scripting security bug.

Upgraded zlib code to version 1.1.4 (although analog wasn't vulnerable to the security bug in zlib 1.1.3).

The PROGRESSFREQ messages now go to the screen as well as to the ERRFILE.

The second argument to SUBDOMAIN can now contain *'s and $'s.

Added eight new domains to many of the domains files.

Revised Japanese language files.

5.21 (20-Feb-02)

Corrected the MacHTTP log format, which didn't work in 5.2.

All the BARSTYLEs redrawn, and two new BARSTYLEs added, adapted from an idea by Dave Holle. (You will have to move the new graphics into your IMAGEDIR in order to use them.)

5.2 (13-Feb-02)

You can now plot the lower levels of hierarchical reports on the pie charts by using the new CHARTEXPAND family of commands.

Added MACHTTP to the list of built-in log formats that analog recognises automatically.

Recognises ; as well as & as query-string separator.

The rules for generating "organisations" from numerical addresses have changed.

Filenames given on the command line are now relative to the current directory, not the analog directory.

Ignores completely blank lines at the top of a logfile.

Makefiles for Microsoft Visual C++ can be found in the new src/build directory. Makefiles for other platforms have moved out of the source tree into there too.

You can now refer to kilobytes as kibibytes by editing your language file.

Revised versions of Japanese language files.

Revised the Licence.

Advertised new donations page.

5.1 (07-Nov-01)

The Browser Summary now recognises the Konqueror and Galeon browsers, and distinguishes Mozilla and Netscape. (If you are using a configuration file from an older version of analog, you need to remove one line.)

The Operating System Report now recognises Windows XP.

Numerical addresses are now included in the Organisation Report.

Logfile and cache file names can now contain date codes, in the same way as the OUTFILE.

The person setting up the form interface can now specify a list of allowed commands, rather than a list of forbidden commands.

The LATEX output style now works with the ISO-Latin-2 character set as well as ISO-Latin-1. New command PDFLATEX.

New LOGFORMAT token %Z for two- or four-digit years.

In the COMPSEP, you can now use \t to represent a tab.

New warnings category G; some warnings reclassified.

Various small bugs fixed. Also there is some rewriting internally, but it should be transparent to the user.

The Windows executable has been compiled using MinGW instead of Microsoft Visual C++.

Finnish translation, and corrections to German and Serbian. Brazilian Portuguese report descriptions files.

5.03 (07-Jul-01)

Bug fixes, including one bug that caused a crash under Windows.

Spanish translation, and corrections to SWEDISH-ALT translation.

5.02 (23-Jun-01)

Understands the new IIS log format with four-digit years.

Traditional Chinese and Brazilian Portuguese language files.

Japanese is now available in four character sets.

Four language files which were already distributed with the program are now available through the LANGUAGE command: BULGARIAN-MIK (MIK-16 charset), CZECH-1250 and SLOVENE-1250 (Windows-1250), and SWEDISH-ALT (alternative translation avoiding Anglicisms).

5.01 (19-May-01)

New command PNGIMAGES.

New token "PLAIN:" in ALIAS commands.

Unix man page included in source distribution.

New example configuration file examples/bigbyrep.cfg.

A few small bug fixes.

Italian report descriptions files; corrections to German language files.

5.0 (01-May-01)

How-To's introduced.

Wildcards now allowed in LOGFILE names from the form interface.

UNCOMPRESS now works on filenames with spaces in, and other small bug fixes.

Makefiles and build scripts for OpenVMS.

This version is available in 24 languages: Armenian, Bulgarian, Catalan, Czech, Danish, Dutch, English, US English, French, German, Hungarian, Italian, Japanese, Korean, Latvian, Norwegian (Bokmål & Nynorsk), Polish, Portuguese, Serbian, Slovene, Swedish, Turkish and Ukrainian.

4.91beta1 (10-Apr-01)

Output EXCLUDEs are now more fully excluded, producing significant changes in results.

New JPEGCHARTS command (only available if you compile analog yourself, with your own libgd).

New mailing list instructions to include subscriptions over the web, and the digest version of the analog-help list.

Mac version decompresses pkzip and bzip2 logfiles.

Ports to Mac OS X (Carbon) and IBM OS/390.

Pie charts work with EBCDIC character sets; and other improved EBCDIC support.

Wildcards in LOGFILE directories now work under more Unices.

The one-page version of the documentation, whole.html, is no longer distributed with the program. (It's still available at my site.)

This version is available in 21 languages: Armenian, Bulgarian, Catalan, Dutch, English, US English, French, German, Italian, Japanese, Korean, Latvian, Norwegian (Bokmål & Nynorsk), Polish, Portuguese, Serbian, Slovene, Swedish, Turkish and Ukrainian.

4.90beta4 (26-Mar-01)

Wildcards in directory names in LOGFILE commands now work under Unix. (Thanks to Owen Cliffe for this code.)

The CHARTDIR and LOCALCHARTDIR can now contain date codes, in the same way as the OUTFILE.

My own configuration file included in the examples directory.

This version is available in sixteen languages: Armenian, Bulgarian, Catalan, English, US English, French, German, Italian, Japanese, Korean, Latvian, Polish, Portuguese, Slovene, Swedish and Ukrainian.

4.90beta3 (13-Feb-01)

Security fix for buffer overflow bug.

4.90beta2 (05-Feb-01)

New commands CHARTDIR and LOCALCHARTDIR.

The pie charts are now easier to read, and work properly in non-European character sets.

*.asp is no longer included in the default definition of "pages". (See how to upgrade).

The computer-readable output style now gives the REPORTSPANs.

On Unix, now follows symlinks when finding the name of the analog binary to construct other filenames.

Various other minor bug fixes from 4.90beta1.

This version is available in nine languages: Armenian, Bulgarian, Catalan, English, US English, French, Portuguese, Swedish and Ukrainian.

4.90beta1 (22-Jan-01)

This is the first beta test for version 5.

• Twelve new reports: Yearly Report, Quarterly Report, Five-Minute Summary, Quarter-Hour Summary, Hour of the Week Summary, Host Failure Report, Host Redirection Report, Virtual Host Failure Report, Virtual Host Redirection Report, User Redirection Report, Internal Search Word Report and Internal Search Query Report.
• Pie charts are now included at the top of most reports. *CHART commands to control them.
• New command GENSUMLINES to control which lines are listed in the General Summary.
• The "Distinct hosts" line now appears in the General Summary even if you aren't making a Host, Organisation or Domain Report.
• New commands DESCRIPTIONS and DESCFILE to provide descriptions of each report.
• New commands REPORTSPAN and REPORTSPANTHRESHOLD to list the time period covered by each report.
• New commands ROBOTINCLUDE and ROBOTEXCLUDE for listing robots in the Operating System Report.
• New command INTSEARCHENGINE to allow the internal search reports.
• New command DNSTIMEOUT (on some platforms) to reduce the time taken for failed DNS lookups.
• New LINKINCLUDE and LINKEXCLUDE commands. (See how to upgrade).
• New columns E and e for time of first request; and S, s, Q, q, C and c for data in last 7 days; also corresponding SORTBY and FLOOR commands.
• Substantial internal changes to allow these new columns without using extra memory if they are not wanted, and to substantially reduce code size (from what it would have been otherwise!).
• LaTeX output style.
• All files are now looked for in the "right" directories. This should improve usability substantially, especially from the form interface. (See how to upgrade).
• If you specify a CACHEFILE command but no LOGFILE command, analog won't read the default logfile. (See how to upgrade).
• US English now uses the 12-hour clock by default. (See how to upgrade).
• Computer readable output now reports the busiest time period for the time reports.
• Several commands have changed to better names, but the old names should still work.

In addition, version 4.14 was also released on the same date, and this version includes all its changes.

This version is only available in English and US English.

What was new in version 4?

What was new in version 3?

What was new in version 2?

What was new in version 1?

What was new in version 4?

What's new in version 6?

What was new in version 5?

4.16 (13-Feb-01)

Security fix for buffer overflow bug.
Small correction to Brazilian Portuguese language file.

4.15 (01-Feb-01)

Bug fixes for accented letters in dates and for EBCDIC character set.

4.14 (22-Jan-01)

Commas are allowed in regular expressions.
Can do a FLOOR beyond Terabytes.
OS X & MPE/iX ports.
Bug fixes, especially for multibyte character sets.
Unprintable characters in the report are now replaced by '?'.
Traditional Chinese, Portuguese, Brazilian Portuguese, US English and corrected French domains files.
Rewrote the documentation on Cache files. Added some new data to How the Web Works.

4.13 (11-Oct-00)

Corrected infelicity in compilation procedure.

4.12 (05-Oct-00)

Recognises Windows Me for Operating System Report.
Can count beyond Terabytes.
PCRE code upgraded to version 3.4.
AS/400 port. Patches to compile cleanly on Cygwin and 64-bit Solaris.
Bulgarian and Croatian language files. Catalan and Finnish domains files.
Various bug fixes.
New Licence (mostly less restrictive than the previous one).

4.11 (31-May-00)

The default definition of "pages" is now case insensitive, and also includes *.asp.
Reads the extended logs from IIS 5 correctly.
Version number displayed before any warning or debugging messages.
The "number of days" at the top of the report now obeys DECPOINT (and is also now to 2 decimal places).
Improved OpenVMS build procedure.
Hungarian and Romanian language files, and corrected Spanish language files and English domains files.
Italian and Spanish form interfaces.
The FAQ now has a list of contents.

4.1 (30-Mar-00)

Regular expressions in ALIASes and INCLUDEs are now available on all platforms.
Regular expressions are now Perl-syntax regular expressions. (Thanks to Philip Hazel's PCRE library.)
"Repeated fields" in logfile header lines are now allowed.
New commands STATUSINCLUDE and STATUSEXCLUDE, and 304ISSUCCESS.
New output style PLAIN (like ASCII but with accents). New language files for this.
In the computer-readable output, hierarchical reports now have an extra column, indicating the depth of the item in the hierarchy.
All referrers now count as "pages." (See upgrade notes.)
Configuration commands can be continued across lines with a backslash.
New token %s in LOGFORMAT, allowing per-line selection of client-name and client-IP fields.
New log format WEBSTAR-EXTENDED to allow for a small bug in WebSTAR's implementation of the extended log format.
Korean language files. Also alternative Swedish translation.

4.04 (21-Mar-00)

The analog home page has moved to www.analog.cx
New column d in non-time reports.
The RUNTIME command now turns off the "Program started at" line as well as the "Running Time" line.
Non-alphanumeric characters are now allowed in the REPORTORDER as separators.
Correctly parses more APACHELOGFORMATs.
Better detection of Windows 2000 in Operating System Report.
Better warning messages when the erroneous command contains a space.
Code for NeXTSTEP operating system.
Better treatment of multibyte character sets.
Icelandic language files. Corrections to Bosnian, French, Italian, Japanese & Swedish.

4.03 (21-Feb-00)

Fixed several small bugs.
New command RUNTIME.
Brazilian Portuguese language files and Swedish domains files. Corrections to Dutch.

4.02 (31-Jan-00)

New command SEARCHCHARCONVERT.
Support for Apache's new %q code in APACHELOGFORMAT.
Fix for search reports causing crashes on Windows.
New language: Czech. Corrections for Serbian, Slovene and Ukrainian.

4.01 (17-Dec-99)

New command USERCASE.
Some of the default paths have changed in anlghead.h.
Improvements to OpenVMS port.
Language files included for Armenian, Bosnian, Catalan, traditional Chinese, Dutch, Finnish, German, Italian, Slovak, Slovene, Spanish, Swedish & Ukrainian; corrections to Russian & Turkish.

4.0 (16-Nov-99)

Simplified Chinese, Danish, Japanese, Portuguese & Serbian language files included.
Otherwise only small changes since 3.90beta2.

3.90beta2 (02-Nov-99)

It is now recommended that you don't run analog as a CGI program for security reasons. (The CGI command is still present, but it is now not documented.)
The Organisation Report is now hierarchical.
The Browser Summary is now arranged by major version number. (See notes on upgrading.)
Non-exact bytes are now given to 3 decimal places.
GOTOS FEW puts the "Go To" lines just at the top and bottom of the output.
PRINTVARS has been renamed SETTINGS.
-settings output improved, especially with OUTPUT NONE.
Split PAGEWIDTH into HTMLPAGEWIDTH and ASCIIPAGEWIDTH.
Includes language files for French, Greek, Norwegian (Bokmål & Nynorsk), Polish, Russian and Turkish.
New configuration file examples/big.cfg containing most commands.

3.90beta1 (07-Oct-99)

First beta test for version 4. The most important new features are:

• Five new reports: Organisation Report, Operating System Report, Search Word Report, Search Query Report, Processing Time Report.
• Browser Summary improved (will change results).
• Form interface completely rewritten, and considerably simplified.
• Multiple *'s now allowed on left-hand side of ALIASes.
• Regular expressions allowed in INCLUDEs & EXCLUDEs, and ALIASes.
• The output INCLUDEs and EXCLUDEs now apply to the lower levels of a hierarchical report as well as the top level.
• New commands: CGI, STYLESHEET and ERRLINELENGTH.
• New column N in most reports.
• DEBUG C now reports which part of a corrupt logfile line is corrupt.
• Non-exact bytes are now displayed as, e.g., 47.68 Mbytes instead of 48,832 kbytes. This should be less confusing.
• Timestamps added to PROGRESSFREQ reports.
• The DNS file has a new time encoding.
• Header files split up to make anlghead.h simpler.
• Form interfaces in German and U.S. English included.
• New documentation about search arguments.
• New examples directory.
• New licence. (Nearly the same, just clarified, and slightly loosened).

What was new in version 3?

What was new in version 2?

What was new in version 1?

What was new in version 3?

What's new in version 6?

What was new in version 5?

What was new in version 4?

3.32 (02-Sep-99)

Bug fixes, including:

• Drag-and-drop on Mac now works.
• Unsafe characters in hyperlinks now escaped.
• One bug that caused crashes when printing deep Directory Reports fixed.

New VMS build scripts. Let me know of any compilation problems.
Computer-readable output now reports version of analog used.
Improved some diagnostic messages.
New language Serbo-Croatian; new domains files for Italian and Russian; corrected Polish language files.
New documentation on Analog's reports and Quick reference.
Now uses named anchors throughout the documentation, so that cross-references link to the right part of a page.

3.31 (19-Jun-99)

New command BARSTYLE; you will need to use new images.
Russian language file corrected.
Some bug fixes, including one important one correcting cache file output.

3.3 (19-May-99)

New commands ERRFILE, DNSLOCKFILE, APACHELOGFORMAT and APACHEDEFAULTLOGFORMAT.
Can include the date in the name of the OUTFILE and the CACHEOUTFILE.
Support for WebSite logfiles.
New token %U in log formats for "Unix time" (seconds since 1970).
Won't overwrite old cache files.
Now works properly on SunOS 4.
Fix for occasional crashes on Windows.
Checks language files are not too long.
"Last seven days" data now calculated more accurately and displayed more clearly.
Computer-readable output now reports SORTBY's as well as floors.
Revised Makefile will work with older make's.
Corrected Catalan language files.
Includes form interfaces in French and Japanese.
LOGFORMAT documentation now includes the LOGFORMAT commands for all built-in log formats.

3.2 (04-May-99)

Bug fixes: in particular REFLINKINCLUDE pages now works; and cache files now include all items even if they're not wanted for the main report.
Lines without a particular item now work properly with INCLUDE and EXCLUDE commands. This can cause differences in results from previous versions.
New version of form interface to work round bug in Microsoft Internet Information Server.
New command NOROBOTS.
Backslashes are now coerced to forward slashes in filenames and usernames. While not always correct technically, it usually is in practice, and it makes them behave correctly in other parts of the program.
Usernames are now treated as case insensitive. Let me know if this causes a problem on any system.
Computer-readable output style now reports floors.
Rewritten Unix Makefile, and VMS build script. Let me know of any compilation problems.
New languages: Catalan, Icelandic, Japanese, Korean, Latvian, Lithuanian. Corrected Spanish language files and French domains file.
LANGUAGE now selects local domains file automatically, where available.
Removed support for NetPresenz logs. The reasons are in the section on how to upgrade.
Form interface documentation rewritten; FAQ broken into sections; sections on logfiles and log formats separated and rewritten; new section on helper applications; and dozens of other improvements to the documentation.

3.11 (26-Nov-98)

Bug fix version.
Microsoft's attempt at W3 extended format is now understood even if there is a second #Fields: line in the logfile.
There is also a fix for a new Microsoft bug which results in an non-standard common format.
Intermittent crashes under Windows fixed.
Mailing lists announced.

3.1 (17-Oct-98)

Understands Microsoft's attempt at W3 extended format.
Several bugs fixed, including one that caused occasional crashes and one that caused the output to grow and grow.
Form interface works on Windows.
Allows aliases with two or more *'s on left hand side, if right hand side contains no *'s.
Aliases work properly with CASE INSENSITIVE.
Numerical SUBDOMAINs fixed.
Understands more WebSTAR and Netscape tokens.
Accents in domains file work.
LOGFORMAT removed from form interface as security risk.
Several warning messages improved.
Report aliases and in/exclusions shown in settings output.
Character set declared at top of output.
Spanish, Dutch, Norwegian (Bokmål and Nynorsk), Finnish, Turkish, Greek, Polish, Russian & Chinese language files included.

3.0 (15-Jun-98)

Corrected W3 extended format.
Fix for broken strcmp() function on SunOS 5.
Portuguese, Brazilian Portuguese, Danish and Hungarian language files included.
Precompiled executable for OS/2 available.

2.91beta1 (04-Jun-98)

Form interface included.
Uses less memory when compiling reports.
New operating system, BS2000/OSD, and code for EBCDIC character set.
New command DEFAULTLOGFORMAT.
LASTSEVEN and BASEURL reinstated.
More information added to PRINTVARS output.
AppleScript support for Unix-style command lines added to Mac version.
Now works on SunOS 4, and other small bug fixes.
French, German, Swedish, Czech, Slovak, Slovene and Romanian language files included.
One page version of the Readme included in the documentation.

2.90beta4 (09-Apr-98)

Mended DNS cache file reading, which I broke in yesterday's release.

2.90beta3 (08-Apr-98)

Fixed bug that caused a crash while giving warning messages on SunOS; bug that caused configuration files that called other configuration files not to be completed; and other smaller bugs.
Italian language files included.

2.90beta2 (03-Apr-98)

Separate LOGFORMATs for North American and international date formats, when using Microsoft or Netpresenz logs.
Understands the AppleShare IP server's attempt at the WebSTAR format.
Directory report now works properly even if you use the second argument to the LOGFILE command.
Wild cards in filenames work properly on the Mac.
Other small bug fixes.
One speed improvement (I gain about 3%).
Several corrections and clarifications to the documentation.

2.90beta1 (27-Mar-98)

This version is a completely rewritten version. Every single line of code is new. The whole code is shorter despite considerable improvements in functionality. Several people have reported that it is significantly faster. The most important new features are:

• Eleven new reports (Quarter-Hour, Five-Minute, Redirection, Failure, File Size, Referring Site, Redirected Referrer, Failed Referrer, Virtual Host, User, User Failure).
• Reads logfiles in user-customisable format.
• Analyses user and virtual host data, and failed requests.
• Hierarchical reports list subdirectories under directories, and allow analysis of browser version numbers.
• Faster sorting of long reports.
• Floor and sort method made independent.
• "Last date" column in reports, and can floor and sort by date.
• Busiest time period at bottom of time reports.
• "Not listed" line at bottom of other report.
• Knows HTTP/1.1 status codes.
• General Summary can go anywhere in the report.
• General Summary and "Go To"s can now be turned on and off independently.
• Status Code Report can be sorted in different ways.
• Time offset commands.
• Much better checking of invalid configuration options and invalid logfile lines.
• Only reads logfiles it might need.
• Improvements in DNS functionality: can now read the DNS file without further lookups: also, separate recheck intervals for successful and failed lookups.
• Hash sizes now chosen automatically.
• More flexible language support.
• Mac version reads gzipped logfiles.
• Mac version supports drag-and-drop onto program icon.
• Readme files completely re-written. Broken into lots of files, and new sections on Starting to use analog and What the results mean, as well as an index.

The following features have been abolished.

• No Error Report. The error log was always intended for humans rather than computers to read. Moreover, its format varied from server to server, and even between different versions of the same server. The place of the Error Report has largely been taken by the new reports, particularly the Failure Report.
• The approximate host counting has been abolished for the time being. I can put it back if there is a significant demand for it.
• Only one * can now appear on the left-hand side of aliases. This is to avoid ambiguities.
• For changes in the names and syntax of configuration options and command line arguments, see the section about upgrading.

The following features are not yet present, but will be added by version 3.

• The form interface.
• Most of the languages.

What was new in version 2?

What was new in version 1?

What was new in version 2?

What's new in version 6?

What was new in version 5?

What was new in version 4?

What was new in version 3?

2.11 (14-Mar-97)

Minor bug fixes to yesterday's release.

2.1 (13-Mar-97)

Language support rewritten, causing reduction in code size of 2200 lines.
New configuration command LANGFILE.
New Acorn RiscOS version.
Page requests per day reported.
Bug fix: CASE INSENSITIVE could cause %7E-type conversions not to take place.

2.0.2 (04-Mar-97)

DNS lookups and wildcards should now work in the Win32 version.
New configuration command PRINTVARS.
Fix for zero length hostnames after DNS lookups.
Minor corrections in French and Spanish translations.

2.0 (10-Feb-97)

New native Win32 version.
Wildcards allowed in filenames on Mac.
Ignores browser "-".

1.93beta (18-Jan-97)

New commands BROWALIAS, CONFIGFILE and PROGRESSFREQ.
Form program can now call configuration files.
Form program now uses the default choices if none specified.
Domain report prints correctly in preformatted output.
Specifying +1 and +V2 doesn't crash the program.
-v reports dates correctly.
Trailing dots on hostnames removed.
Second argument to LOGFILE command can't be obliterated by /../

1.92beta (08-Oct-96)

DNS lookups added on Mac.
Netpresenz format understood on Mac.
New languages: Spanish, Italian and Danish.
Extra information when debugging turned on.
*.htm are now pages on all machines.
A few small bugs fixed.

1.91beta4 (13-Jul-96)

Cache file now includes page request information.
DNS bug fixed.
New command DNSHASHSIZE.
Bug in browser reports fixed.

1.91beta3 (09-Jul-96)

BSD/OS compilation bug believed fixed.
Fixed HOSTALIAS which I broke yesterday.
DNS bug (causing too many lookups) identified, although not yet fixed.

1.91beta2 (08-Jul-96)

Some bug fixes (including: HOSTEXCLUDE and CASE INSENSITIVE didn't work properly; selecting "no links" failed on the form; less fussy about what can appear on the form).
Mac version no longer includes source code, so is much shorter.

1.91beta1 (05-Jul-96)

Now DNS code doesn't look up a name twice, even if one is a failed request.

1.91beta (05-Jul-96)

Will now output in any of several languages.
Preformatted output introduced.
New File Type Report.
Can limit the number of rows in the time reports.
Number of requests for pages (as opposed to raw requests) now calculated throughout.
DNS lookup returns, with caching across runs.
Logfiles can include wildcards.
Wildcards can include multiple *'s.
Can process case insensitive logfiles.
OUTPUTALIAS commands introduced.
New commands to specify exactly what is included, and what linked, in the request report and referrer report.
FILEALIAS a a and FILEALIAS a b; FILEALIAS b c now work.
New ALLOW options to cancel INCLUDES.
REPSEPCHAR and DECPOINT introduced.
DIRSUFFIX introduced.
Debugging reports number of corrupt lines in other logs.
Hash sizes can now be allocated at run time.
stdin can now be used for any input file, but not for two.
Macintosh version now quits automatically if no warnings have been issued.
Form interface made more secure.
"Mozilla (compatible)" separated out in Browser Summary.
Major internal changes should improve speed.
Code for non-Unix platforms integrated into main code.
"Referrer" spelled correctly.
Licence introduced.
Update file introduced.
Readme updated to include non-Unix instructions.

(19-Apr-96)

First Mac version.

1.9beta6 (11-Apr-96)

Two bug fixes (number of bytes was incorrectly reported in some cases, and -v would overwrite the OUTFILE).
Documentation improved.

1.9beta5 (06-Mar-96)

More bug fixes...

1.9beta4 (05-Mar-96)

One important bug fix (I broke GRAPHICAL OFF in 1.9beta3).
New form cgi options: ch, gr and ou=3.
Code shortened.

(05-Mar-96)

First DOS version.

1.9beta3 (01-Mar-96)

Mainly bug fixes and improved documentation.
Browser and referer reports now include failed requests.
The WARNINGS option can now be specified on the form.

1.9beta2 (07-Feb-96)

Small bug fixes.

1.9beta (06-Feb-96)

Lots of changes. The most important new features are

• Six new reports (hourly report, browser report, browser summary, referer report, status code report and error report).
• Analysis of NCSA/Apache referer log, agent log and combined log formats.
• Graphical time reports that still work on text-based browsers.
• Configurable columns in the time reports.
• Time reports can run backwards.
• Time graphs can be plotted by bytes instead of by requests.
• Can cache old data so that old logfiles need not be kept.
• Can process several logfiles.
• Can combine logfiles from several different hosts.
• Will uncompress compressed logfiles.
• All configuration options can now be specified on the commandline.
• Mandatory configuration file added.
• Lots of new options in the form processing program.
• Wildcards greatly improved throughout.
• Alphabetical host report right-aligned.
• Bytes now quoted as MBytes etc. instead of long number.
• Produces HTML2.0 compliant output.
• New sort method RANDOM (saves time for long reports).
• Floors for reports now work properly.
• Can now specify a report FROM 100 or more days ago.
• Option to turn off warnings.
• Considerable savings in code length over previous versions.

What was new in version 1?

What was new in version 1?

What's new in version 6?

What was new in version 5?

What was new in version 4?

What was new in version 3?

What was new in version 2?

1.2.6 (08-Jun-96)

Minor bug fix; will only affect those with corrupt logfiles.

1.2.5 (05-Feb-96)

Minor bug fix for weekly report.

1.2.4 (23-Jan-96)

Patch for Spyglass server logfile format.

1.2.3 (22-Nov-95)

A couple of bug fixes (wild subdomains sometimes caused crashes).
-v option now gives the version number.

1.2.2 (13-Nov-95)

Patch for proxy servers: http:// not translated to http:/

1.2 (11-Nov-95)

Can configure columns in reports to give percentage requests and number of bytes.
Wild subdomains (e.g., *.com).
Nameless subdomains.
Subdomains now listed in alphabetical order.
Proper support for numerical hostnames in HOSTIGNORE, HOSTONLY, SUBDOMAIN and alphabetical sorting.
New BASEURL command allowing statistics to be displayed on other servers.
Output always says how things are sorted.
"Last 7 days" now behaves sensibly with TO.
Filenames containing /../, /./ and // translated.
Header and footer options removed from form (for security reasons).

1.1 (02-Oct-95)

Form interface introduced.
ASCII output now possible as well as HTML.
Output file can now be specified in the configuration file.
FROM and TO commands more powerful.
DEBUG and BACKGROUND introduced.
One bug fix: alphabetical sorting doesn't now swap some hostnames.
List of primes included in distribution.

1.0 (12-Sep-95)

Only minor changes since 0.94beta.

0.94beta (30-Aug-95)

New configuration variables SEPCHAR and REPORTORDER.
New configuration commands WITHARGS and WITHOUTARGS.
New commandline options +-A and +-x. (Config.: ALL and GENERAL).
Logfile entries with - as the return code are now regarded as successes, not corrupt entries.
Fixed bugs in host report when aliases or numerical hosts are present.
Documentation rewritten.

0.93beta (27-Jul-95)

Approximate hostname counting now possible in fixed memory.
New configuration commands ISPAGE and ISNOTPAGE.
New commandline option -v.
New configuration command WEEKBEGINSON.
Proper error message when memory exceeded.
Program split into several files.

0.92beta (11-Jul-95)

New reports introduced: hostname, full daily, and weekly.
FROM and TO commands introduced.
Header and footer files introduced.
More helpful warning messages.
Ability to read configuration instructions from stdin.
Subdomain commands moved from domains file to configuration file.
Makefile provided.

0.91beta (04-Jul-95)

Configuration file introduced, enabling many new options.
Some bug fixes and speed improvements.
Ability to print "top n" reports (rather than "everything higher than n").
Request report can print only pages.
Ability to try and resolve numerical addresses.
Now less fussy about the format of the domains file.
Logo added.
Readme converted to HTML.

0.9beta

More speed improvements, and some bug fixes.
Introduced -u option.
Introduced subdomain analysis.
Included "not modified" replies as successes, not redirects.
First public release at 0.9beta3. (29-Jun-95)

0.89beta (21-Jun-95)

Commandline arguments.
Efficiency improvements.
Host count and "last 7 day" statistics.

0.8beta (14-Jun-95)

Initial program, just default options.

Quick reference

This section is divided into the following parts:

• Notation
• Input and output files
• LOGFORMAT commands
• ALIAS commands
• INCLUDE/EXCLUDE commands
• DNS commands
• Sub-item commands
• LOWMEM commands
• Report commands
• GRAPH commands
• BACK commands
• ROWS commands
• COLS commands
• SORTBY commands
• FLOOR commands
• CHART commands
• Hyperlinks
• Language commands
• Cosmetic and miscellaneous commands
• Diagnostics

Notation

"stuff"        the word stuff
x y            x followed by y
(x | y)        x or y
[x]            optional x
subset("...")  any letters from the string, in any order
perm("...")    all the letters from the string, in any order
*x             x may contain wildcards * and ? (and often comma-separated list)
x := y         x is defined to be y
COMMAND        the command under discussion

char           a single character
string         a string
digit          a digit
number         a non-negative integer (i.e. a string of digits)
real           a non-negative real number
regexp         a Perl-syntax regular expression
file           a filename within your server's filespace;
                   e.g. /index.html
localfile      a filename within your system's filespace;
                   e.g. /usr/local/analog.html
                      or analog.html
                 if no directory specified, placed within suitable
                 directory specified at compile-time
localfmtfile   as localfile, but may contain date codes;
                   e.g. /usr/local/analog%y%M.html
referrer       a URL of a referring page;
                   e.g. http://search.yahoo.com/
URL            a URL which may be absolute, or relative to the output page;
                   e.g. images/ or /~fred/images/
                      or http://www.fred.com/images/
fmtURL         as URL, but may contain date codes

Note: I have occasionally opted for clarity above strict accuracy where I don't think it will cause any confusion!

The syntax for commands in general was given earlier: remember that an argument which contains a hash or a space must be put in quotes or parentheses.

Input and output files

Syntax

LOGFILE (*localfmtfile | "-" | "none") [prefix_string]
OUTFILE (localfmtfile | "-" | "none")
CACHEFILE (*localfmtfile | "-" | "none")
CACHEOUTFILE (localfmtfile | "-" | "none")
UNCOMPRESS *localfile program

Examples

LOGFILE /httpd/logs/*
LOGFILE c:\logs\log1,c:\logs\log2
OUTFILE "Hard Disk:Report%Y%M.html"
UNCOMPRESS *.Z "/usr/bin/uncompress -c"

LOGFORMAT commands

Syntax

format_string := (see documentation)
Apache_format_string := (see Apache documentation)
logformat := ("COMMON" | "COMBINED" | "REFERRER" | "BROWSER" | "EXTENDED" |
              "MICROSOFT-NA" | "MICROSOFT-INT" | "WEBSITE-NA" | "WEBSITE-INT" |
              "MS-EXTENDED" | "WEBSTAR-EXTENDED" | "MS-COMMON" | "NETSCAPE" |
              "WEBSTAR" | "MACHTTP" | "AUTO" | format_string)
LOGFORMAT logformat
DEFAULTLOGFORMAT logformat
APACHELOGFORMAT Apache_format_string
APACHEDEFAULTLOGFORMAT Apache_format_string

Notes

LOGFORMAT and APACHELOGFORMAT only affect logfiles occurring later in the same configuration file.

Examples

LOGFORMAT (%S - %u [%d/%M/%Y:%h:%n:%j %j] "%j %r %j" %c %b)
DEFAULTLOGFORMAT MS-EXTENDED
APACHELOGFORMAT (%h %l %u %t \"%r\" %s %b)

ALIAS commands

1. Commands (items)

FILEALIAS, HOSTALIAS, BROWALIAS, REFALIAS, USERALIAS, VHOSTALIAS

Syntax

COMMAND *olditem ["PLAIN:"]newitem
COMMAND ("REGEXP:" | "REGEXPI:")regexp ["PLAIN:"]newitem

Notes

Aliases item in all reports. Items with the same resultant name are combined. newitem may contain $1, $2 etc., representing the *'s in olditem or the bracketed subexpressions in regexp. (PLAIN: makes $'s on the right-hand side into literal $'s.)

Examples

FILEALIAS /*/football/* /$1/soccer/$2

USERALIAS REGEXP:^([^U].*) U$1

2. Commands (reports)

TYPEALIAS, HOSTREPALIAS, REDIRHOSTALIAS, FAILHOSTALIAS, REQALIAS, REDIRALIAS, FAILALIAS, DIRALIAS, DOMALIAS, ORGALIAS, REFREPALIAS, REFSITEALIAS, REDIRREFALIAS, FAILREFALIAS, BROWREPALIAS, BROWSUMALIAS, OSALIAS, VHOSTREPALIAS, REDIRVHOSTALIAS, FAILVHOSTALIAS, USERREPALIAS, REDIRUSERALIAS, FAILUSERALIAS

Syntax

COMMAND *item ["PLAIN:"]string
COMMAND ("REGEXP:" | "REGEXPI:")regexp ["PLAIN:"]string

Notes

Aliases item on one line of one report only. $1, $2 etc. in string are interpreted as explained above.

Examples

REQALIAS /football/ "/football/ (Main football page)"

REFREPALIAS REGEXP:^(http://([^/]*\.)?(maths|stats)\.uxy\.edu.*) ([$3] $1)

3. Other commands: syntax

CASE ("SENSITIVE" | "INSENSITIVE")
USERCASE ("SENSITIVE" | "INSENSITIVE")
SEARCHCHARCONVERT ("ON" | "OFF")
DIRSUFFIX suffix
LOGTIMEOFFSET ["+" | "-"] number
TIMEOFFSET ["+" | "-"] number
304ISSUCCESS ("ON" | "OFF")

Examples

CASE SENSITIVE
DIRSUFFIX index.htm
LOGTIMEOFFSET -300

INCLUDE/EXCLUDE commands

1. Commands (items)

FILEINCLUDE, FILEEXCLUDE, HOSTINCLUDE, HOSTEXCLUDE, BROWINCLUDE, BROWEXCLUDE, REFINCLUDE, REFEXCLUDE, USERINCLUDE, USEREXCLUDE, VHOSTINCLUDE, VHOSTEXCLUDE

Syntax

COMMAND (*item | "")
COMMAND ("REGEXP:" | "REGEXPI:")regexp

Notes

Excludes all logfile entries containing an excluded item from all reports. Includes and excludes are done after aliases, so the item is the aliased name, if applicable. HOSTINCLUDE and HOSTEXCLUDE can also take IP address ranges and subnet masks.

Examples

FILEINCLUDE /jim/*,/jane/*
FILEINCLUDE REGEXP:^/~[^/]*/$
HOSTEXCLUDE proxy*.aol.com
USEREXCLUDE ""

2. Syntax (including and excluding status codes)

range := (number | number "-" number | number "-" | "-" number | "*")
STATUSINCLUDE range [, ranges]
STATUSEXCLUDE range [, ranges]

Notes

All numbers must be in the range 100-599.

Example

STATUSINCLUDE 200-299,304

3. Syntax (including and excluding dates)

partdate := ["+" | "-"] digit digit
date := partdate partdate partdate [":" partdate partdate]
FROM date
TO date

Examples

FROM 990719:1200
TO -00-0101

4. Commands (reports)

REQINCLUDE, REQEXCLUDE, REDIRINCLUDE, REDIREXCLUDE, FAILINCLUDE, FAILEXCLUDE, TYPEINCLUDE, TYPEEXCLUDE, DIRINCLUDE, DIREXCLUDE, HOSTREPINCLUDE, HOSTREPEXCLUDE, REDIRHOSTINCLUDE, REDIRHOSTEXCLUDE, FAILHOSTINCLUDE, FAILHOSTEXCLUDE, DOMINCLUDE, DOMEXCLUDE, ORGINCLUDE, ORGEXCLUDE, REFREPINCLUDE, REFREPEXCLUDE, REFSITEINCLUDE, REFSITEEXCLUDE, SEARCHQUERYINCLUDE, SEARCHQUERYEXCLUDE, SEARCHWORDINCLUDE, SEARCHWORDEXCLUDE, INTSEARCHQUERYINCLUDE, INTSEARCHQUERYEXCLUDE, INTSEARCHWORDINCLUDE, INTSEARCHWORDEXCLUDE, REDIRREFINCLUDE, REDIRREFEXCLUDE, FAILREFINCLUDE, FAILREFEXCLUDE, BROWREPINCLUDE, BROWREPEXCLUDE, BROWSUMINCLUDE, BROWSUMEXCLUDE, OSINCLUDE, OSEXCLUDE, VHOSTREPINCLUDE, VHOSTREPEXCLUDE, REDIRVHOSTINCLUDE, REDIRVHOSTEXCLUDE, FAILVHOSTINCLUDE, FAILVHOSTEXCLUDE, USERREPINCLUDE, USERREPEXCLUDE, REDIRUSERINCLUDE, REDIRUSEREXCLUDE, FAILUSERINCLUDE, FAILUSEREXCLUDE

Syntax

COMMAND *item
COMMAND ("REGEXP:" | "REGEXPI:")regexp

Notes

Excludes an excluded item from one report only. HOSTREP*, REDIRHOST* and FAILHOST* can also take IP address ranges and subnet masks.

Example

REQINCLUDE pages

5. Commands (hyperlinks)

See below.

6. Syntax (miscellaneous)

PAGEINCLUDE *file
PAGEEXCLUDE *file
ARGSINCLUDE *file
ARGSEXCLUDE *file
REFARGSINCLUDE *referrer
REFARGSEXCLUDE *referrer
ROBOTINCLUDE *browser
ROBOTEXCLUDE *browser

Notes

These can be regular expressions too.

Examples

PAGEINCLUDE *.jsp
ROBOTINCLUDE *crawler*

DNS commands

Syntax

DNSFILE localfile
DNS ("NONE" | "READ" | "LOOKUP" | "WRITE")
DNSLOCKFILE localfile
DNSGOODHOURS number
DNSBADHOURS number
DNSTIMEOUT number

Examples

DNSFILE dnscache.txt
DNS WRITE
DNSBADHOURS 48

Sub-item commands

Syntax

SUBDIR *file
SUBDOMAIN *subdomain
SUBDOMAIN subdomain name
SUBORG *subdomain
SUBTYPE *extension
SUBBROW *browser
REFDIR *referrer

Examples

SUBDIR /jim/*/*
SUBTYPE *.gz

LOWMEM commands

Commands

FILELOWMEM, HOSTLOWMEM, BROWLOWMEM, REFLOWMEM, USERLOWMEM, VHOSTLOWMEM

Syntax

COMMAND ("0" | "1" | "2" | "3")

Example

HOSTLOWMEM 3

Report commands

Commands

GENERAL, ALL, YEARLY, QUARTERLY, MONTHLY, WEEKLY, DAILYREP, DAILYSUM, HOURLYREP, HOURLYSUM, WEEKHOUR, QUARTERREP, QUARTERSUM, FIVEREP, FIVESUM, HOST, REDIRHOST, FAILHOST, ORGANISATION, DOMAIN, REQUEST, DIRECTORY, FILETYPE, SIZE, PROCTIME, REDIR, FAILURE, REFERRER, REFSITE, SEARCHQUERY, SEARCHWORD, INTSEARCHQUERY, INTSEARCHWORD, REDIRREF, FAILREF, BROWSERREP, BROWSERSUM, OSREP, VHOST, REDIRVHOST, FAILVHOST, USER, REDIRUSER, FAILUSER, STATUS

Syntax

REPORTCOMMAND ("ON" | "OFF")

Examples

ALL OFF
HOURLYREP ON

GRAPH commands

Commands

ALLGRAPH, YEARGRAPH, QUARTERLYGRAPH, MONTHGRAPH, WEEKGRAPH, DAYREPGRAPH, DAYSUMGRAPH, HOURREPGRAPH, HOURSUMGRAPH, WEEKHOURGRAPH, QUARTERREPGRAPH, QUARTERSUMGRAPH, FIVESUMGRAPH, FIVEREPGRAPH

Syntax

COMMAND ("R" | "r" | "P" | "p" | "B" | "b")

Example

ALLGRAPH B

BACK commands

Commands

ALLBACK, YEARBACK, QUARTERLYBACK, MONTHBACK, WEEKBACK, DAYREPBACK, HOURREPBACK, QUARTERREPBACK, FIVEREPBACK

Syntax

COMMAND ("ON" | "OFF")

Example

ALLBACK ON

ROWS commands

Commands

YEARROWS, QUARTERLYROWS, MONTHROWS, WEEKROWS, DAYREPROWS, HOURREPROWS, QUARTERREPROWS, FIVEREPROWS

Syntax

COMMAND number

Example

QUARTERREPROWS 192

COLS commands

1. Commands (time reports)

TIMECOLS, YEARCOLS, QUARTERLYCOLS, WEEKCOLS, MONTHCOLS, WEEKCOLS, DAYREPCOLS, DAYSUMCOLS, HOURREPCOLS, HOURSUMCOLS, WEEKHOURCOLS, QUARTERREPCOLS, QUARTERSUMCOLS, FIVEREPCOLS, FIVESUMCOLS

Syntax

cols1 := subset("RrPpBb")
COMMAND cols1

Example

MONTHCOLS bRP

2. Commands (most success reports)

HOSTCOLS, ORGCOLS, DOMCOLS, DIRCOLS, REFCOLS, REFSITECOLS, SEARCHQUERYCOLS, SEARCHWORDCOLS, INTSEARCHQUERYCOLS, INTSEARCHWORDCOLS, BROWREPCOLS, BROWSUMCOLS, OSCOLS, VHOSTCOLS, USERCOLS

Syntax

cols2 := subset("NDdEeRrSsPpQqBbCc")
COMMAND cols2

Example

USERCOLS BD

3. Syntax (Request and File Type Reports)

REQCOLS subset("NDdEeRrSspqBbCc")
TYPECOLS subset("NDdEeRrSsBbCc")

Example

TYPECOLS NRb

4. Commands (failure, redirection and Status Code reports)

REDIRCOLS, FAILCOLS, REDIRHOSTCOLS, FAILHOSTCOLS, REDIRREFCOLS, FAILREFCOLS, REDIRVHOSTCOLS, FAILVHOSTCOLS, REDIRUSERCOLS, FAILUSERCOLS, STATUSCOLS

Syntax

cols4 := subset("NDdEeRrSs")
COMMAND cols4

Example

FAILCOLS D

5. Commands (Size and Processing Time Reports)

SIZECOLS, PROCTIMECOLS

Syntax

cols5 := subset("DdEeRrSsPpQqBbCc")
COMMAND cols5

Example

SIZECOLS RB

SORTBY commands

1. Commands (most success reports)

HOSTSORTBY, ORGSORTBY, DOMSORTBY, DIRSORTBY, REFSORTBY, REFSITESORTBY, SEARCHQUERYSORTBY, SEARCHWORDSORTBY, INTSEARCHQUERYSORTBY, INTSEARCHWORDSORTBY, BROWREPSORTBY, BROWSUMSORTBY, OSSORTBY, VHOSTSORTBY, USERSORTBY, SUBDIRSORTBY, SUBDOMSORTBY, SUBORGSORTBY, SUBBROWSORTBY, SUBOSSORTBY, REFDIRSORTBY, REFARGSSORTBY

Syntax

sortby1 := ("REQUESTS" | "REQUESTS7" | "PAGES" | "PAGES7" |
            "BYTES" | "BYTES7" | "DATE" | "FIRSTDATE" |
            "ALPHABETICAL" | "RANDOM")
COMMAND sortby1

Example

DOMSORTBY ALPHABETICAL

2. Commands (Request and File Type Reports)

REQSORTBY, TYPESORTBY, REQARGSSORTBY, SUBTYPESORTBY

Syntax

sortby2 := ("REQUESTS" | "REQUESTS7" | "BYTES" | "BYTES7" |
            "DATE" | "FIRSTDATE" | "ALPHABETICAL" | "RANDOM")
COMMAND sortby2

Example

REQSORTBY REQUESTS

3. Commands (failure, redirection and Status Code reports)

REDIRSORTBY, FAILSORTBY, REDIRHOSTSORTBY, FAILHOSTSORTBY, REDIRREFSORTBY, FAILREFSORTBY, REDIRVHOSTSORTBY, FAILVHOSTSORTBY, REDIRUSERSORTBY, FAILUSERSORTBY, STATUSSORTBY, REDIRARGSSORTBY, FAILARGSSORTBY, REDIRREFARGSSORTBY, FAILREFARGSSORTBY

Syntax

sortby3 := ("REQUESTS" | "REQUESTS7" | "DATE" | "FIRSTDATE" |
            "ALPHABETICAL" | "RANDOM")
COMMAND sortby3

Example

FAILSORTBY DATE

FLOOR commands

Commands (top-level)

HOSTFLOOR, REDIRHOSTFLOOR, FAILHOSTFLOOR, ORGFLOOR, DOMFLOOR, REQFLOOR, DIRFLOOR, TYPEFLOOR, REDIRFLOOR, FAILFLOOR, REFFLOOR, REFSITEFLOOR, SEARCHQUERYFLOOR, SEARCHWORDFLOOR, INTSEARCHQUERYFLOOR, INTSEARCHWORDFLOOR, REDIRREFFLOOR, FAILREFFLOOR, BROWREPFLOOR, BROWSUMFLOOR, OSFLOOR, VHOSTFLOOR, REDIRVHOSTFLOOR, FAILVHOSTFLOOR, USERFLOOR, REDIRUSERFLOOR, FAILUSERFLOOR, STATUSFLOOR

Commands (lower levels)

REQARGSFLOOR, REDIRARGSFLOOR, FAILARGSFLOOR, REFARGSFLOOR, REDIRREFARGSFLOOR, FAILREFARGSFLOOR, SUBDIRFLOOR, SUBDOMFLOOR, SUBORGFLOOR, SUBTYPEFLOOR, SUBBROWFLOOR, SUBOSFLOOR, REFDIRFLOOR

Syntax

partdate := ["+" | "-"] digit digit
date := partdate partdate partdate [":" partdate partdate]
COMMAND number ("r" | "s" | "p" | "q")
COMMAND number ["k" | "M" | "G" | "T" | "P" | "E" | "Z" | "Y"] ("b" | "c")
COMMAND real ("%" | ":") ("r" | "s" | "p" | "q" | "b" | "c")
COMMAND date ("d" | "e")
COMMAND "-" number ("r" | "s" | "p" | "q" | "b" | "c" | "d" | "e")

Notes

Actually, this syntax isn't quite correct. REQFLOOR, TYPEFLOOR, REQARGSFLOOR and SUBTYPEFLOOR aren't allowed to be of type "p" or "q"; and REDIRFLOOR, FAILFLOOR, REDIRHOSTFLOOR, FAILHOSTFLOOR, REDIRREFFLOOR, FAILREFFLOOR, REDIRVHOSTFLOOR, FAILVHOSTFLOOR, REDIRUSERFLOOR, FAILUSERFLOOR, STATUSFLOOR, REDIRARGSFLOOR, FAILARGSFLOOR, REDIRREFARGSFLOOR and FAILREFARGSFLOOR aren't allowed to be of type "p", "q", "b" or "c".

Examples

TYPEFLOOR -20r
REQARGSFLOOR 0.1%b

CHART commands

1. Commands (most success reports)

HOSTCHART, ORGCHART, DOMCHART, REQCHART, DIRCHART, REFCHART, REFSITECHART, SEARCHQUERYCHART, SEARCHWORDCHART, INTSEARCHQUERYCHART, INTSEARCHWORDCHART, BROWREPCHART, BROWSUMCHART, OSCHART, VHOSTCHART, USERCHART, SIZECHART, PROCTIMECHART

Syntax

chart1 := ("ON" | "OFF" | "REQUESTS" | "REQUESTS7" | "PAGES" | "PAGES7" |
            "BYTES" | "BYTES7")
COMMAND chart1

Example

DOMCHART BYTES

2. Commands (failure, redirection and Status Code reports)

REDIRHOSTCHART, FAILHOSTCHART, REDIRCHART, FAILCHART, REDIRREFCHART, FAILREFCHART, REDIRVHOSTCHART, FAILVHOSTCHART, REDIRUSERCHART, FAILUSERCHART, STATUSCHART

Syntax

chart2 := ("ON" | "OFF" | "REQUESTS" | "REQUESTS7")
COMMAND chart2

Example

FAILCHART ON

3. Syntax (TYPECHART)

TYPECHART ("ON" | "OFF" | "REQUESTS" | "REQUESTS7" | "BYTES" | "BYTES7")

4. Syntax (ALLCHART)

ALLCHART ("ON" | "OFF")

5. CHARTEXPAND commands

DIRCHARTEXPAND, DOMCHARTEXPAND, ORGCHARTEXPAND, TYPECHARTEXPAND, BROWCHARTEXPAND, OSCHARTEXPAND, REQCHARTEXPAND, REDIRCHARTEXPAND, FAILCHARTEXPAND, REFCHARTEXPAND, REDIRREFCHARTEXPAND, FAILREFCHARTEXPAND, REFSITECHARTEXPAND

Syntax

COMMAND item[,item...]

Example

DOMCHARTEXPAND .com,.net

Hyperlinks

Syntax

REQLINKINCLUDE *file
REQLINKEXCLUDE *file
REDIRLINKINCLUDE *file
REDIRLINKEXCLUDE *file
FAILLINKINCLUDE *file
FAILLINKEXCLUDE *file
REFLINKINCLUDE *referrer
REFLINKEXCLUDE *referrer
REDIRREFLINKINCLUDE *referrer
REDIRREFLINKEXCLUDE *referrer
FAILREFLINKINCLUDE *referrer
FAILREFLINKEXCLUDE *referrer
BASEURL prefix_string

Notes

The LINK commands can be regular expressions too.

Examples

REQLINKINCLUDE pages
REFLINKINCLUDE *.cgi,*.cgi?*
BASEURL http://www.mycompany.com

Language commands

Syntax

LANGUAGE ("ARMENIAN" | "BASQUE" | "BULGARIAN" | "BULGARIAN-MIK" |
          "CATALAN" | "SIMP-CHINESE" | "TRAD-CHINESE" | "CZECH" |
          "CZECH-1250" | "DANISH" | "DUTCH" | "ENGLISH" | "US-ENGLISH" |
          "FINNISH" | "FRENCH" | "GERMAN" | "HUNGARIAN" | "INDONESIAN" |
          "ITALIAN" | "JAPANESE-EUC" | "JAPANESE-JIS" | "JAPANESE-SJIS" |
          "JAPANESE-UTF" | "KOREAN" | "LATVIAN" | "NORWEGIAN" | "NYNORSK" |
          "POLISH" | "PORTUGUESE" | "BR-PORTUGUESE" | "RUSSIAN" |
          "RUSSIAN-1251" | "SERBIAN" | "SLOVAK" | "SLOVAK-1250" | "SLOVENE" |
          "SLOVENE-1250" | "SPANISH" | "SWEDISH" | "SWEDISH-ALT" | "TURKISH" |
          "UKRAINIAN")
LANGFILE localfile
DOMAINSFILE localfile
DESCFILE localfile

Notes

I hope that the other languages which were available in previous versions of analog -- BOSNIAN, SIMP-CHINESE, TRAD-CHINESE, CROATIAN, FINNISH, GREEK, ICELANDIC, LITHUANIAN, BR-PORTUGUESE, ROMANIAN, SLOVAK and SPANISH -- will be available for download from the analog home page soon.

Examples

LANGUAGE ITALIAN
LANGFILE hindi.lng

Cosmetic and miscellaneous commands

Syntax

OUTPUT ("XHTML" | "HTML" | "PLAIN" | "ASCII" | "LATEX" | "COMPUTER" | "XML" | "NONE")
GOTOS ("ON" | "OFF" | "FEW")
RUNTIME ("ON" | "OFF")
DESCRIPTIONS ("ON" | "OFF")
REPORTSPAN ("ON" | "OFF")
REPORTSPANTHRESHOLD number
LASTSEVEN ("ON" | "OFF")
REPORTORDER perm("x1QmWDdHwh4657oZSlLujJkKfsNnBbpvRMcPztiEIYyr")
GENSUMLINES ("ALL" | ["+" | "-"] subset("BCDEFGHIJKLMN"))
IMAGEDIR URL
CHARTDIR fmtURL
LOCALCHARTDIR localfmtfile
JPEGCHARTS ("ON" | "OFF")
PNGIMAGES ("ON" | "OFF")
NOROBOTS ("ON" | "OFF")
LOGO (URL | "none")
LOGOURL (URL | "none")
HOSTNAME string
HOSTURL (URL | "none")
HEADERFILE (localfile | "none")
FOOTERFILE (localfile | "none")
STYLESHEET (URL | "none")
CSSPREFIX (string | "none")
SEPCHAR (char | "none")
REPSEPCHAR (char | "none")
DECPOINT char
COMPSEP string
RAWBYTES ("ON" | "OFF")
BYTESDP number
HTMLPAGEWIDTH number
PLAINPAGEWIDTH number
LATEXPAGEWIDTH number
PDFLATEX ("ON" | "OFF")
BARSTYLE ("a" | "b" | "c" | "d" | "e" | "f" | "g" | "h" | "i" | "j")
MARKCHAR char
MINGRAPHWIDTH number
WEEKBEGINSON ("SUNDAY" | "MONDAY" | "TUESDAY" | "WEDNESDAY" | "THURSDAY" | "FRIDAY" | "SATURDAY")
SEARCHENGINE *referrer comma-separated-strings
INTSEARCHENGINE *file comma-separated-strings

Examples

Too many to list. See the documentation on each individual command.

Diagnostics

Syntax

SETTINGS ("ON" | "OFF")
DEBUG ("ON" | "OFF" | ["+" | "-"] subset("CDFSU"))
WARNINGS ("ON" | "OFF" | ["+" | "-"] subset("CDEFLMR"))
PROGRESSFREQ number
ERRFILE localfile
ERRLINELENGTH number

Examples

DEBUG ON
DEBUG CF
WARNINGS -DL
PROGRESSFREQ 50000

Index

This is the index for this Readme. Follow the numbers after each name to find references to that command or concept. Families of commands are indexed under the second part of the name: for example, HOSTEXCLUDE is under *EXCLUDE, not under HOST.

This index includes all of analog's configuration commands: if a command you used in previous versions is not here, see the section on Upgrading from earlier versions. All commands are also listed in the Quick reference with their syntax and examples, and that section is not cross-referenced from this index.

Acknowledgements [1]
Addresses, numerical [1]
*ALIAS [1]
Aliases [1]
ALL [1]
ALLBACK [1]
ALLGRAPH [1]
analog.cfg [1][2][3][4]
anlgform.html [1]
anlgform.pl [1]
anlghead.h [1][2]
Announcements [1]
APACHEDEFAULTLOGFORMAT [1]
APACHELOGFORMAT [1]
ARGSEXCLUDE [1]
*ARGSFLOOR [1]
ARGSINCLUDE [1]
*ARGSSORTBY [1]
Arguments in URLs [1][2]
ASCII output style [1]
*BACK [1]
Bar charts [1]
BARSTYLE [1]
BASEURL [1]
Basic commands [1]
Broken pipe [1][2]
BROW* commands - see under second part of name
Browser Report [1][2][3]
Browser Summary [1][2][3][4]
BROWREP* commands - see under second part of name
BROWSERREP [1]
BROWSERSUM [1]
BROWSUM* commands - see under second part of name
Bugs, reporting [1]
Bytes, how displayed [1]
BYTESDP [1]
bzip2 logfiles [1]
Cache files [1]
CACHEOUTFILE [1]
CACHEFILE [1]
CASE [1]
CGI program [1]
*CHART [1]
CHARTDIR [1]
*CHARTEXPAND [1]
"Click-thru"s [1]
Colours [1]
*COLS [1][2]
Comma separated value output [1]
Command line arguments [1][2][3]

• logfile name (LOGFILE) [1]
• - (LOGFILE stdin) [1]
• 1 (Yearly Report) [1]
• 4 (Quarter-Hour Report) [1]
• 5 (Five-Minute Report) [1]
• 6 (Quarter-Hour Summary) [1]
• 7 (Five-Minute Summary) [1]
• A (All reports) [1]
• a (HTML/PLAIN output) [1]
• B (Browser Report) [1][2]
• b (Browser Summary) [1][2]
• c (Status Code Report) [1][2]
• C (Arbitrary configuration command) [1]
• D (Daily Report) [1]
• d (Daily Summary) [1]
• E (Redirection Report) [1][2]
• F (FROM date) [1]
• f (Referrer Report) [1][2]
• G (Default configuration file) [1]
• g (Other configuration files) [1]
• H (Hourly Report) [1]
• h (Hourly Summary) [1]
• I (Failure Report) [1][2]
• i (Directory Report) [1][2]
• J (User Failure Report) [1][2]
• j (User Redirection Report) [1][2]
• K (Failed Referrer Report) [1][2]
• k (Redirected Referrer Report) [1][2]
• L (Host Failure Report) [1][2]
• l (Host Redirection Report) [1][2]
• M (Virtual Host Failure Report) [1][2]
• m (Monthly Report) [1]
• N (Search Query Report) [1][2]
• n (Search Word Report) [1][2]
• O (Output file) [1]
• o (Domain Report) [1][2]
• P (Processing Time Report) [1]
• p (Operating System Report) [1][2]
• Q (Quarterly Report) [1]
• q (Warnings) [1]
• R (Virtual Host Redirection Report) [1][2]
• r (Request Report) [1][2]
• S (Host Report) [1][2]
• s (Referring Site Report) [1][2]
• settings (Settings of all variables) [1][2]
• T (TO date) [1]
• t (File Type Report) [1][2]
• U (Cache file) [1]
• u (User Report) [1][2]
• V (Debugging) [1]
• v (Virtual Host Report) [1][2]
• version (Just give the version number) [1]
• W (Weekly Report) [1]
• w (Hour of the Week Summary) [1]
• X (Goto's) [1]
• x (General Summary) [1]
• Y (Internal Search Query Report) [1][2]
• y (Internal Search Word Report) [1][2]
• Z (Organisation Report) [1][2]
• z (File Size Report) [1]