Example Schema: Variable Width Columns (Combined Web Server Log)

This example is similar to the regular Web Server Access Log, but has additional fields at the end. This is a common "Combined" log format.

database /tmp/testdb
table    htlog2
recdelim \x0a
multiple
trimspace
datefmt dd/mmm/yyyy:HH:MM:SS
# this expression should be on one line with no intervening spaces
recexpr  >>^\P=[^ ]+ +-?[^ ]* +-?[^ ]* +\[=[^\]]+]= +"=[^ ]+ +
[^ "]+ HTTP/?\digit?\.?\digit?"= +[^ ]+ +[^ ]+ +"=[^"]*"= +"=
[^/"]*/*[^ "]* *[^"]*"[^\x0a]*
#       Name            Type            Tag
field   Client          varchar(40)     2
field   Ident           varchar(40)     5
field   User            varchar(20)     8
field   Date            date            11
field   Method          varchar(10)     15
field   Request         varchar(100)    17
field   Protocol        varchar(10)     18-21
field   Status          integer         24
field   Bytes           integer         26
field   Referrer        varchar(100)    29
field   Agentname       varchar(10)     33
field   Agentver        varchar(10)     35
field   Agentinfo       varchar(50)     37
#field   Agent           varchar(50)     33-37

Here is sample data for the above schema. (The long lines don't fit on one line of the printed page).

thunder.thunderstone.com - - [05/Dec/1997:12:45:38 -0500] "GET /
HTTP/1.0"
200 8465 "" "Mozilla/3.01 (X11; I; Linux 1.2.13 i586)"
thunder.thunderstone.com - - [05/Dec/1997:12:45:39 -0500]
 "GET /jump/tbg.gif HTTP/1.0" 200 2684 "http://www.thunderstone.
com/" "Mozilla/3.01 (X11; I; Linux 1.2.13 i586)"
thunder.thunderstone.com - - [05/Dec/1997:12:45:42 -0500] "GET
/jump/Demonstrations.html HTTP/1.0" 200 3749 "http://www.
thunderstone.com/" "Mozilla/3.01 (X11; I; Linux 1.2.13 i586)"
thunder.thunderstone.com - - [05/Dec/1997:12:45:42 -0500]
 "GET /jump/smdrop.gif HTTP/1.0" 200 2886 "http://www.
thunderstone.com/jump/Demonstrations.html" "Mozilla/3.01 (X11;
I; Linux 1.2.13 i586)"
thunder.thunderstone.com - - [05/Dec/1997:12:45:52 -0500] "GET
/texis/demos/news/ HTTP/1.0" 200 230 "http://www.thunderstone.com/
jump/Demonstrations.html" "Mozilla/3.01 (X11; I; Linux 1.2.13
i586)"
thunder.thunderstone.com - - [05/Dec/1997:12:45:53 -0500] "GET
/texis/demos/news/upperframe.html HTTP/1.0" 200 6873
"http://www.thunderstone.com/jump/Demonstrations.html"
"Mozilla/3.01 (X11; I; Linux 1.2.13 i586)"
thunder.thunderstone.com - - [05/Dec/1997:12:45:53 -0500] "GET
/texis/demos/news/lowerframe.html HTTP/1.0" 200 31864
"http://www.thunderstone.com/jump/Demonstrations.html"
"Mozilla/3.01 (X11; I; Linux 1.2.13 i586)"