Статистика обращений

[sedikpoll]

Есть логи такого содержания:

Код: Выделить всё

10.20.1.2 - - [21/Nov/2010:09:42:28 +0100] "GET /forum/clientscript/vbulletin_global.js?v=5 HTTP/1.1" 304 - "https://site.org/forum/" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.12 (KHTML, like Gecko) Chrome/9.0.587.0 Safari/534.12"
10.20.1.2 - - [21/Nov/2010:09:42:28 +0100] "GET /forum/clientscript/vbulletin_menu.js?v=5 HTTP/1.1" 304 - "https://site.org/forum/" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.12 (KHTML, like Gecko) Chrome/9.0.587.0 Safari/534.12"
10.20.1.2 - - [21/Nov/2010:09:42:28 +0100] "GET /forum/clientscript/vbulletin_read_marker.js?v=5 HTTP/1.1" 304 - "https://site.org/forum/" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.12 (KHTML, like Gecko) Chrome/9.0.587.0 Safari/534.12"
10.20.1.2 - - [21/Nov/2010:09:42:28 +0100] "GET /forum/images/visions/buttons/lastpost.gif HTTP/1.1" 304 - "https://site.org/forum/" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.12 (KHTML, like Gecko) Chrome/9.0.587.0 Safari/534.12"
10.20.1.2 - - [21/Nov/2010:09:42:28 +0100] "GET /forum/images/visions/statusicon/forum_new.png HTTP/1.1" 304 - "https://site.org/forum/" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.12 (KHTML, like Gecko) Chrome/9.0.587.0 Safari/534.12"
10.05.1.2 - - [21/Nov/2010:09:42:28 +0100] "GET /forum/ HTTP/1.1" 200 12955 "https://site.org/forum/showthread.php?t=911" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.3 (KHTML, like Gecko) Chrome/6.0.472.63 Safari/534.3"
10.05.1.2 - - [21/Nov/2010:09:42:28 +0100] "GET /forum/usercp.php HTTP/1.1" 200 10163 "https://site.org/forum/" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.3 (KHTML, like Gecko) Chrome/6.0.472.63 Safari/534.3"
10.20.1.2 - - [21/Nov/2010:09:42:29 +0100] "GET /forum/images/visions/statusicon/subforum_new.gif HTTP/1.1" 304 - "https://site.org/forum/" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.12 (KHTML, like Gecko) Chrome/9.0.587.0 Safari/534.12"
10.20.1.2 - - [21/Nov/2010:09:42:29 +0100] "GET /forum/images/visions/statusicon/forum_new_lock.png HTTP/1.1" 304 - "https://site.org/forum/" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.12 (KHTML, like Gecko) Chrome/9.0.587.0 Safari/534.12"
15.6.1.2 - - [21/Nov/2010:09:42:34 +0100] "GET /forum/clientscript/vbulletin_important.css?v=35 HTTP/1.1" 200 1735 "https://site.org/forum/" "Opera/9.80 (Windows NT 6.1; U; ru) Presto/2.6.31 Version/10.70"
15.6.1.2 - - [21/Nov/2010:09:42:36 +0100] "GET /forum/clientscript/yui/yahoo-dom-event/yahoo-dom-event.js?v=35 HTTP/1.1" 200 36628 "https://site.org/forum/" "Opera/9.80 (Windows NT 6.1; U; ru) Presto/2.6.31 Version/10.70"
5.10.1.2 - - [21/Nov/2010:09:42:45 +0100] "GET /forum/archive/index.php/t-99.html HTTP/1.1" 200 1121 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
10.05.1.2 - - [21/Nov/2010:09:42:54 +0100] "GET /forum/forumdisplay.php?f=1 HTTP/1.1" 200 19218 "https://site.org/forum/showthread.php?t=911" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.3 (KHTML, like Gecko) Chrome/6.0.472.63 Safari/534.3"
10.05.1.2 - - [21/Nov/2010:09:42:56 +0100] "GET /forum/showthread.php?t=91 HTTP/1.1" 200 21521 "https://site.org/forum/forumdisplay.php?f=15" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.3 (KHTML, like Gecko) Chrome/6.0.472.63 Safari/534.3"

Хочется составить статистику обращений, но пока не получается.
Пытаюсь сделать её в следующем виде:

Код: Выделить всё

10.20.1.2
/forum/clientscript/vbulletin_global.js?v=5
/forum/clientscript/vbulletin_read_marker.js?v=5
/forum/images/visions/statusicon/forum_new.png

10.05.1.2
/forum/
/forum/usercp.php

...
т.е.

IP
"запрашиваемые" файлы.

Какой логике тут следовать и как рационально составить данную статистику?
Спасибо.

[/dev/random]

1) Вначале sed'ом представить лог в виде:
IP файл
IP файл
...
2) Затем отсортировать с дедупликацией (sort -u).
3) Затем скриптом на sed, bash или awk (в чём лучше разбираетесь) обработать каждую строку, выведя 3 строки (пустую, IP, файл) если IP отличается от предыдущего, или 1 строку (файл) если не отличается.

[SLEDopit]

Код: Выделить всё

 $ sort -u test | awk '{if ($1 == prev) print $7; else print $1"\n"$7;prev=$1;}'
10.05.1.2
/forum/
/forum/forumdisplay.php?f=1
/forum/showthread.php?t=91
/forum/usercp.php
10.20.1.2
/forum/clientscript/vbulletin_global.js?v=5
/forum/clientscript/vbulletin_menu.js?v=5
/forum/clientscript/vbulletin_read_marker.js?v=5
/forum/images/visions/buttons/lastpost.gif
/forum/images/visions/statusicon/forum_new_lock.png
/forum/images/visions/statusicon/forum_new.png
/forum/images/visions/statusicon/subforum_new.gif
15.6.1.2
/forum/clientscript/vbulletin_important.css?v=35
/forum/clientscript/yui/yahoo-dom-event/yahoo-dom-event.js?v=35
5.10.1.2
/forum/archive/index.php/t-99.html
 $ cat test
10.20.1.2 - - [21/Nov/2010:09:42:28 +0100] "GET /forum/clientscript/vbulletin_global.js?v=5 HTTP/1.1" 304 - "https://site.org/forum/" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.12 (KHTML, like Gecko) Chrome/9.0.587.0 Safari/534.12"
10.20.1.2 - - [21/Nov/2010:09:42:28 +0100] "GET /forum/clientscript/vbulletin_menu.js?v=5 HTTP/1.1" 304 - "https://site.org/forum/" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.12 (KHTML, like Gecko) Chrome/9.0.587.0 Safari/534.12"
10.20.1.2 - - [21/Nov/2010:09:42:28 +0100] "GET /forum/clientscript/vbulletin_read_marker.js?v=5 HTTP/1.1" 304 - "https://site.org/forum/" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.12 (KHTML, like Gecko) Chrome/9.0.587.0 Safari/534.12"
10.20.1.2 - - [21/Nov/2010:09:42:28 +0100] "GET /forum/images/visions/buttons/lastpost.gif HTTP/1.1" 304 - "https://site.org/forum/" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.12 (KHTML, like Gecko) Chrome/9.0.587.0 Safari/534.12"
10.20.1.2 - - [21/Nov/2010:09:42:28 +0100] "GET /forum/images/visions/statusicon/forum_new.png HTTP/1.1" 304 - "https://site.org/forum/" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.12 (KHTML, like Gecko) Chrome/9.0.587.0 Safari/534.12"
10.05.1.2 - - [21/Nov/2010:09:42:28 +0100] "GET /forum/ HTTP/1.1" 200 12955 "https://site.org/forum/showthread.php?t=911" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.3 (KHTML, like Gecko) Chrome/6.0.472.63 Safari/534.3"
10.05.1.2 - - [21/Nov/2010:09:42:28 +0100] "GET /forum/usercp.php HTTP/1.1" 200 10163 "https://site.org/forum/" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.3 (KHTML, like Gecko) Chrome/6.0.472.63 Safari/534.3"
10.20.1.2 - - [21/Nov/2010:09:42:29 +0100] "GET /forum/images/visions/statusicon/subforum_new.gif HTTP/1.1" 304 - "https://site.org/forum/" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.12 (KHTML, like Gecko) Chrome/9.0.587.0 Safari/534.12"
10.20.1.2 - - [21/Nov/2010:09:42:29 +0100] "GET /forum/images/visions/statusicon/forum_new_lock.png HTTP/1.1" 304 - "https://site.org/forum/" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.12 (KHTML, like Gecko) Chrome/9.0.587.0 Safari/534.12"
15.6.1.2 - - [21/Nov/2010:09:42:34 +0100] "GET /forum/clientscript/vbulletin_important.css?v=35 HTTP/1.1" 200 1735 "https://site.org/forum/" "Opera/9.80 (Windows NT 6.1; U; ru) Presto/2.6.31 Version/10.70"
15.6.1.2 - - [21/Nov/2010:09:42:36 +0100] "GET /forum/clientscript/yui/yahoo-dom-event/yahoo-dom-event.js?v=35 HTTP/1.1" 200 36628 "https://site.org/forum/" "Opera/9.80 (Windows NT 6.1; U; ru) Presto/2.6.31 Version/10.70"
5.10.1.2 - - [21/Nov/2010:09:42:45 +0100] "GET /forum/archive/index.php/t-99.html HTTP/1.1" 200 1121 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
10.05.1.2 - - [21/Nov/2010:09:42:54 +0100] "GET /forum/forumdisplay.php?f=1 HTTP/1.1" 200 19218 "https://site.org/forum/showthread.php?t=911" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.3 (KHTML, like Gecko) Chrome/6.0.472.63 Safari/534.3"
10.05.1.2 - - [21/Nov/2010:09:42:56 +0100] "GET /forum/showthread.php?t=91 HTTP/1.1" 200 21521 "https://site.org/forum/forumdisplay.php?f=15" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.3 (KHTML, like Gecko) Chrome/6.0.472.63 Safari/534.3"