Monitoring Web Server Usage Across A Community

How many public Web servers are there at your University? And how have the numbers changed over the past 5 years? Are you running more servers, as the range of services you provide grows, or have the numbers of servers decreased due to rationalisation in order to avoid duplication of effort across the institution?

Server numbers in March 2002I published an article on A Survey Of Numbers of UK University Web Servers in June 2000, with a follow-up article on An Update Of A Survey Of The Numbers of UK University Web Servers which was published in March 2002.  

The survey was carried out using the online Netcraft service, by using a wildcard (* to obtain details of the numbers of Web servers in, in this case, the Oxford university domain.  This process was repeated manually for all (~160) UK HEIs. A histogram for the results of the 2002 survey is illustrated.

Netcraft listing of Oxford Web serversHow have things changed in the past 5 years? It would be possible to repeat the manual survey – as can be seen, the online Netcraft survey service is still available.

However in a Web 2.0 environment in which many lightweight Web-based tools are available it would not be sensible to repeat the methodology.  It strikes me that the Netcraft results page is well-suited for screen-scraping (immediately after the “Results for *” text is a line which says “Found 356 sites“. So while this interface remains, the data can be programatically extracted, stored and displayed, possibly in a graphical format). 

The Dapper application could, perhaps, could be used for this purpose. After all, as I’ve described previously, Dapper has been used to create Blotter, which scrapes Technorati ranking data on a daily basis, stores this data and display the trends graphically.

But rather than doing this myself, I’d like to suggest that this might be a suitable example for the IWMW 2007 Innovation Competition – this should be lightweight and user-focussed (providing data which can detect trends across the community). It could be possible to provide an interface for a user to supply their own domain name, although another approach might be to take the domain names for the community (or perhaps a regional subset of the community) and display variations across the community – that, I think would be cool (and ‘coolness’ is one of the criteria for the competition). 


  1. Is this a sensible metric for comparison? The number of domain names listed on Netcraft can be wildly inaccurate and even if it was, does it mean much? I could set up a blog system similar to which gave 10,000 students Alternatively an institution could use a proxying system to map content from dozens of servers onto a single hostname.

    I agree that it would be interesting to find out more about the number of physical servers[1] and the services institutions offer online but I’m not sure an automated survey can discover the answers.

    [1] Although this is also reducing in importance as more people adopt server virtualisation – I have half a dozen VMs running on a single server, and expect that to increase significantly this year.

  2. Hi Mike
    After I announced one of the server usage surveys a few years ago, someone suggested that real soon now every institution would have thousands of servers, as desktop PCs would act as servers. That hasn’t happened – but it might be interesting to observe this if it started to happen (or the approach started to take off.
    Your point about physical servers is very interesting. I hadn’t considered that institutions may wish to monitor growth of physical servers as this could potentially be in conflict with an institutions environmental policy.

  3. Any updates on the stats that you published? Would be interesting!



  1. Web Server Monitoring - [...] Monitoring Web Server Usage Across A Community « UK Web Focus UK Web Focus Reflections On The Web …

Submit a Comment

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>