ia play

This article is part of a series about search log analysis which includes bounce rates, spotting real opportunities and the geographical element.

As I explained on Friday I’ve been categorising the top 500 referring keywords for RNIB.org.uk.

Before I did the categorisation, Helen Keller and RNIB were far and away the biggest referrers. Both are a magnitude of 10 larger than any other individual query. But the third largest query is for Glaucoma and other specific eye conditions appeared frequently throughout the logs (as do a multitude of Helen Keller related queries) so I was interested in other significant types of query.

So after my arbitrary categorising I ended up with these types (these are not actual keywords):

1. Helen Keller
2. Specific eye conditions
3. RNIB
4. Welfare support and benefits
5. Keyboard shortcuts
6. Eye tests
7. Equality and disability rights
8. Fundraising ideas
9. Louis Braille
10. Talking Books

(these ten cover about 50% of all the keywords)

As I said earlier the categories are a bit arbitrary. I could easily have grouped eye conditions and eye tests into ‘eye health’ and Helen Keller and Louis Braille could be “historical figures”. The categorising helps more as a activity for immersing myself in the full 500 list and getting a feel for what is significant in the logs.  The actual rankings are a bit of a red herring.

Next: bounce rates

This article is part of a series about search log analysis which includes what people are searching for, bounce rates, spotting real opportunities and the geographical element.

I’ve been rooting around in the search logs for RNIB.org.uk. We use Google Analytics which isn’t accessible so most data has to be exported and shared in Excel.

So far I’ve  got my hands on:

• the top 500 keyword referrers from external search engines (2008)
• top 500 keywords used on site search (last  six months of 2008)
• top referring search engines

But that’s plenty to be getting on with.

I have to remind myself I’m only looking at the most popular terms and there’s a whole long tail I have no visibility of.  There’s also some clearly dubious queries in the logs.

So far I’ve gone through the top 500 from external search engines and loosely categorised them. The categories aren’t particularly scientific;  I’ve grouped all eye conditions into one category and grouped all queries about Helen Keller into another. Those don’t seem particularly equivalent categories but there are similar in size of queries. I’m following my instincts a bit at this stage.

For each category I’ve added up the total visits, and then worked up the average bounce, time on site and new visits per query type. I’ve also started adding information about whether the query is likely to be answered with a quick fact or should generate a longer journey.

Some of the questions I am trying to answer:

• Which queries should influence  navigation design?
• Where should we be encouraging further/longer journeys?
• What content isn’t represented in the logs? We might need to work on optimising those.
• Which queries are a poor opportunity since the referral was accidental or mis-directed

As a side benefit I’ve already learnt what Bump-ons are.

Next: what people are searching for

My first six months at the RNIB are up. I’ve spent them mostly playing catch-up and firefighting on the intranet project. Now it is time to actually do some planning.

My department’s focus is mostly internal, I’m a bit of anomaly with my work on the website.

Some constraints to bear in mind:

• I’m the only IA
• Must be extremely responsible with the charity’s money
• Small organisation, doing many things, so staff are time-poor
• Software development is not what the organisation specialises in

At the moment I’m thinking we could get alot of value of focusing quite heavily on search. I’ll have to do that anyway for the website relaunch but there are big gains to be had from getting the intranet search working well and then exploring enterprise search.

I’m also interested in what we could do with automating related links. Never quite got there at the BBC but there’s an even stronger case here. Authors have very little time to create rich related links and the tendancy is to only promote content produced by their own teams, as ever.

Some simple reference data management might also be needed, possibly to support the enterprise search and automated links plan but also to improve interoperability of all sorts of systems.

Just first thoughts…

We’re starting the website phase of our programme of work. The intranet and Teamsites haven’t launched yet but development is wrapping up. There’s still testing, content migration, and training going on so that is hampering our ability to spend as much time on the website planning as I would like.

There are reasons to be optimistic that this phase will run more smoothly than the earlier two:

• we’re using the same supplier and methodology and technology so we should be able to correct ‘mistakes’ we made previously
• we’ll re-use some functionality from earlier phases
• our content authors more likely to write their content in time – the web is always more motivating than an intranet!

On the otherhand:

• the web is much higher profile, with many more stakeholders and the look and feel will be a more significant issue
• accessibility could be even more challenging as we have to design for many different access technologies rather than the RNIB supported ones for staff.

Wish me luck!

One of my great hopes for our current intranet project is to significantly improve the intranet search.  The current set-up used the search bundled with Stellent. It is universally derided within the organisation and with good reason (the Stellent search itself may not be at fault, I imagine some changes to the configuration could fix some of the more significant problems).

I’ve heard mixed reports of Sharepoint search. Our suppliers are very positive about it, and it does seem hard to imagine how it could be worse that what we currently have.

At the TFPL conference I attended Sharon Richardson of Joining Dots defended SharePoint search. She went a bit far with the statement “…so the problem with search is not the technology, it’s the users” but there’s some interesting stuff in the ‘research‘ she referred to.

55% The content was badly named, didn’t contain the words the users was searching for, wasn’t easily identifiable in search results (e.g. if you have 2 results both called Cafe – which is for London and which is for Manchester?)

30% The content users were looking for didn’t exist

10% Users were using wide or strange search terms (why would somebody search for ‘google’ on the intranet? what exactly did they want to find when they searched for ‘form’?)

5% Search wasn’t finding appropriate content or ranking wasn’t appropriate

I’ve been keeping track of failed or problematic searches on our current intranet. Not particularly scientific but it has been an interesting starting point for evaluating the new search.

30% mismatches in language
25% inappropriate date ordering
15% lack of stemming
15% overly rigid phrase order matching
10% ambiguous queries
5% inappropriate alphabetic ordering of results

If a number of results are assigned the same relevancy then they are returned in date order, and if there are a number of results published on the same day then they are returned in alphabetical order. The relevancy scores don’t seem to distinguish between enough results, so the date and alpha ordering are regularly skewing the results.

The mismatched language and the ambiguous queries are sure to still be problems with the new search. I’m not going to endeavour to ‘fix the users’ here. There are plenty of solutions (best bets, related searches, faceted filters and synonym control) that we can utilise.

Interestingly my experiences with our existing search have suggested that searching for just ‘form’ can be an intelligent, considered tactic in less than ideal circumstances. If you are looking for the sickness form but you are not sure if it is actually called that (absence form, sick form etc) then searching for form and scanning the results can be your least worst option. Given our current search is pedantic in it’s insistence on exact phrase order, I find myself conducting single word searches far more often than usual.

Related posts
SharePoint search: Inside the Index book ‘review’
SharePoint search: some ranking factors

Most days it is a problem for the RNIB that most commercial IT software isn’t accessible. But there are some (very slight) perks.

One of my colleagues fends off lots of cold calls from all sorts of companies wanting to speak to various directors. The IT suppliers seem to be the easiest to get rid off as she politely asks them if the software is accessible (she usually has to explain why that’s important to the RNIB) and that seems to do the trick in getting rid of 90% of the sales folk.

RNIB training seems to include lots of gentle games and quizzes. Our sight loss training included a game that went as follows:

• Give participants the same information printed in different fonts sizes.
• Ask them to answer a question about the information
• Give the first person to answer correctly a metaphorical pat on the back (they need sweets here)

(it was more interesting than it sounds)

The idea being that you very quickly get what font sizes are easy to read. And therefore understand why all RNIB information is printed in 14 point and at first looks screamingly ‘loud’.

It is a little strange working for the same company as my big sister, Catherine.

The RNIB is a small enough organisation that,  in-spite of her working in a completely different part of the country, many of my colleagues in London know my sister. They say we look alike. They also say she’s told them all my secrets.

I’ve already had to go to the Leeds office for a meeting so got to stay with Cath and see her in her professional guise. All her colleagues said we look alike too.

I know we’re the same size but I’m not convinced we’re that identi-kit. She does like purple too though.

I had the second part of my RNIB induction last week. The first bit was  sight loss training whereas this part was more about corporate awareness.

As a result I now know that:

• the RNIB does not chug
• one of the bestselling RNIB products is a liquid level indicator which emits a tone when the cup is full
• three out of four blind and partially sighted people of working age are not in work
• nearly 30 per cent of RNIB’s income comes from legacies
• having the org structure explained doesn’t help understand it
• Sooty boxes raises over £850,000 each year
• the CEO knows about what I do

Guide dogs vomiting at work is not actually a new thing for me. That happened at the BBC too. The story became the stuff of office legend but at the RNIB it merits a pan-building email (admittedly to explain why the second floor had been without power):

“Dog of the Week – a new slot. So there’s Jimmy, lying under John’s desk, dreaming of chasing rabbits and sniffing “things” whilst John beavers away earning a crust. Sod this for a game of soldiers thinks Jimmy, I want to go home, and promptly throws up in the floor-box, thus short circuiting the electricity supply to the second floor. Nice one Jimmy – the lads in my Team only do that at Christmas.”