Fred Truman
Design & Code Creative
Learning Bit by Bit – Week 03: Tokenizer
February 18, 2011
For this week’s Learning Bit by Bit assignment we were asked to program a stop tokenizer to normalize an input sentence as if it was being passed to a search engine – and to think about what makes a good stop list. We talked a bit about this in class last week, so I thought I’d being by looking at what a typical search engine’s stop list might look like. Not sure how accurate this is as it seems a little aggressive, but here’s an idea of what a typical search engine’s stop list might look like. I’m not sure if I would include words like “greetings” or words that specify locations like “fifth” or “underneath” – but what do I know. What if you searched for “animals underneath the sea” and you found “animals above the sea”?? What then? The humanity.
I find it particularly hysterical after searching tirelessly through support forums for technical issues I will inevitably resort to searching with aggression: “Just tell me how to get the f*cking jquery plugin to work!” I usually do this to lighten my mood and remind myself of the absurdity of my predicament, but sometimes I mimic other people’s frustration that they’ve posted to a forum so well that it turns out to be a good lead. So I find it interesting that this stop list has words like “necessary” but not “f*cking”. Maybe being an emphatic swarthy pirate searcher has its benefits.
To test this out, I brought in this stop list and tried some searches. The class with the stop list plopped in with some punctuation added to the beginning and I’ve included it in. Otherwise it’s literally just the example from the book:
I tried out:
Who's the best pirate? START END TOKEN 15 21 |pirate|
Not so good, huh? Why would you want to include qualitative statements in the stop list? I’m not sure. But, if we try:
Just tell me how to get the f*cking jquery plugin to work! START END TOKEN 28 35 |f*cking| 36 42 |jquery| 43 49 |plugin| 53 57 |work| 57 58 |!|
I’m really not sure what I’ve achieved, if anything…
[image credit: testpattern]