base/str.cc:
Fix tokenize so that it doesn't behave incorrectly when there
are empty strings.
test/tokentest.cc:
Clean up the test function so it's easier to see what's going on
--HG--
extra : convert_revision : c7a3db7bc516d3575b1cc4ab7afbd0f1fbe1ec6f
