Sometimes your tokenization process is so complex that cannot be captured by a simple regular expression that you can pass to the scikit-learn TfidfVectorizer. Instead you just want to pass a list of tokens, resulting of a tokenization process, to initialize a TfidfVectorizer object.

There are manly two things that need to be done. First, in the initialization of the TfidfVectorizer object you need to pass a dummy tokenizer and preprocessor that simply return what they receive. Note, you can instead of a dummy_fun also pass a lambda function, e.g.: lambda x: x, but be aware that if you then want to use the cool n_jobs=10 for training classifiers or doing parameter grid search pickle cannot handle lambda functions.

from sklearn.feature_extraction.text import TfidfVectorizer

def dummy_fun(doc):
return doc

tfidf = TfidfVectorizer(
analyzer='word',
tokenizer=dummy_fun,
preprocessor=dummy_fun,
token_pattern=None)


Then you can fit a collections of documents already tokenized

docs = [
['Two', 'wrongs', 'don\'t', 'make', 'a', 'right', '.'],
['The', 'pen', 'is', 'mightier', 'than', 'the', 'sword'],
['Don\'t', 'put', 'all', 'your', 'eggs', 'in', 'one', 'basket', '.']
]

tfidf.fit(docs)
tfidf.vocabulary_
{'.': 0,
'Don\'t': 1,
'The': 2,
'Two': 3,
'a': 4,
'all': 5,
'don\'t': 7,
'eggs': 8,
'in': 9,
'is': 10,
'make': 11,
'mightier': 12,
'one': 13,
'pen': 14,
'put': 15,
'right': 16,
'sword': 17,
'than': 18,
'the': 19,
'wrongs': 20,
'your': 21}


The next thing to keep in mind is that whenever you want to compute the tf-idf score for a document that is already tokenized you should wrap it in a list when you call the transform() method from TfidfVectorizer, so that it is handled as a single document instead of interpreting each token as a document.

doc = ['Don\'t', 'count', 'your', 'chickens', 'before', 'they', 'hatch']

vector_1 = tfidf.transform(doc)
vector_2 = tfidf.transform([doc])
vector_1.shape
(7, 22)
vector_2.shape
(1, 22)