Python Text Processing - Munging

Munging in general means cleaning up anything messy by transforming them. In our case we will see how we can transform text to get some result which gives us some desirable changes to data. At a simple level it is only about transforming the text we are dealing with.

Example - Munging

In the below example we plan to shuffle and then rearrange all the letters of a sentence except the first and the last one to get the possible alternate words which may get generated as a mis-spelled word during writing by a human. This rearrangement helps us in

main.py

import random

import re

def replace(t):
    inner_word = list(t.group(2))
    random.shuffle(inner_word)
    return t.group(1) + "".join(inner_word) + t.group(3)
text = "Hello, You should reach the finish line."
print(re.sub(r"(\w)(\w+)(\w)", replace, text))

print(re.sub(r"(\w)(\w+)(\w)", replace, text))

Output

When we run the above program we get the following output −

Hlleo, You slohud recah the fniish line.
Hello, You soulhd reach the fniish line.

Here you can see how the words are jumbled except for the first and the last letters. By taking a statistical approach to wrong spelling we can decided what are the commonly misspelled words and supply the correct spelling for them.