Commit Graph

14 Commits

Author SHA1 Message Date
Ines Montani 4e95737c6c Add base tag map 2016-12-18 16:54:28 +01:00
Ines Montani 2b2ea8ca11 Reorganise language data 2016-12-18 16:54:19 +01:00
Ines Montani bc40dad7d9 Add entity rules 2016-12-18 15:36:53 +01:00
Ines Montani eaa3b1319d Fix formatting 2016-12-18 15:36:53 +01:00
Ines Montani 62655fd36f Add ENT_ID constant 2016-12-18 15:36:53 +01:00
Ines Montani f324311249 Add global language data utils 2016-12-17 12:27:41 +01:00
Ines Montani e47ee94761 Split punctuation into its own file 2016-12-08 19:46:43 +01:00
Ines Montani e8ae588be9 Add emoticons 2016-12-08 19:45:18 +01:00
Ines Montani 5908c0ed9f Fix formatting 2016-12-08 19:45:11 +01:00
Ines Montani 0d07d7fc80 Apply emoticon exceptions to tokenizer 2016-12-07 21:11:59 +01:00
Ines Montani 9413bcd9ee Declare encoding and unicode literals 2016-12-07 21:10:34 +01:00
Ines Montani a280ff2657 Fix __all__ 2016-12-07 21:10:12 +01:00
Ines Montani ba8721953c Add missing emoticons 2016-12-07 21:09:44 +01:00
Ines Montani 79dce0aabe Add emoticons 2016-12-07 20:33:28 +01:00