From 997551241fd9433af31b78aca727039ec114bd17 Mon Sep 17 00:00:00 2001 From: Matthew Honnibal Date: Mon, 7 Jul 2014 05:09:22 +0200 Subject: [PATCH] * Upd ptb tokenization rules --- data/en_ptb/tokenization | 165 +++++++++++++++++++++------------------ 1 file changed, 88 insertions(+), 77 deletions(-) diff --git a/data/en_ptb/tokenization b/data/en_ptb/tokenization index 64c9525fa..0895068f7 100644 --- a/data/en_ptb/tokenization +++ b/data/en_ptb/tokenization @@ -4,90 +4,101 @@ #*---* --- #*'s 's -ain't are not -aren't are not -can't can not -could've could have -couldn't could not -couldn't've could not have -didn't did not -doesn't does not -don't do not -hadn't had not -hadn't've had not have -hasn't has not -haven't have not -he'd he would -he'd've he would have -he'll he will +cannot can not +d'ye d' ye +gimme gim me +gonna gon na +lemme lem me +more'n more 'n +'tis 't is +'twas 't was +wanna wan na +whaddya wha dd ya +whatcha wha t cha +ain't ai n't +aren't are n't +can't can n't +could've could 've +couldn't could n't +couldn't've could n't 've +didn't did n't +doesn't does n't +don't do n't +hadn't had n't +hadn't've had n't 've +hasn't has n't +haven't have n't +he'd he 'd +he'd've he 'd 've +he'll he 'll he's he 's -how'd he would -how'll he will +how'd he 'd +how'll he 'll how's how 's -I'd I would -I'd've I would have -I'll I will -I'm I am -I've I have -isn't is not -it'd it would -it'd've it would have -it'll it will +I'd I 'd +I'd've I 'd 've +I'll I 'll +I'm I 'm +I've I 've +isn't is n't +it'd it 'd +it'd've it 'd 've +it'll it 'll it's it 's let's let 's -mightn't might not -mightn't've might not have -might've might have -mustn't must not -must've must have -needn't need not -not've not have -shan't shall not -she'd she would -she'd've she would have -she'll she will +mightn't might n't +mightn't've might n't 've +might've might 've +mustn't must n't +must've must 've +needn't need n't +not've not h've +shan't sha n't +she'd she 'd +she'd've she 'd 've +she'll she 'll she's she 's -should've should have -shouldn't should not -shouldn't've should not have +should've should 've +shouldn't should n't +shouldn't've should n't 've that's that 's -there'd there would -there'd've there would have -there's there is -they'd there would -they'd've they would have -they'll they will -they're they are -they've they have -wasn't was not -we'd we would -we'd've we would have -we'll we will -we're we are -we've we have -weren't were not -what'll what will -what're what are +there'd there 'd +there'd've there 'd 've +there's there 's +they'd there 'd +they'd've they 'd 've +they'll they 'll +they're they 're +they've they 've +wasn't was n't +we'd we 'd +we'd've we 'd h've +we'll we 'll +we're we 're +we've we h've +weren't were n't +what'll what 'll +what're what 're what's what 's -what've what have +what've what 've when's when 's -where'd where would +where'd where 'd where's where 's -where've where have -who'd who would -who'll who will -who're who are +where've where 've +who'd who 'd +who'll who 'll +who're who 're who's who 's -who've who have -why'll who will -why're why are -why's why is -won't will not -would've would have -wouldn't would not -wouldn't've would not have -you'd you would -you'd've you would have -you'll you will -you're you are -you've you have +who've who 've +why'll why 'll +why're why 're +why's why 's +won't will n't +would've would 've +wouldn't would n't +wouldn't've would n't 've +you'd you 'd +you'd've you 'd 've +you'll you 'll +you're you 're +you've you 've