Rooba
/
spaCy
mirror of https://github.com/explosion/spaCy.git
1
1
Fork 0
Code Issues Packages Projects Releases Wiki Activity
3,816 Commits 70 Branches 177 Tags 298 MiB
Commit Graph

415 Commits

Author SHA1 Message Date
Matthew Honnibal 04d0686b26 * Make TransitionSystem.add_action idempotent, i.e. ignore duplicate added actions. 2016-01-19 20:10:04 +01:00
Matthew Honnibal 65c5bc4988 * Add add_label method, to allow users to register new entity types and dependency labels. 2016-01-19 19:11:02 +01:00
Matthew Honnibal 151aa0b0e2 * Allow users to add_label, in order to extend the entity recogniser to new classes. Does not by itself add a class to the model 2016-01-19 19:09:33 +01:00
Matthew Honnibal c8e0011ebc * Add iterators to the NER and parser transition systems, to get the action types 2016-01-19 19:07:43 +01:00
Matthew Honnibal 04177debd0 * Unwind limit to sentence boundary detection that prevents it from inserting boundaries on whitespace. Replace it with a check for whitespace in StateClass.fast_forward, so that whitespace is LeftArced when it's on the stack. This should prevent the previous problem of whitespace-only sentences. Should fix Issue #184, but may cause further problems. Needs testing. 2016-01-19 02:54:15 +01:00
Matthew Honnibal 3dc398b727 * Fix merge conflict in requirements.txt 2016-01-16 16:20:49 +01:00
Matthew Honnibal c025a0c64b * Check for KeyboardInerrupt in parser.__call__ 2016-01-16 16:18:44 +01:00
Matthew Honnibal aec130af56 Use util.Package class for io 
Previous Sputnik integration caused API change: Vocab, Tagger, etc
were loaded via a from_package classmethod, that required a
sputnik.Package instance. This forced users to first create a
sputnik.Sputnik() instance, in order to acquire a Package via
sp.pool().

Instead I've created a small file-system shim, util.Package, which
allows classes to have a .load() classmethod, that accepts either
util.Package objects, or strings. We can later gut the internals
of this and make it a proxy for Sputnik if we need more functionality
that should live in the Sputnik library.

Sputnik is now only used to download and install the data, in
spacy.en.download
 2015-12-29 18:00:48 +01:00
Matthew Honnibal 5623242b3e * Adjust NER rules, so that U entries in gazetteer don't become B moves to the model 2015-11-12 04:48:23 +11:00
Matthew Honnibal 44fbdc7260 * Fix bug in NER transition system, that sometimes left no valid moves 2015-11-08 16:19:12 +01:00
Matthew Honnibal e92371bb54 * Fix rule that made Last action invalid if there was a preset of O, since if the entity is already open, that ship has sailed. 2015-11-08 22:17:51 +11:00
Matthew Honnibal 6f47074214 * Make constructor of ParserModel and TaggerModel the same as AveragedPerceptron, for each pickling. 2015-11-07 18:25:17 +11:00
Matthew Honnibal 1cfa20fb17 * Fix sentence-final whitespace issue 2015-11-07 17:34:46 +11:00
Matthew Honnibal 888c05a7fa * Fix variable naming in StepwiseState, for thinc 4.0 2015-11-07 11:02:44 +11:00
Matthew Honnibal fc2185bfe3 * Fix variable naming in StepwiseState, for thinc 4.0 2015-11-07 10:48:31 +11:00
Matthew Honnibal 954442a807 * Fix variable naming in StepwiseState, for thinc 4.0 2015-11-07 10:30:45 +11:00
Matthew Honnibal af70dc166a * Fix Last restriction, that was supposed to prevent conflicts with presets, but was incorrect. 2015-11-07 09:52:00 +11:00
Matthew Honnibal a06e3c8963 * Fix bone-headed mistake in StateClass.E 2015-11-07 07:35:28 +11:00
Matthew Honnibal d24b8509e4 * Correct screw ups from the previous commits 2015-11-07 06:51:41 +11:00
Matthew Honnibal 5efad178b5 * Set ent tag when close entity 2015-11-07 06:09:25 +11:00
Matthew Honnibal 9285f01d26 * Fix broken StateClass.E tracking 2015-11-07 06:06:39 +11:00
Matthew Honnibal 19136b0e7d * Add better debug message for illegal move 2015-11-07 05:34:37 +11:00
Matthew Honnibal 2733816b7b * Fix whitespace 2015-11-07 05:31:06 +11:00
Matthew Honnibal 01ab464383 * Prevent Begin and In moves from applying in NER if we're at the last token of a sentence, as this would mean the entity would span over a sentence boundary. Re Issue #169 2015-11-07 05:30:44 +11:00
Matthew Honnibal b65633f270 * Fix function that returns nth entity in StateClass. Was only returning the first. 2015-11-07 05:29:11 +11:00
Matthew Honnibal 3c162dcac3 * Refactor away from the _ml module, to use thinc 4.0. Still some work needs to be done, e.g. to add __reduce__ to the models, more testing, etc. 2015-11-07 03:24:30 +11:00
Matthew Honnibal b9991fbd20 * Update to use thinc 3.0 2015-11-06 00:25:59 +11:00
Matthew Honnibal 68f479e821 * Rename Doc.data to Doc.c 2015-11-04 00:15:14 +11:00
Matthew Honnibal 329ae57520 * Fix whitespace attachment thing 2015-10-13 09:46:38 +02:00
Matthew Honnibal 37919eac82 * Fix whitespace attachment in simpler way. Leaves problem with setting left/right children. 2015-10-13 18:23:24 +11:00
Matthew Honnibal c70eb776ae * Fix whitespace attachment, so that left/right children are consistent with head. 2015-10-13 15:58:22 +11:00
Matthew Honnibal 20fd36a0f7 * Very scrappy, likely buggy first-cut pickle implementation, to work on Issue #125: allow pickle for Apache Spark. The current implementation sends stuff to temp files, and does almost nothing to ensure all modifiable state is actually preserved. The Language() instance is a deep tree of extension objects, and if pickling during training, some of the C-data state is hard to preserve. 2015-10-13 13:44:41 +11:00
Matthew Honnibal 9dd2f25c74 * Fix Issue #131: Force whitespace characters to attach syntactically to previous token, and ensure they cannot serve as stand-alone 'sentence' units. 2015-10-10 15:53:30 +11:00
Matthew Honnibal 8b39feefbe * Add dependency post-process rule to ensure spaces are attached to neighbouring tokens, so that they can't be sentence boundaries 2015-10-10 15:32:13 +11:00
Matthew Honnibal 0e24d099a1 * Fix L/R edge bug, by ensuring l_edge and r_edge are preset, and fixing the way the edge update in del_arc. Bugs keep arising here because the edges are absolute positions, where everything else is relative. I'm also not 100% convinced that del_arc is handled correctly. Do we need to update the parents? 2015-09-09 03:40:44 +02:00
Matthew Honnibal 86c888667f * Merge in changes from de branch 2015-09-06 19:49:28 +02:00
Matthew Honnibal 5edac11225 * Wrap self.parse in nogil, and break if an invalid move is predicted. The invalid break is a work-around that papers over likely bugs, but we can't easily break in the nogil block, and otherwise we'll get an infinite loop. Need to set this as an error flag. 2015-09-06 04:15:00 +02:00
Matthew Honnibal a3d5e6c0dd * Reform constructor and save/load workflow in parser model 2015-08-26 19:19:01 +02:00
Matthew Honnibal bf38b3b883 * Hack on l/r reversal bug 2015-08-10 05:58:43 +02:00
Matthew Honnibal 6116413b47 * Fix label prediction in StepwiseState 2015-08-10 05:05:31 +02:00
Matthew Honnibal 2c9753eff2 * Whitespace 2015-08-10 00:09:02 +02:00
Matthew Honnibal 9de98f5a6f * Add Parser.stepthrough method, with context manager 2015-08-10 00:08:46 +02:00
Matthew Honnibal fe43f8cf39 * Whitespace 2015-08-09 02:31:53 +02:00
Matthew Honnibal 9c090945e0 * Add Parser.predict method, and clean up Parser.get_state 2015-08-09 02:29:58 +02:00
Matthew Honnibal 04fccfb984 * Fix get_state for parser prediction 2015-08-09 02:11:22 +02:00
Matthew Honnibal 55fde0e240 * Fix get_state 2015-08-09 01:45:30 +02:00
Matthew Honnibal f0f4fa9838 * Fix Parser.get_state 2015-08-09 01:40:13 +02:00
Matthew Honnibal 18331dca89 * Add continue_for argument to parser 'partial' function, which is now renamed to get_state 2015-08-09 01:31:54 +02:00
Matthew Honnibal 0653288fa5 * Fix stateclass.queue 2015-08-09 00:39:02 +02:00
Matthew Honnibal 9de218b7ba * Fix Parser.partial function 2015-08-08 23:45:18 +02:00
Matthew Honnibal cc9deae960 * Add is_valid method to transition_system 2015-08-08 23:36:18 +02:00
Matthew Honnibal 2a46c77324 * Whitespace 2015-08-08 23:35:59 +02:00
Matthew Honnibal 7bafc789e7 * Add stack and queue properties to stateclass, for python access 2015-08-08 23:32:42 +02:00
Matthew Honnibal 3af938365f * Add function partial to Parser 2015-08-08 23:32:15 +02:00
Matthew Honnibal 76a1f0481a * Whitespace 2015-08-08 23:31:54 +02:00
Matthew Honnibal 59c3bf60a6 * Ensure entity recognizer doesn't over-write preset types 2015-08-06 16:09:08 +02:00
Matthew Honnibal 9c1724ecae * Gazetteer stuff working, now need to wire up to API 2015-08-06 00:35:40 +02:00
Matthew Honnibal a8bbd7312c * Hackishly patch long dependencies problem 2015-07-28 00:14:29 +02:00
Matthew Honnibal bb583f7f09 * Hackishly patch long dependencies problem 2015-07-27 23:14:33 +02:00
Matthew Honnibal 823ef4a00b * Remove profile declarations 2015-07-25 18:13:06 +02:00
Matthew Honnibal aa28e2e01d * Release the GIL around parse function 2015-07-24 04:53:27 +02:00
Matthew Honnibal d5255aad77 * Update freqs for missing tags in ner, for serializer 2015-07-23 01:17:11 +02:00
Matthew Honnibal 12699a1152 * Set initial freqs, to avoid missing values in serializer 2015-07-23 01:16:27 +02:00
Matthew Honnibal 317cbbc015 * Serialization round trip now working with decent API, but with rough spots in the organisation and requiring vocabulary to be fixed ahead of time. 2015-07-19 15:18:17 +02:00
Matthew Honnibal b1d74ce60d * Remove unused joint.pyx and joint.pxd files 2015-07-17 23:31:44 +02:00
Matthew Honnibal fb0a641a2d * Don't release the gil around Parser.parse. Does this indicate thread problems? 2015-07-17 23:07:37 +02:00
Matthew Honnibal e29daea85f * Fix bint/int typing problem in TransitionSystem. In C++ bint* means bool*, but in C it means int*. So, type-casting to bint* is unsafe. 2015-07-17 22:37:24 +02:00
Matthew Honnibal 45ae1ce428 * Remove unused declaration in parser 2015-07-16 01:27:11 +02:00
Matthew Honnibal 9a8db9743c * Remove gil from parser.call 2015-07-14 23:47:33 +02:00
Matthew Honnibal 38ca0c33f5 Merge branch 'neuralnet' into refactor 
Mostly refactors parser, to use new thinc3.2 Example class.
Aim is to remove use of shared memory, so that we can parallelize
over documents easily.

Conflicts:
	setup.py
	spacy/syntax/parser.pxd
	spacy/syntax/parser.pyx
	spacy/syntax/stateclass.pyx
 2015-07-14 14:13:47 +02:00
Matthew Honnibal 6eef0bf9ab * Break up tokens.pyx into tokens/doc.pyx, tokens/token.pyx, tokens/spans.pyx 2015-07-13 20:20:58 +02:00
Matthew Honnibal 55f1042443 * Improve efficiency of L and R features, correcting the non-linear-in-length problem. 2015-07-09 12:17:26 +02:00
Matthew Honnibal 70d2acb579 * Fix edge features 2015-07-09 12:15:01 +02:00
Matthew Honnibal adb868bdad * Add warning for models not found in parser 2015-07-08 20:04:55 +02:00
Matthew Honnibal 05b28ec9eb * Add warning for models not found in parser 2015-07-08 20:02:13 +02:00
Matthew Honnibal ef700401a6 * Add warning for models not found in parser 2015-07-08 20:00:46 +02:00
Matthew Honnibal 6218d8b389 * Add warning for models not found in parser 2015-07-08 19:59:16 +02:00
Matthew Honnibal f6a6c39ce8 * Add warning for models not found in parser 2015-07-08 19:52:30 +02:00
Matthew Honnibal 0ceb1f71c2 * Update parse features 2015-07-08 19:11:36 +02:00
Matthew Honnibal bb522496dd * Rename Tokens to Doc 2015-07-08 18:53:00 +02:00
Matthew Honnibal ff885e8511 * Add ParserFactory convenience function 2015-07-08 12:35:46 +02:00
Matthew Honnibal 52fd80c6c6 * Add experimental supersense features for parsing, based on lookup into wordnet. 2015-07-01 20:12:44 +02:00
Matthew Honnibal e20106fdff * Begin reorganizing neuralnet work 2015-06-30 14:26:32 +02:00
Matthew Honnibal 3bb5876c5a * Inline methods in StateClass 2015-06-29 01:10:14 +02:00
Matthew Honnibal 313a7f87b3 * Inline methods in StateClass 2015-06-29 01:06:28 +02:00
Matthew Honnibal a02fd3af5d * Check valency in L and R feature methods, to make feaure calculation faster 2015-06-29 00:27:56 +02:00
Matthew Honnibal 5d870720bc * Check valency in L and R feature methods, to make feaure calculation faster 2015-06-29 00:17:29 +02:00
Matthew Honnibal f4986d5d3c * Use new Example class 2015-06-28 22:36:03 +02:00
Matthew Honnibal 735f1af91f * Fix neural net stuff 2015-06-28 11:44:58 +02:00
Matthew Honnibal e7003f1cf3 * Remove hard-coding of vector lengths 2015-06-28 11:37:17 +02:00
Matthew Honnibal 897dd0dd0b * Merge changes, and adjust Example to use memoryview 2015-06-28 11:36:11 +02:00
Matthew Honnibal 9282a8e72c * Prepare for new models to be plugged in by using Example class 2015-06-28 11:02:35 +02:00
Matthew Honnibal 75aeccc064 * Rejig parser interface to use new thinc.api.Example class, in prep of theano model. Comment out beam search 2015-06-28 11:02:34 +02:00
Matthew Honnibal bbef71f213 * Fix min function in fill_context 2015-06-28 10:46:39 +02:00
Matthew Honnibal 142b6f9510 * Revert last changes 2015-06-28 10:44:28 +02:00
Matthew Honnibal b06962f18b * Pad buffers in state 2015-06-28 10:36:14 +02:00
Matthew Honnibal 53be72387c * Hack at fill_context to investigate performance loss 2015-06-28 10:34:28 +02:00
Matthew Honnibal 71a4e876a9 * Fix parse features 2015-06-28 09:27:33 +02:00
Matthew Honnibal 5af500909c * Remove unused directve from parser.pyx 2015-06-28 06:20:21 +02:00
Matthew Honnibal d5b4090705 * Add profile directive 2015-06-28 06:19:33 +02:00
Matthew Honnibal 2b5421e60c * Add profile directive 2015-06-28 06:07:04 +02:00
Matthew Honnibal 8b5de4a411 * Add word / tag / label sets, for use in neural net 2015-06-28 05:46:53 +02:00
Matthew Honnibal ed40a8380e * Remove hard-coding of vector lengths 2015-06-27 04:18:47 +02:00
Matthew Honnibal ebe630cc8d * Enable more features for NN 2015-06-27 04:17:29 +02:00
Matthew Honnibal f8bb43475e * Bridge to Theano working. Very disorganised. Using thinc adb60aba966ed2 2015-06-27 02:39:18 +02:00
Matthew Honnibal 2fe98b8a9a * Prepare for new models to be plugged in by using Example class 2015-06-26 13:51:39 +02:00
Matthew Honnibal 6896455884 * Rejig parser interface to use new thinc.api.Example class, in prep of theano model. Comment out beam search 2015-06-26 06:25:36 +02:00
Matthew Honnibal 02b171ee67 * Bug fixes to edge calculation 2015-06-24 04:28:02 +02:00
Matthew Honnibal 7f9384f53c * Remove deprecated _state module 2015-06-23 17:28:24 +02:00
Matthew Honnibal 6dbe182491 * Fix merge conflicts 2015-06-23 17:28:00 +02:00
Matthew Honnibal 579735a095 * Remove import of _state module 2015-06-23 17:25:08 +02:00
Matthew Honnibal 88f55d136b * Remove deprecated _state module 2015-06-23 17:19:51 +02:00
Matthew Honnibal 9ab9dd2bf7 * Clean up unused orig_arc_eager and tree_arc_eager modules, which were only added for EMNLP experiments 2015-06-23 17:17:33 +02:00
Matthew Honnibal 7ebfe4b983 * Fixes to edge features 2015-06-23 16:32:54 +02:00
Matthew Honnibal 7b125f5a86 * Fixes to edge features 2015-06-23 16:31:01 +02:00
Matthew Honnibal 35c290bee4 * Fix edge features 2015-06-23 15:50:56 +02:00
Matthew Honnibal 221e2e485f * Assign 'ROOT' as label, not 'root' 2015-06-23 15:09:54 +02:00
Matthew Honnibal a7bf7b0626 * Rename sent_start to sent_end, to reflect its new usage in the Break transition 2015-06-23 05:39:43 +02:00
Matthew Honnibal ee3e56f27b * Fix bounds checking on entities 2015-06-23 04:35:08 +02:00
Matthew Honnibal 43ef5ddea5 * Ensure root albel is spelled ROOT, for backwards compatibility 2015-06-23 04:14:03 +02:00
Matthew Honnibal 065c2e1d2d * Add some bounds checking around state arrays 2015-06-23 04:13:09 +02:00
Matthew Honnibal f01b3d043e * Add padding to arrays in stateclass. May be papering over a deeper bug. 2015-06-23 03:03:41 +02:00
Matthew Honnibal 69507bc729 * Re-enable Break transition in arc_eager.pyx 2015-06-23 00:03:30 +02:00
Matthew Honnibal ab110be125 * Remove debugging in parser.pyx 2015-06-16 23:37:25 +02:00
Matthew Honnibal 9b13d11ab3 * Fix handling of entities in StateClass 2015-06-16 23:35:21 +02:00
Matthew Honnibal c40a2c661c * Add tree_arc_eager 2015-06-15 08:23:24 +02:00
Matthew Honnibal 5da5cf7084 * Add some more features for S1/S0 2015-06-15 04:07:13 +02:00
Matthew Honnibal 8156a01bca * Fix root label for orig_arc_eager 2015-06-15 02:54:55 +02:00
Matthew Honnibal 21930ede15 * Switch toggle on USE_ROOT_ARC_SEGMENT 2015-06-15 02:54:32 +02:00
Matthew Honnibal 38a6afa484 * Make possibly dubious correction to the unshift oracle 2015-06-15 02:50:00 +02:00
Matthew Honnibal f66228f253 * Add some more features, esp for labels 2015-06-14 21:18:02 +02:00
Matthew Honnibal 3da8e0f317 * Add orig_arc_eager 2015-06-14 20:31:44 +02:00
Matthew Honnibal ea8a103007 * Fix import of TransitionSystem in parser.pyx 2015-06-14 19:01:26 +02:00
Matthew Honnibal e0984ca139 * Fix valency features in StateClass 2015-06-14 17:50:26 +02:00
Matthew Honnibal 763cbd23d5 * Upd stateclass.print_state 2015-06-14 17:44:29 +02:00
Matthew Honnibal bdd07bf000 * Fix Break oracle, but disable the Break transition for now, while we finalize the gold-standard experiments 2015-06-14 17:44:03 +02:00
Matthew Honnibal 399f15fbdf * Add flag to toggle handling of multi-root inputs without the Break transition. Clear up now unused best_valid stuff. 2015-06-14 00:28:37 +02:00
Matthew Honnibal 75289b4761 * Don't refuse to parse single token sentences, incase some transition system needs them, e.g. single word entity. Instead fix error in _init_state. 2015-06-13 22:55:55 +02:00
Matthew Honnibal 77d7e79c7e * Fix r/l and distance features. 2015-06-12 13:06:15 +02:00
Matthew Honnibal 15e177d7a1 * Fixes to unshift/fast-forward strategy. Getting 91.55 greedy on NW dev, gold preproc 2015-06-12 01:50:23 +02:00
Matthew Honnibal afd77a529b * Prepare for break transition, with fast-forwarding. 86.5 on 1k nw gold preproc 2015-06-10 14:08:30 +02:00
Matthew Honnibal 495f528709 * Add support for sentence breaks in stateclass 2015-06-10 12:34:28 +02:00
Matthew Honnibal b7b18c279d * Fix Reduce oracle. Getting 86.35 2015-06-10 11:33:39 +02:00
Matthew Honnibal bb09b5d91a * Fix shifted bit vector in stateclass --- should reflect whether the word has been *unshifted*. 2015-06-10 11:33:09 +02:00
Matthew Honnibal aa9625f688 * Do non-monotonic Unshift. Every word can be shifted at most 1 time. When the Reduce move is used, if S0 has no head, we put the word back on the buffer. Gets 86.4 on nw 1k with gold pre-proc. Break transition not yet implemented for this. 2015-06-10 10:15:56 +02:00
Matthew Honnibal 7bf6b7de3e * Add unshift action to StateClass, and track which moves have been shifted 2015-06-10 10:13:03 +02:00
Matthew Honnibal f7c8069e65 * Fix bug in distance feature 2015-06-10 10:12:17 +02:00
Matthew Honnibal abd07c067a * Inline B and S methods on stateclass 2015-06-10 07:22:33 +02:00
Matthew Honnibal e2f9a80713 * Remove old _state imports 2015-06-10 07:09:17 +02:00
Matthew Honnibal e9aaecc619 * Remove from_struct method from StateClass 2015-06-10 06:58:27 +02:00