Libraryless. Click here for Pure Java version (2640L/17K).
1 | // clever index of tokenized strings |
2 | // maintaining one CI-sorted list per token position |
3 | sclass PositionalTokenIndex2 {
|
4 | new L<PositionalTokenIndex> indices; // index = token count |
5 | |
6 | *() {}
|
7 | *(Iterable<LS> toks) { addAll(toks); }
|
8 | |
9 | void addAll(Iterable<LS> toks) {
|
10 | fOr (LS tok : toks) |
11 | add(tok); |
12 | } |
13 | |
14 | void add(LS tok) {
|
15 | listGetOrCreate_full PositionalTokenIndex(indices, l(tok)/2).add(tok); |
16 | } |
17 | |
18 | // return all tokenizations with token t in position i |
19 | // having tokenCount code tokens |
20 | // may return null |
21 | LLS byToken(int i, S t, int tokenCount) {
|
22 | PositionalTokenIndex idx = get(indices, tokenCount); |
23 | ret idx == null ? null : idx.byToken(i, t); |
24 | } |
25 | |
26 | PositionalTokenIndex indexForTokenCount(int tokenCount) {
|
27 | ret get(indices, tokenCount); |
28 | } |
29 | |
30 | void clean() {
|
31 | for (PositionalTokenIndex idx : indices) idx.clean(); |
32 | } |
33 | } |
Began life as a copy of #1025688
download show line numbers debug dex old transpilations
Travelled to 6 computer(s): bhatertpkbcr, mqqgnosmbjvj, pyentgdyhuwx, pzhvpgtvlbxg, tvejysmllsmz, vouqrxazstgt
No comments. add comment
| Snippet ID: | #1025695 |
| Snippet name: | PositionalTokenIndex2 - PositionalTokenIndex grouped by token count [OK] |
| Eternal ID of this version: | #1025695/5 |
| Text MD5: | 7e6789bc590fbe1f2345c9c982b43993 |
| Transpilation MD5: | c18c7ce3c15822be879095443303bc2a |
| Author: | stefan |
| Category: | javax / agi.blue |
| Type: | JavaX fragment (include) |
| Public (visible to everyone): | Yes |
| Archived (hidden from active list): | No |
| Created/modified: | 2019-10-13 23:45:55 |
| Source code size: | 904 bytes / 33 lines |
| Pitched / IR pitched: | No / No |
| Views / Downloads: | 181 / 544 |
| Version history: | 4 change(s) |
| Referenced in: | [show references] |