Download Jar. Uses 3874K of libraries. Click here for Pure Java version (11185L/75K).
1 | !7 |
2 | |
3 | static LL<S> allTokens; |
4 | |
5 | p-experiment { |
6 | loadSnippetsDBForAnalysis(); |
7 | print("Tokenizing."); |
8 | allTokens = new L; |
9 | time { |
10 | for (CSnippet s) |
11 | if (nempty(s.text)) { |
12 | L<S> tokens = internAll(javaTok(s.text())); |
13 | allTokens.add(tokens); |
14 | s.text = tokens; |
15 | } |
16 | } |
17 | print(n2(totalLength(allTokens)) + " tokens"); |
18 | } |
Began life as a copy of #1015255
download show line numbers debug dex old transpilations
Travelled to 13 computer(s): aoiabmzegqzx, bhatertpkbcr, cbybwowwnfue, cfunsshuasjs, gwrvuhgaqvyk, ishqpsrjomds, lpdgvwnxivlt, mqqgnosmbjvj, pyentgdyhuwx, pzhvpgtvlbxg, tslmcundralx, tvejysmllsmz, vouqrxazstgt
No comments. add comment
Snippet ID: | #1015256 |
Snippet name: | Keep all snippets tokenized [smart version with string sharing] |
Eternal ID of this version: | #1015256/7 |
Text MD5: | ca1f9375a7a2590ec9ffcbc0255fc44f |
Transpilation MD5: | 7dbd41ff49a1400b58bfa2e347adaa38 |
Author: | stefan |
Category: | javax |
Type: | JavaX source code (desktop) |
Public (visible to everyone): | Yes |
Archived (hidden from active list): | No |
Created/modified: | 2018-05-11 03:34:02 |
Source code size: | 365 bytes / 18 lines |
Pitched / IR pitched: | No / No |
Views / Downloads: | 357 / 1006 |
Version history: | 6 change(s) |
Referenced in: | [show references] |