Download Jar. Uses 3874K of libraries. Click here for Pure Java version (11185L/75K).
!7 static LL<S> allTokens; p-experiment { loadSnippetsDBForAnalysis(); print("Tokenizing."); allTokens = new L; time { for (CSnippet s) if (nempty(s.text)) { L<S> tokens = internAll(javaTok(s.text())); allTokens.add(tokens); s.text = tokens; } } print(n2(totalLength(allTokens)) + " tokens"); }
Began life as a copy of #1015255
download show line numbers debug dex old transpilations
Travelled to 13 computer(s): aoiabmzegqzx, bhatertpkbcr, cbybwowwnfue, cfunsshuasjs, gwrvuhgaqvyk, ishqpsrjomds, lpdgvwnxivlt, mqqgnosmbjvj, pyentgdyhuwx, pzhvpgtvlbxg, tslmcundralx, tvejysmllsmz, vouqrxazstgt
No comments. add comment
| Snippet ID: | #1015256 |
| Snippet name: | Keep all snippets tokenized [smart version with string sharing] |
| Eternal ID of this version: | #1015256/7 |
| Text MD5: | ca1f9375a7a2590ec9ffcbc0255fc44f |
| Transpilation MD5: | 7dbd41ff49a1400b58bfa2e347adaa38 |
| Author: | stefan |
| Category: | javax |
| Type: | JavaX source code (desktop) |
| Public (visible to everyone): | Yes |
| Archived (hidden from active list): | No |
| Created/modified: | 2018-05-11 03:34:02 |
| Source code size: | 365 bytes / 18 lines |
| Pitched / IR pitched: | No / No |
| Views / Downloads: | 358 / 1008 |
| Version history: | 6 change(s) |
| Referenced in: | #1015259 - Keep all snippets tokenized [v3 with preloading] #3000526 - Smart Bot's answer to: !fresh latestPrograms() |