Keep all snippets tokenized [smart version with string sharing] [1015256]

18
LINES

< > BotCompany Repo | #1015256 // Keep all snippets tokenized [smart version with string sharing]

JavaX source code (desktop) [tags: use-pretranspiled] - run with: x30.jar

Download Jar. Uses 3874K of libraries. Click here for Pure Java version (11185L/75K).

!7

static LL<S> allTokens;

p-experiment {
  loadSnippetsDBForAnalysis();
  print("Tokenizing.");
  allTokens = new L;
  time {
    for (CSnippet s)
      if (nempty(s.text)) {
        L<S> tokens = internAll(javaTok(s.text()));
        allTokens.add(tokens);
        s.text = tokens;
      }
  }
  print(n2(totalLength(allTokens)) + " tokens");
}

Author comment

Began life as a copy of #1015255

download show line numbers debug dex old transpilations

Travelled to 13 computer(s): aoiabmzegqzx, bhatertpkbcr, cbybwowwnfue, cfunsshuasjs, gwrvuhgaqvyk, ishqpsrjomds, lpdgvwnxivlt, mqqgnosmbjvj, pyentgdyhuwx, pzhvpgtvlbxg, tslmcundralx, tvejysmllsmz, vouqrxazstgt

No comments. add comment

Snippet ID:	#1015256
Snippet name:	Keep all snippets tokenized [smart version with string sharing]
Eternal ID of this version:	#1015256/7
Text MD5:	ca1f9375a7a2590ec9ffcbc0255fc44f
Transpilation MD5:	7dbd41ff49a1400b58bfa2e347adaa38
Author:	stefan
Category:	javax
Type:	JavaX source code (desktop)
Public (visible to everyone):	Yes
Archived (hidden from active list):	No
Created/modified:	2018-05-11 03:34:02
Source code size:	365 bytes / 18 lines
Pitched / IR pitched:	No / No
Views / Downloads:	245 / 724
Version history:	6 change(s)
Referenced in:	[show references]

1	!7
2
3	static LL<S> allTokens;
4
5	p-experiment {
6	loadSnippetsDBForAnalysis();
7	print("Tokenizing.");
8	allTokens = new L;
9	time {
10	for (CSnippet s)
11	if (nempty(s.text)) {
12	L<S> tokens = internAll(javaTok(s.text()));
13	allTokens.add(tokens);
14	s.text = tokens;
15	}
16	}
17	print(n2(totalLength(allTokens)) + " tokens");
18	}