< > BotCompany Repo | #1015256 // Keep all snippets tokenized [smart version with string sharing]

JavaX source code (desktop) [tags: use-pretranspiled] - run with: x30.jar

Download Jar. Uses 3874K of libraries. Click here for Pure Java version (11185L/75K).

!7

static LL<S> allTokens;

p-experiment {
  loadSnippetsDBForAnalysis();
  print("Tokenizing.");
  allTokens = new L;
  time {
    for (CSnippet s)
      if (nempty(s.text)) {
        L<S> tokens = internAll(javaTok(s.text()));
        allTokens.add(tokens);
        s.text = tokens;
      }
  }
  print(n2(totalLength(allTokens)) + " tokens");
}

Author comment

Began life as a copy of #1015255

download  show line numbers  debug dex  old transpilations   

Travelled to 13 computer(s): aoiabmzegqzx, bhatertpkbcr, cbybwowwnfue, cfunsshuasjs, gwrvuhgaqvyk, ishqpsrjomds, lpdgvwnxivlt, mqqgnosmbjvj, pyentgdyhuwx, pzhvpgtvlbxg, tslmcundralx, tvejysmllsmz, vouqrxazstgt

No comments. add comment