Not logged in.  Login/Logout/Register | List snippets | | Create snippet | Upload image | Upload data

16
LINES

< > BotCompany Repo | #1015262 // tokenizedSnippetDB

JavaX fragment (include)

1  
static LL<S> allTokens;

2  


3  
svoid tokenizedSnippetDB() {



4  
  loadSnippetsDBForAnalysis();



5  
  print("Tokenizing.");



6  
  allTokens = new L;



7  
  time {



8  
    for (CSnippet s)



9  
      if (nempty(s.text)) {



10  
        L<S> tokens = internAll(javaTok(s.text()));



11  
        allTokens.add(tokens);



12  
        s.text = tokens;



13  
      }



14  
  }



15  
  print(n2(totalLength(allTokens)) + " tokens");



16  
}



      
download  show line numbers  debug dex  old transpilations   
Travelled to 13 computer(s): aoiabmzegqzx, bhatertpkbcr, cbybwowwnfue, cfunsshuasjs, gwrvuhgaqvyk, ishqpsrjomds, lpdgvwnxivlt, mqqgnosmbjvj, pyentgdyhuwx, pzhvpgtvlbxg, tslmcundralx, tvejysmllsmz, vouqrxazstgt


No comments. add comment
      
      
  

  

    Snippet ID:
    #1015262
  

  

    Snippet name:
    tokenizedSnippetDB
  

    

    Eternal ID of this version:
    #1015262/1
  

  

    Text MD5:
    c2bbf23ee404e6ffa216dbfbf36a955f
  

    

    Author:
    stefan
  

  

    Category:
    javax
  

  

    Type:
    JavaX fragment (include)
  

        

    Public (visible to everyone):
    Yes
  

  

    Archived (hidden from active list):
    No
  

  

    Created/modified:
      2018-05-11 04:08:53
  

  

    Source code size:
    373 bytes / 16 lines
  

  

    Pitched / IR pitched:
    No / No
  

    

    Views / Downloads:
    407 / 438
  

      

    Referenced in:
[show references]