Libraryless. Click here for Pure Java version (1129L/7K/24K).
!752 p { //makeBot("Tokenizer Bot"); } answer { if "javatok *" ret structure(javaTok(m.unq(0))); if "pythontojavatok *" ret structure(pythonToJavaTok(m.unq(0))); s = s.trim(); if (swic(s, "snltok ")) { s = dropPrefixIgnoreCase("snltok ", s); ret structure(codeTokens(snlTok(s))); } }
download show line numbers debug dex old transpilations
Travelled to 15 computer(s): aoiabmzegqzx, bhatertpkbcr, cbybwowwnfue, cfunsshuasjs, gwrvuhgaqvyk, ishqpsrjomds, lpdgvwnxivlt, mqqgnosmbjvj, onxytkatvevr, pyentgdyhuwx, pzhvpgtvlbxg, teubizvjbppd, tslmcundralx, tvejysmllsmz, vouqrxazstgt
No comments. add comment
| Snippet ID: | #1002301 | 
| Snippet name: | Tokenizer Bot | 
| Eternal ID of this version: | #1002301/1 | 
| Text MD5: | 69869e11876bbad49f361ec04540297a | 
| Transpilation MD5: | bc02ffb8963abd3d9f86c9b2dd7a240b | 
| Author: | stefan | 
| Category: | javax | 
| Type: | JavaX source code | 
| Public (visible to everyone): | Yes | 
| Archived (hidden from active list): | No | 
| Created/modified: | 2016-03-12 17:51:40 | 
| Source code size: | 337 bytes / 19 lines | 
| Pitched / IR pitched: | No / No | 
| Views / Downloads: | 937 / 1888 | 
| Referenced in: | [show references] |