Not logged in.  Login/Logout/Register | List snippets | | Create snippet | Upload image | Upload data

50
LINES

< > BotCompany Repo | #1005070 // Try scraping medium.com - fixing

JavaX source code [tags: use-pretranspiled] - run with: x30.jar

Libraryless. Click here for Pure Java version (2534L/17K/56K).

1  
!752
2  
3  
p {
4  
  S username = "stefanreich";
5  
  Map<S, O> data = loadMediumJSON(username);
6  
  printStructureLines(data);
7  
  
8  
  pcall {
9  
    Map user = (Map) data.get("user");
10  
    print("Name: " + user.get("name"));
11  
    print("Bio: " + user.get("bio"));
12  
    print("Image ID: " + user.get("imageid"));
13  
    // Could do more here, like Twitter name
14  
  }
15  
  
16  
  pcall {
17  
    Map references = (Map) data.get("references");
18  
    psl("References: ", references);
19  
    Map<S, Map> posts = (Map) references.get("Post");
20  
    psl("Posts: ", posts);
21  
    for (S postID : keys(posts)) {
22  
      print("Post " + postID);
23  
      Map post = posts.get(postID);
24  
      psl("Post: ", post);
25  
      
26  
      S uniqueSlug = getString(post, "uniqueSlug");
27  
      S link = "https://medium.com/@" + username + "/" + uniqueSlug;
28  
      S title = getString(post, "title");
29  
      print("Title: " + title);
30  
      print("Probable link: " + link);
31  
      //psl(post);
32  
      print();
33  
    }
34  
  }
35  
}
36  
37  
sbool loadMediumJSON_verbose;
38  
39  
static Map<S, O> loadMediumJSON(S username) {
40  
  S html = loadPageWithUserAgent("https://medium.com/@" + username, "Mac Safari");
41  
  if (loadMediumJSON_verbose) print(html);
42  
  
43  
  S prefix = [[window["obvInit"](]];
44  
  int i = indexOf(html, prefix);
45  
  if (i < 0) null;
46  
  int j = indexOf(html, "\n", i);
47  
  if (j < 0) null;
48  
  S json = dropSuffix(")", trim(substring(html, i+l(prefix), j)));
49  
  ret jsonDecodeMap(json);
50  
}

Author comment

Began life as a copy of #1004101

download  show line numbers  debug dex  old transpilations   

Travelled to 14 computer(s): aoiabmzegqzx, bhatertpkbcr, cbybwowwnfue, cfunsshuasjs, ddnzoavkxhuk, gwrvuhgaqvyk, ishqpsrjomds, lpdgvwnxivlt, mqqgnosmbjvj, pyentgdyhuwx, pzhvpgtvlbxg, tslmcundralx, tvejysmllsmz, vouqrxazstgt

No comments. add comment

Snippet ID: #1005070
Snippet name: Try scraping medium.com - fixing
Eternal ID of this version: #1005070/1
Text MD5: 2aaba4723f055b121c24afb345b9150c
Transpilation MD5: 39b64bf31a6b959fb74618f5c9ea3d42
Author: stefan
Category: javax / networking
Type: JavaX source code
Public (visible to everyone): Yes
Archived (hidden from active list): No
Created/modified: 2016-10-04 21:59:11
Source code size: 1431 bytes / 50 lines
Pitched / IR pitched: No / No
Views / Downloads: 493 / 535
Referenced in: [show references]