You will probably run into a healthy mix of "that's cool" / "I did that before you!" / "but how will it make money?". Ignore it and do your thing. If you figure out how to monetize it, great! Even if you don't or if you have no desire to, you will have learned and grown during the course of the project. That is invaluable.
Have fun and screw the haters...
Maybe I'm just jealous or something, but it rubs me the wrong way.
We're meant to be an inclusive community of smart people. The idea is we'll encourage the poster and offer constructive criticism (or praise).
If the post is useful to no one, it simply won't get discussed or upvoted. When something does, it's validated as an idea, or as something of interest.
I was trying to implement a Firefox add-on which navigates the web based on speech for my senior project (the theme was assistive technology). The major blocker with developing any tools to help navigate web page despite some effort in ARIA is that there are so many actions one cannot parse nor do without writing custom code for every single website. i.e. how can you tell where the login button is? what to click? frankly if you look at Gmail the DOM was a huge huge compressed mess (names all rewrote, without gmail.js I wouldn't be able to get Gmail working in my project). If every website exposes a standard set of APIs that can reduce the barrier by a good percentage. So think of combinging HATEOS from REST and this. Here we turn things into JSON with href allowing a client to navigate, sort of a first step making a website more "web client" compatible... funny is it?
this.get(html, selector, function(s){
var es = new DOMParser().parseFromString(html, 'text/html').querySelectorAll(selector);
return [].slice.call(es).map(function(n){ return n.innerText });
};
This is basically a subset of "I did that before you!"
For example:
curl https://news.ycombinator.com/news | tq -tj ".title a"
$ curl -d url=https://news.ycombinator.com/ -d json_data='{"title":"title"}' http://www.jamapi.xyz/
{
"title": "Hacker News"
}
curl -d url=https://news.ycombinator.com/ -d json_data='{"title":[{"elem":".title > a","value":"text"}]}' http://www.jamapi.xyz/
This is cool :)
EDIT:
Incidentally, you don't really need to have that "index" key inside the values of an array, because in an array the order is preserved anyway. Unless I've misunderstood what it means?
I currently do quite a bit of scraping for my own use (generating RSS feeds for sites, making simple commandline interfaces to automate common tasks, etc.). I've found xidel to be pretty good for this: it starts off pretty simple (e.g. with CSS selectors or XPath), but gets pretty gnarly for semi-complicated things. For example, it allows templating the output, using a language I struggle to grasp. This service seems to address that middle ground, e.g. restricting its output to JSON, and hence making the specification of the output much simpler (a nice JSON structure, rather than messing around with splicing text together).
Eg. Post with some sort of css selecters and then a "cookie memory".
If you are the application owner, check your logs for details.
I'm using Apifier at the moment, which I really like, but my biggest gripe is the awkwardness of source (and VCS) integration. The best I've come up with is to export the JSON config (which contains the scraper source code as a value - yuck) and try to remember to keep re-exporting and checking it in.
Having also had to hack around the inability to parameterise the scrape url (e.g. 'profile/$username') - which they've since added support for - I started to wonder if I mightn't as well just use BeautifulSoup (Python HTML parser lib) and check it in properly.
This is probably my ideal. I can keep it all in source control because it's just an HTTP request body, and I can parameterise it because, well, it's just an HTTP request body!
It's also open source because you're an amazing person; so if I had one little concern left about the availability of your site I can dismiss it right away since I could run my own on Heroku should jamapi.xyz prove unsustainable. It's possibly a better idea to do that anyway, but I often wonder if Heroku doesn't consider that a problem - multiple instances of the same app running on free dynos under different accounts...
Edit: Well no, it's only some sites. E. g. https://medium.com
Try using a backend language or just curl and it should be fine.
I found it: The API responds with an HTTP 500 error if you use CSS selectors that don't select anything or are simply invalid.
Probably makes sense to add some Exception handling right there.
Nonetheless, a great idea!
Edit: Just tried it in Node and it works brilliantly. Cool project.
Welcome feedback on my "Apply HN" on doing exactly this: https://news.ycombinator.com/item?id=11583348
#hnmain > tbody > tr:nth-child(3) > td > table > tbody > tr.athing > td.default
What about DOM nodes generated by JavaScript? Will Jam render the page before scraping?
Understatement of the year.
You'd need to either re-implement an entire browser stack or run a headless version of gecko of webkit server-side.
The former entails millions of man-hours of work. The latter opens up your server to all sorts of exploits. Overall a really bad idea.
Besides, single page applications are the worst junk in the entire Web 2.0 cesspool. If you really need to scrape them, they usually come with their own JSON API which you can just piggyback.
Unfortunately the aforementioned tools are generally pretty slow (especially headless browsers). Also I can't imagine it's particularly safe running such a service.
Say I wanted the Hacker News links + first comment?
As crawling complexity increases, pretty soon you want an actual programming language to specify things like crawl order and cache behavior. Multi-page behavior was very hard to describe declaratively for misbehaving sites.
Also, it's a terrible default (for security reasons) to let the web pages you're parsing automagically initiate new requests to arbitrary urls.
Such as it is, I believe that the following works in some version of parsley, though I doubt its an official release.
{
"articles": [ {
"title": ".title a",
"comment_link": "follow(.subtext a:nth-child(3) @href) .athing:nth-child(1) .default"
} ]
}
Right. We'd have to only grab the article-id, validate that it is in fact an interger in the right range, and only then piece the url back together and request it.
On the other hand, maybe just checking that we stay within the domain is enough. If the website wants to screw with us, they can send us any reply they want to any url anyway.
Don't feel :/ . The complexity is essential, and located in the remote website, not your code or your ideas. You still win isolating all the nasty stuff to one and precisely one location. :/ is on them, not you!
Running this sort of thing as a service/api never panned out for us because you are almost universally robots.txt denied and/or blocked.
We briefly tried, and supported a wiki of json extraction scripts at parselets.org, but it went nowhere after a few months.
You can write tests around whether your selectors are returning data, but even simple refactors from a dev team quickly break your selector profiles multiple times a week or month.
Just wasn't worth the hassle.
The trick is to emulate x11 with xvfb and control it with selenium web driver.
Phantom isn't the only choice, just the one most people talk about
As for non js heavy website, it's fairly trivial to find a library that will parse the dom for you, pretty every language have one
real issue is that popularity will get you blocked fairly quickly. see also: yahoo pipes.
Disclaimer: I'm a cofounder there
Disclaimer: I'm the founder there ;)
This one, for example http://www.videlibri.de/xidel.html#examples
But how would you monetize it?
Unlike an RSS feed, you really don't know how the JSON response would be used, so you can't inject ads into it.
And if you charge for it, wouldn't people assume it would continue to work, but site "scrapers", regardless of how they are configured, are likely to break, so it would be tougher having customers pay for something that could break at any time leaving them having to figure out if its the service that's changed/broken or the page that's changed.
Don't get me wrong- some great businesses have been/are based on "scraping" in one way or another. However, as cool as this is, it's just another way to "scrape". If the person hosting the page would provide an API or JSON view instead, you'd be loads better off.
In any case, not everything has to be monetised.
Isn't that the point? The demo seems like it'd be a lot easier, less verbose, and probably less brittle, than using cUrl/xpaths or otherwise parsing that HTML yourself.
The barrier for starting with JamAPI is impressively low, though! Kudos on the developer-friendly user interface.
- mentions the term that this concept falls under (no where on the OP's page, so he may not know that there is an entire set of software, plugins, etc that does this)
- provides one alternative implementation
- adds commentary related to why such services are necessary, and that they should be able to be monetized
So yes, he starts off with something along the lines of "I did that before you!", but he doesn't use a condescending phrase, and he provides additional useful information.
Overstatement of the year.
Why on Earth would the OP start from scratch? Besides, though not a solo and OSS effort, Apifier does this; certainly without "millions" of hours having been spent on it.
curl -X POST http://www.jamapi.xyz/ -d url=http://news.ycombinator.com -d json_data='{"title": "title","paragraphs": [{ "elem": "td.title a", "value": "text", "location": "href"}]}'
Use the http URL to call www.jamapi.xyz because calling https I get an Error code: SSL_ERROR_BAD_CERT_DOMAIN
Also, thanks, it's really cool to see people liking this :)
[1] http://www.json.org/: An array is an ordered collection of values. An array begins with [ (left bracket) and ends with ] (right bracket). Values are separated by , (comma).
Also, jq has a `--sort-keys` option which tries to make the output as reproducible/canonical as possible. From the manual:
> The keys are sorted "alphabetically", by unicode codepoint order. This is not an order that makes particular sense in any particular language, but you can count on it being the same for any two objects with the same set of keys, regardless of locale settings.
It would be strange for a JSON tool to go to such lengths to normalise data, if array order were unpredictable.
Plenty of mediocre stuff gets to the frontpage and plenty of gems fall through the cracks.