PageScrape

PageScrape SMS

2006-07-28T11:06:00.000+01:00

As part of the afore mentioned experiment into using PageScrape to build a simple SMS text message based weather forcast system I registered with Clickatell. They run a Gateway that allows clients to send SMS messages to mobile phones. Clients can submit message for sending via HTTP GET requests --> Enter PageScrape!

It's free to register and they give you 10 credits free. It seems that sending txt messages mostly costs about 0.8 credits each, with each credit costing 0.04 cent. So, sending a txt to Vodafone or o2 costs about 0.032c which isn't too bad....

The minimum number of credits you can purchase is 200 which works out at about a minimum payment of €8.80 to get going, again this is grand, some gateways expect you to spend way more money!

Anyway to send a txt you just need your ApId, username and Password, target mobile number and a message, you can send a txt by issuing the following GET:

http://api.clickatell.com/http/sendmsg?api_id=26437&user=username&password=passwd&to=&text=The Message!

Using PageScrape:

pscrape -u"http://api.clickatell.com/http/sendmsg?api_id=26437&user=username&password=passwd&to=&text=The Message!" -e"(.*)" -g

It seems to work really well, the messages arrive quickly &c.

They have a full HTTP API with methods for querying your credit balance and the like, all of these could be invoked using PageScrape.

Anyway that's the outgoing part of the Weather Forecast System more or less squared away, next, how do we receive SMS messages? Although it costs to send the weather forecast txts, we want to make sure that it costs us _nothing_ to receive the initial request SMSsesses!

Now where's that box of dusty old mobiles....

Gosh it's Hot

2006-07-24T14:27:00.000+01:00

Well, it's has been boiling here lately, very uncharacteristically warm and sunny! This has resulted in 'the weather' overtaking 'the house prices' as the topic of most conversations (be they in the pub or elsewhere).

Anyway, sticking to the current conversational norm - Met Éireann has updated its web page, its new web page is much nicer than its old one, but it also means that we need a new PageScrape expression to scrape the weather forecast, The following does the trick, it gets Today's weather....

pscrape -u"www.met.ie/forecasts/" -e"> Today.*p>([^<]*)<"

This should result in something like -->

"Most places will have another warm, dry day with very good sunshine. However outbreaks of rain and drizzle will affect western and northwester areas throughout the day, some thundery downpours are possible. Highest
temperatures of 20 to 25 degrees."

Here's a question -- armed with the above information, how would you go about building an SMS txt message based system which would send out this weather forecast text to a requester by SMS. The system where possible should use old stuff (perhaps mobile phone(s)?) found lying around the house at the bottom of boxes and dusty drawers as well as Broadband!

Here's one interesting starting point - gnokii

CamScrape

2006-07-24T11:33:00.000+01:00

These days many devices (such as routers, mp3 streamers, IP cameras &c.) have little embedded HTTP web servers, they allow themselves to be configured and controlled via these servers. Control and communication with the device normally occurs through the user doing stuff from a web browser connected to the device.

As usual (In the PageScrape spirit) there is no reason why these HTTP requests cannot be issued programmatically, and if the device relies primarily on GET requests there is no reason why PageScrape can't be used to control the device (and analyise the returned data).

A few years ago I used PageScrape to control Turtle Beach's AudioTron, a network based mp3 streamer. The user could have full control of the features of the player all through simple GET requests, list tracks, albums etc, play, pause etc etc, it was all available via a published set of commands. I was easily able to implement a WiFi based remote control by invoking PageScrape with different GET requests and associated RegularExpressions.

More recently I have been playing with a really nice WiFi IP Camera (Peak SOHO) that I came across in a review by ElReg.

It's an amazing camera for the price, and once again deals with the world via a HTTP interface. So now I am wondering what changes need to be made to PageScrape to allow for it to acquire an image from the camera and then save it to disk? I want to automatically acquire images through a script and I hope that with only a few small changes, PageScrape can help out.....

By the way I found it quite hard to find a decent distributer in the UK to purchase the camera from, the distributers listed on the Peak Website were useless to a man!

In the end I bought it from BigPockets.

Formatting Fiasco

2006-07-21T15:39:00.000+01:00

One of PageScrape's stalwart users has reported a bug in the output formatting option (-f). The functionality only works for the first 9 regular expression buffers, it does not work for buffers 10 and higher. For example consider the following:

pscrape -u"www.webscrape.com" -e"([^<]*)" -f"The Title is: \$1"

This will return the title of the webscrape web page as follows:

The Title is: PageScrape - A HTML Screen Scrape Utility for Web Pages

In the format string, \$1 refers to the first result buffer and causes the contents of this buffer to be inserted into the output string on a successful match. This works, and so will \$9 for the ninth buffer (if there is one in the corresponding Regular Expression), however it will not work for buffers donated by \$10 and higher.

The formatting algorithm only checks for one decimal digit!

This bug should be corrected soon.

Broken Examples

2006-07-21T15:02:00.000+01:00

So it appears that some of the examples on the website are now broken I haven't been checking them lately and hadn't noticed! This is a fundamental problem encountered when screen scraping, when the target screen format changes your scrapes may well stop working.

As PageScrape uses regular expressions it is possible to design scrape expressions that are quite robust in the face of simple HTML format changes, but it is obviously never possible to protect against all changes, and so it becomes necessary to periodically check that your scrapes are still working. The good news is that this process can be automated, however it appears that I am just to bone lazy to automate the testing of the PageScrape examples!

I hope to get around to fixing the broken example soon... I might remove the stock quote examples as they are a bit complex, a bit boring, and just a bit cliched and I am getting loads of stock related spam!

Brave New World

2006-07-21T14:43:00.000+01:00

I have decided to set up a BLOG for posting any vaguely interesting guff relating to PageScrape and Screen Scraping in general. So welcome one and all!

http://www.webscrape.com