Getting the most recent comic

[phlip]

Sorry, davean, but I'm hijacking your OP.

There is a JSON file at http://xkcd.com/info.0.json which contains the details for the current strip. Details for past strips are also available at http://xkcd.com/123/info.0.json. If you need to access it from another page with JavaScript, there is a JSONP proxy at http://dynamic.xkcd.com/api-0/jsonp/comic/ for the latest strip, or http://dynamic.xkcd.com/api-0/jsonp/comic/123 for a specific strip... this accepts the usual ?callback=whatever parameter, so you can use it with jQuery or whatever. The JSONP proxy is slightly less reliable, though, and if you can use the static JSON file directly then you should.

This is, quite simply, the best way to get the latest comic. It's easy to parse, and most likely to remain consistent... compared to scraping information out of the HTML page (harder, and more fragile if the page format changes), or scraping from the RSS feed (even more fragile, for the reasons davean goes into below). The JSON, however, is easy to get, easily machine-readable (there are libraries out there for pretty much any language you want to use), simple to get the data out of, and unlikely to change drastically in the future. Save yourself the pain and use it. Thank you.

The original OP has been thrown into a spoiler tag for being out-of-date (it recommends scraping the HTML, only because the JSON didn't exist yet at the point this post was made).

Spoiler:

Please people, using the RSS like you are doing is just plain broken, not only does it contain other items, it only contains 3, and it might be possible none of those are comics thus requiring the script to save state and handle several edge conditions.

There are two sane ways to get the current comic, ether use http://xkcd.com/current/ which will redirect the most recent comic image or parse http://xkcd.com/ directly at which point your code will work consistently and invariantly unless the site templates change (and any time you code to an API, API changes may cause program changes).

Broken scripts are not cool. I'll sticky good scripts that are their own topic if people want.

Just please stop with the RSS junk, your code hurts me eyes. (Not only is the RSS more complicated (and none of you have even tried to do it right) but it can only be done correctly as your poll interval goes to zero)

[lachlanmiller]

http://xkcd.com/current/ : 404 - Not Found

Edit: Is Currently the best way to get the most recent comic pass the index page through a regex ?

Regards,
Lachlan

[coldsteelrail]

I have written some parsing code which will effectively parse xkcd.com via DOM. This is a much better way that doing regex scraping. However I still have to figure out how to schedule cron jobs to get the most recent comics. i.e how does the code find out that a new comic has been published. The fact that the posting times are very regular, we could make use of that fact and just go fetch a new comic on M/W/F at the stipulated times.

[abrenecki]

I have written some parsing code which will effectively parse xkcd.com via DOM. This is a much better way that doing regex scraping.

Is your code open source? What language? Can it retrieve the alt-text?
xkcdmailer uses a horrible regex (see it's thread in my sig if you want to see how ghastly the regex is (or want to be emailed xkcd, for that matter </plug>)) at the moment. It works, but it feels a little icky.

[thinglie]

Returns a 404. Shit.

[chibu]

Hi everyone,

I saw this thread and it looks like there are not many examples of how to get the most recent comic. So, in the past few minutes I decided to throw together a function in PHP that can be included in a website to display the most recent XKCD Comic. I'm not sure if this is useful to anyone or if this is really what the thread is about, but I thought I'd post it anyway.

Code: Select all

function getNewestXKCD() {
   $document = new DOMDocument();
   // Load the HTML document from xkcd.com. The '@' is to suppress error messages (since there is one!).
   $load = @$document->loadHTMLFile('http://www.xkcd.com/');
   
   $src = '';
   $title = '';
   $alt = '';
   
   // If we can connect to the site:
   if ($load) {
      // Make an array of all of the image tags in the page.
      $images = $document->getElementsByTagName('img');
   
      // Go through each image tag to until we find the comic.
      foreach ($images as $image) {
         // If it's the one with the comic:
         if (substr($image->getAttribute('src'), 0, 28) == 'http://imgs.xkcd.com/comics/') {
            $src = $image->getAttribute('src');
            $title = $image->getAttribute('title');
            $alt = $image->getAttribute('alt');
            break; // We don't need to look anymore.
         }
      }
   
      // Create output (no need to use the DOM for this part):
      $return = '<div id="newKXCD" style="text-align:center;"><h1 style="font-size:20px;font-variant:small-caps;font-family:Lucida, sans-serif;font-weight:bold;">'.$alt.'</h1>';
      $return .= '<a href="http://www.xkcd.com" style="size:14pt;color:black;text-decoration:none;"><img src="'.$src.'" title="'.$title.'" alt="'.$alt.'" border="0" /></a></div>';
   }
   // If we cannot connect to the site, print an error message.
   else $return = '<div id="newKXCD" style="text-align:center"><a href="http://www.xkcd.com">Newest XKCD Comic cannot be accessed<br>at this time, please try again later!</a></div>';
   return $return;

} // End function

// Call the function which returns the HTML to display the image
echo getNewestXKCD();

// Oh and: Unexpected end tag : a in http://www.xkcd.com/, line: 48

The example can be shown working at http://www.chibu.net/xkcd/newest.php which is an exact copy of the above. Or you can simply try it for yourself on your own server. Next time I have a minute, I might write one in Javascript or Perl or C or maybe Python!

Well, I hope someone finds this useful,

~ Chibu

[d0nk]

I wrote a perl script that is run via cron. It creates a .js file with two variables:

Code: Select all

var img;
var txt;


img is the url to the image, and txt is the "title" or "alt text".

I have a custom login page for the main application I use at work (it has an html login page) which displays the latest comic, and the caption is displayed under the comic if I click it.

The perl script:

Code: Select all

#!/usr/bin/perl

use strict;
use LWP::UserAgent;

my ($img,$txt);
use constant OUTFILE => '/home/ryan/public_html/xkcd.js';

my $ua = LWP::UserAgent->new;
my $res = $ua->get('http://xkcd.com/');

if( $res->is_success ) {
        $img = $1 if $res->decoded_content =~ /Image URL \(for hotlinking\/embedding\): ([^<]+)<\/h3>/i;
        $txt = $1 if $res->decoded_content =~ /<img src=\"$img\" title=\"([^\"]+)\"/i;

        open( FILE, ">" . OUTFILE );
        print FILE "var img = \"$img\";\n";
        print FILE "var txt = \"$txt\";\n";
        close FILE;
}


My login page is similar to:

Code: Select all

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
  <title>******* Startup Page</title>
  <script language="JavaScript" type="text/javascript" src="http://example.com/xkcd.js"></script>
  <script language="JavaScript" type="text/javascript">
    function $(x) {
      return document.getElementById(x);
    }

    function setCaption() {
      $('caption').innerHTML = "<p>" + txt + "</p>\n";
    }
  </script>

  <style type="text/css">
  /*<![CDATA[*/
  body  {font: 17px Arial, sans-serif; line-height: 1.6;}
  a {color: #67C;}
  a:hover {color: #9CF;}
  img { border: none; }
  /*]]>*/
  </style>
</head>

<body onload="">

<div style="text-align:center">
  <h1>Welcome to *******</h1>
      <a href="javascript:setCaption();"><img id="comic" alt="Latest XKCD" /></a>
  <br />
  <div id="caption" style="display:inline">
   
  </div>
</div>

<p style="text-align:center;">
  <a href="Login URL (its not a standard url)">Log on to Main System</a>
</p>

<script type="text/javascript">
  $('comic').src = img;
</script>

</body>
</html>


Hopefully that will be useful to someone. The perl script will be fine unless the front page changes, etc.

[GameFreak]

This is what I use, it downloads the current comic as well as all of the comics since it was last run, then appends the title
It fails on comic 404, and on any comic involving accented characters.

Code: Select all

#! /bin/bash

prev=`cat ./last`
latest=`curl -s xkcd.com | egrep -o "http://xkcd.com/([1-9][0-9]*)" | sed 's/.*\([1-9][0-9][0-9]\).*/\1/'`
#curr=509


for ((curr=prev+1; curr<=latest; curr++))
do
page="http://www.xkcd.com/$curr/"
#download the source and extract the proper img tag
echo "retrieving comic $curr data"
tag=`curl -s $page | egrep "(<img src=\"http://imgs\.xkcd\.com/comics/.*/>)"`

src=`echo $tag | sed 's/.*src=["]\(.*\)["] title=["]\(.*\)["] alt=["]\(.*\)["].*/\1/'`
title=`echo $tag | sed 's/.*src=["]\(.*\)["] title=["]\(.*\)["] alt=["]\(.*\)["].*/\2/'`
alt=`echo $tag | sed 's/.*src=["]\(.*\)["] title=["]\(.*\)["] alt=["]\(.*\)["].*/\3/'`
filename=`echo $src | sed 's/^.*[/]\(.*\)/\1/'`
newname=$curr"_"$filename
echo "downloding image"
curl -# -o $filename $src

if [[ $filename == *.jpg ]]
then
   echo "converting to png"
   convert $filename `echo $filename | sed 's/\.jpg/\.png/'`
   rm $filename
   filename=`echo $filename | sed 's/\.jpg/\.png/'`
fi

echo "adding captions"
width=`identify -format %w $filename`
convert -background lightblue  -fill blue -size ${width}x caption:"$alt" "title_"$filename
convert -background lightblue  -fill blue -size ${width}x caption:"$title" "caption_"$filename
convert "title_"$filename $filename "caption_"$filename -append $newname

echo "cleaning up"
rm $filename
rm "caption_"$filename
rm "title_"$filename

done
echo "$latest" | cat > ./last


[Comic JK]

Why not just calculate the proper comic number based on the date? That's how I do it on my own site--my php page calculates the days since January 1st, 1970 based on php's time() function, subtracts a bunch, and then works out how many "comic days" have passed (eg, 3 days out of 7).

Much, much simpler than other scripting-based methods proposed here.

[phlip]

And next time he does a 5-comic week, like Parody Week, Choices, 1337 and Secretary? Or if the unthinkable happens and he has to miss an update? How badly will your site break then?

[davean]

Er, use the JSON.

http://xkcd.com/info.0.json
http://xkcd.com/432/info.0.json

The "0" is a version, I'll try to give a 3 month notice before I depreciate a version. I may add fields to an existing version without a version bump. I will not change or remove a field without a version bump.

if you need to get it via JS, there is a proxy to wrap it for that, I'll document it later.

If enough people use it I'll setup a mailing list for announces.

This has actually been there for ages but I forgot about this thread on the forums.

[makc]

use the JSON

any chance to get http://xkcd.com/crossdomain.xml to allow queries to JSON from flash script? see http://api.flickr.com/crossdomain.xml for example.

[Cue]

XML in general would be nice, it's much easier to parse in C# than JSON is, since C# has a nice set of functions for doing such a thing.
Example http://xkcd.com/658/api.xml:

Code: Select all

<comic>
<image>http://imgs.xkcd.com/comics/orbitals.png</image>
<title>Orbitals</title>
<month>11</month>
<num>658</num>
<link>http://www.xkcd.com/658/</link>
<year>2009</year>
<alt>And now someone drunk and hot stumbles in, one thing follows another, and the next roommate to return home sleeps in the hall lounge orbital.</alt>
<day>4</day>
</comic>

You know, just to help those out who want to use something other that JSON... :3
Okay sure the RSS feed has it, but that's only the last 4 comics...

[phlip]

JSON is ridiculously easy to parse, even without a built-in library. But that said, The JSON site has links to about half a dozen C# JSON libraries.

To have XML as well would be redundant and unnecessary.

[Cue]

Yes, but those JSON libraries (most of them, if not all) are not for the Compact Framework, which is the primary reason why I want to mess with this little API here. So I can mess about with some Windows Mobile stuff.

Apparently there's a CF one out there, but it's highly experimental, I guess I'll have to give it a go though, or hack together my own JSON functions.

[hooktail154]

I just recentlty worte some objective-C (For Mac Peoplez) to parse the standard xkcd page and get the image name, image url, altText, and page url

Unforunately, it requires the WebKit framework and a small workaround

Code: Select all

   NSString *source = [[[[webView mainFrame] dataSource] representation] documentSource];
   NSString *refine1 = [source stringByReplacingCharactersInRange:NSMakeRange(0, 2401) withString:@""];
   NSString *refine2 = [refine1 stringByReplacingCharactersInRange:NSMakeRange(([refine1 length] - 5156), 5156) withString:@""];
   /*NAME*/comicName = [refine2 substringWithRange:NSMakeRange(4, (([refine2 rangeOfString:@"</h1>"].location) - 4))];
   NSString *refine3 = [refine2 substringFromIndex:([refine2 rangeOfString:@"img src=\""].location)+9];
   /*IMAGE URL*/imgURL = [refine3 substringToIndex:([refine3 rangeOfString:@"\""].location)];
   NSString *refine4 = [refine3 substringFromIndex:(([refine3 rangeOfString:@"title=\""].location) + 7)];
   /*ALT TEXT*/altText = [refine4 substringToIndex:([refine4 rangeOfString:@"\""].location)];
   NSString *refine5 = [refine4 substringFromIndex:(([refine4 rangeOfString:@"|</a></li>"].location) + 29)];
   /*PAGE LINK*/pageLink = [refine5 substringToIndex:([refine5 rangeOfString:@"</h3>"].location)];
   
   NSArray *objects = [NSArray arrayWithObjects:comicName, imgURL, altText, pageLink, nil];
   NSArray *keys = [NSArray arrayWithObjects:@"comicName", @"imgURL", @"altText", @"pageLink", nil];
   NSDictionary *data = [NSDictionary dictionaryWithObjects:objects forKeys:keys];

Note: There must be a variable named "webView" with a WebView object and this must be called after the xkcd page has been loaded

I suggest then setting the imgURL to the URL of a different WebView and hiding the first WebView

[davean]

JSON is ridiculously easy to parse and much faster to serve.

[phlip]

Well maybe if you edited the JSON link into the OP, and took out the reference to parsing the HTML... I'm just sayin'...

[maafy6]

Out of curiosity - what is the difference between "title" and "safe_title" in the JSON?

[hooktail154]

I've got an app up and running now, a few versions in, that uses a Python script http://forums.xkcd.com/viewtopic.php?f=11&t=25707 from The Onion Knight and Berengal to download all of the comics. I modified it to use the JSON files rather than parse HTML, and added the dates that the JSON provides into the altText.txt file. My app then uses some code to parse the title, altText, and date and insert them into the proper field in the app. The app just uses a WebView to display the comics. The Python is completely seperable, making both the script and the app completely adaptable to fit many situations. I have even adapted it to Questionable Content, which uses a similar script and virtually the same app shell to view the comics.

See my xkcd app at http://www.ht154.com/projects/xkcd/
See my QC app at http://www.ht154.com/projects/qc/

[makc]

any chance to get http://xkcd.com/crossdomain.xml to allow queries to JSON from flash script? see http://api.flickr.com/crossdomain.xml for example.

bump? who do I talk to about this?

[PM 2Ring]

Here's a short HTML / JavaScript file that fetches comics using JSONP. It was written using suggestions from Maelstrom and phlip. Thanks again, guys!

Code: Select all

<!DOCTYPE html>
<html>
<head>
<title>xkcd via JSONP</title>
 
<script type="text/javascript">
var details = {};
function dataloaded(o){details = o;}

function pageloaded()
{ 
   with(document.getElementById("image"))
   {
      src = details.img;
      alt = details.title;
      title = details.alt;
   }

   document.getElementById("title").innerHTML = details.title;     
   document.getElementById("num").innerHTML += details.num;
   document.getElementById("date").innerHTML = details.day + "/" + details.month + "/" + details.year;
   document.getElementById("mouseover").innerHTML = details.alt;
   
        //Format Transcript
   var str = details.transcript;
        //Fix < and >
   str = str.replace(/</g, '&lt;')
   str = str.replace(/>/g, '&gt;')
        //Fix newlines
   str = str.replace(/\n/g, '<br>\n');
        //Add extra breaks before bracketed sections
   str = str.replace(/[[]{2}/g, '<br>\n[[');   
        //Remove alt text from transcript
   str = str.replace(/{.*}/g, '');
   document.getElementById("transcript").innerHTML = str;
   
   var url = "http://xkcd.com/" + num;
   
    with(document.getElementById("original"))
    {
       href = url;
       innerHTML = url;
    }   
}

var num = prompt("Comic number");

var head= document.getElementsByTagName('head')[0];
var script= document.createElement('script');
script.type= 'text/javascript';
script.src = "http://dynamic.xkcd.com/api-0/jsonp/comic/"+num+"?callback=dataloaded";
head.appendChild(script);
</script>
</head>
<body onload="pageloaded()">
<h2 id="title"></h2>
<h3 id="num">#</h3>

<div id="date"></div><br>

<img id="image" style="border: 1px solid #888; padding: 10px;">
<!-- img id="image" style="border: 5px solid #000;"-->

<h3>Mouseover</h3>
<div id="mouseover"></div>
<!-- hr style="border: 1px solid #00F;" -->
<h3>Transcript</h3>
<div id="transcript" style="width: 50%;" ></div>
<br>
<a id=original></a>
<br><hr><br>
</body>
</html>


Any suggestions for optimizations and improvements gratefully accepted!

[SoupFlies]

I didn't like any of the code that I saw... But then again I know next to nothing about coding. So I wrote my own bash script for the lovely linux users! It will need some changes for personalization, but I don't know how to make the code to span across different installs and such... I will make sure to have comments for needed changes. Currently this works for me, and I have it set up to run on boot. Because I did this, I had to make up some way for it to run only once per day, I made a pretty brute force method, but it works!!! Please submit ANY changes, because I need to further my skilllzzz on the code monkeying.

I would suggest setting this up to be run on boot.
For run on boot in Ubuntu: (proved working on natty)
Save as any file, for now lets call it xkcdget.sh
Open terminal and cd to the directory xkcdget.sh is in

Code: Select all

mv xkcdget.sh /etc/init.d
sudo update-rc.d xkcdget.sh defaults
sudo chmod +x /etc/init.d/xkcdget.sh

Features:
Can only be run once per day
Pulls alt-text into separate file (not the best regex ever but its my first)
Set up to be run on boot easily
Pulls the comic on the day of release

AGAIN MAKE SURE TO CHANGE THE PATH

Code: Select all

#! /bin/sh

# Sleep is so mobile devices can get a wifi connection
sleep 15

# Change this path to personal preference
cd /home/Ben/Desktop/XKCD
# Thats it, run it however you please and it should work
_DATE="$(LC_ALL=C date +%A)"
if test "$_DATE" = "Monday"
    then
    rm .RAN2
        if test -e .RAN
            then
                echo "Already been run today"
            else
                touch .RAN
                wget http://xkcd.com/info.0.json
                grep -o http://[^[:space:]]*\.png info.0.json | xargs wget
                grep -o alt[^~]*\" info.0.json > alt
                rm info.0.json
        fi
fi
if test "$_DATE" = "Wednesday"
    then
    rm .RAN
        if test -e .RAN1
            then
                echo "Already been run today"
            else
                touch .RAN1
                wget http://xkcd.com/info.0.json
                grep -o http://[^[:space:]]*\.png info.0.json | xargs wget
                grep -o alt[^~]*\" info.0.json > alt
                rm info.0.json
        fi
fi
if test "$_DATE" = "Friday"
    then
    rm .RAN1
        if test -e .RAN2
            then
                echo "Already been run today"
            else
                touch .RAN2
                wget http://xkcd.com/info.0.json
                grep -o http://[^[:space:]]*\.png info.0.json | xargs wget
                grep -o alt[^~]*\" info.0.json > alt
                rm info.0.json
        fi
fi
exit

[PM 2Ring]

Inspired by a comment in the thread for comic #0974: "The General Problem", I decided to clean up my comic-fetching JavaScript, and add a few more features. I hope it works properly on your browser, if not please let me know.

This script displays the comic date in dd/mm/yyyy format. Feel free to modify that if you prefer a different format.

Fetch xkcd comics from the JSON feed via JavaScript

Code: Select all

<!DOCTYPE html>
<html>
<head>
<title>Fetch xkcd</title>
<meta http-equiv="Content-Script-Type" content="text/javascript">
<style type="text/css">
body { text-align: center; }
#image { border: 1px solid #888; padding: 10px; }
#transcript, #mouseover { width: 50%; text-align: left; margin-left: 25%; }
input { width: 52pt; }
</style>

<script type="text/javascript">
var baseURL = "http://dynamic.xkcd.com/api-0/jsonp/comic/",
    hinum, script, details = {},
    head = document.getElementsByTagName('head')[0];

function dataloaded(obj){details = obj; fill_page(); }

function ById(id) { return document.getElementById(id); }

//Return an error message if n is not a valid comic number.
function valid_num(n)
{
    var msg = '';

    switch(true)
    {
        //The empty string is valid - it's the current comic.
        case (n == ''):
            break;
        case (parseInt(n, 10) != n || n < 1):
            msg = "'" + n + "' is not a positive integer";
            break;
        case (n > hinum):
            msg = n + " is too high.\nThe current comic is " + hinum;
            break;
        case (n == 404):
            msg = "Comic " + n + " doesn't exist";
            break;
    }

    return msg;
}

function build_script(n)
{
    var msg = valid_num(n);
   
    if (msg)
    {
        alert(msg);
        return;
    }

    if (script != undefined)
        head.removeChild(script);
    script = document.createElement('script');
    script.type = 'text/javascript';
    script.src = baseURL + n + "?callback=dataloaded";
    head.appendChild(script);
}

function fill_page()
{
    var t, num = details.num, url = "http://xkcd.com/" + num;

    ById("title").innerHTML = details.title;
    ById("num").innerHTML = 'xkcd # ' + num;
    ById("date").innerHTML = details.day + "/" + details.month + "/" + details.year;

    t = ById("image");
    t.src = details.img;
    t.alt = details.title;
    t.title = details.alt;

    ById("mouseover").innerHTML = details.alt;
    ById("transcript").innerHTML = format_transcript(details.transcript);

    t = ById("original")
    t.href = url;
    t.innerHTML = url;

    if (hinum == undefined)
        hinum = num;

    t = ById("inum");
    if (t.value == '' || hinum == num)
        t.value = num;

    window.scroll(0, 0);
}

function format_transcript(s)
{
    //Fix < and >
    s = s.replace(/</g, '&lt;')
    s = s.replace(/>/g, '&gt;')

    //Fix newlines
    s = s.replace(/\n/g, '<br>\n');

    //Add extra breaks before bracketed sections
    s = s.replace(/[[]{2}/g, '<br>\n[[');

    //Remove alt text from transcript
    s = s.replace(/{.*}/g, '');

    return s;
}

function setnum(delta)
{
    var num = ById("inum");

    if (num.value)
        num.value = num.value - 0 + delta;
    build_script(num.value);
}

function random()
{
    var n, num = ById("inum");

    do n = Math.floor(1 + Math.random() * hinum);
    while (valid_num(n) != '');
    num.value = n;
    build_script(num.value);
}

</script>
</head>

<body onload="build_script('');">
<h2 id="title"></h2>

<h3>
<input type="button" value="prev" onclick="setnum(-1);">
<span id="num"></span>
<input type="button" value="next" onclick="setnum(1);">
</h3>

<div id="date"></div>
<br>
<img id="image">

<h3>Mouseover text</h3>
<div id="mouseover"></div>
<hr>

<h3>Transcript</h3>
<div id="transcript"></div>
<br>
<a id=original></a>
<br><hr><br>

Number<br>
<input type="button" value="prev" onclick="setnum(-1);">
<input type="text" id="inum" value="" onchange="setnum(0);">
<input type="button" value="next" onclick="setnum(1);">
<br>
<input type="button" value="random" onclick="random();">

<br><br>
</body>
</html>