Backup Everything at Once with RSS

My Ghost blog is not serverless. Running a blog on a server takes constant upkeep, but it has a lot going for it, and one big drawback. If the server crashes, getting the posts back out is really hard. As the blog grows, copying every post and photo by hand would get way too tedious. So I wanted a better way.

Problems with Ghost's Built-in Backup

Ghost lets you download a backup file in .json format. It's a full copy of the blog's soul. Everything you can set in Ghost goes in as is, the author's name, the tags, post content and format, upload time, even the summary in the HTML meta tags.

But there are two problems.

• A human can barely read the file. The JSON is minified, the structure is complicated because it holds so much, and the posts are compressed.
• It doesn't back up photos. Restore the blog and every photo shows "not found" (broken images, as everyone calls them). If the server is still alive, or you copied the photos somewhere, you're in luck. If not, some photos may be gone for good.

Goals

Main Goal

• Both posts and photos should be backed up. ★★★

Bonus Goals

• It should be in a human-readable format. (Human-Readable Medium) ★★☆
• It should be clear which photo goes into which location of which post, in preparation for restoring the blog.
• Backup should be convenient. ★★☆
• It should be possible to create a replica outside the blog. ★★☆

Idea

The answer is RSS. RSS came out of the blogging boom of the early 2000s as a way to "subscribe" to sites. Sites or blogs that support RSS provide an RSS feed address. The RSS feed address contains updated content from that site in a machine-readable format. When users enter the RSS feed address into an RSS reader, the reader fetches new content from the RSS feed address each time.

Social media has mostly made RSS obsolete, but it does everything I need here. The feed works as an API for getting posts. Ghost supports RSS by default, so I decided to use it.

General Idea

1. Enter the blog's RSS address and copy the whole feed.
2. Parse it and extract each post's HTML.
3. Create a folder per post and save its HTML there.
4. Download photos from the src addresses in the post's img tags.
5. For posts with photos, create an images folder inside the post folder, save the photos there, and point each img tag's src at the relative path.

Development

Reference

All the examples below are based on v1 of anaclumos/backup-with-rss. By the time you read this, new features or bug fixes may have been added.

Also, the code included in this post is intended to show a general flow, not the entire code. If you try to copy and run it as is, it probably won't work! The complete code is available in the GitHub repository.

1. Copying the RSS Feed Using Feedparser

I copied the RSS feed with Python's Feedparser module.

class RSSReader:
    origin = ""
    feed = ""

    def __init__(self, URL):
        self.origin = URL
        self.feed = feedparser.parse(URL)

    def parse(self):
        return self.feed.entries

RSSReader loads the feed and hands back its entries.

Here's all the code does.

1. Creating an RSSReader stores the RSS address in self.origin, parses it, and stores the result in self.feed.
2. parse() returns the entries from self.feed.

The entries hold the feed's posts as a list. The example below is the RSS for this post.

Structure of self.feed.entries in parse()

// Some parts omitted
{
  "bozo": 0,
  "encoding": "utf-8",
  "entries": [],
  "feed": {
    "generator": "Ghost 3.13",
    "generator_detail": {
      "name": "Ghost 3.13"
    },
    "image": {
      "href": "https://blog.chosunghyun.com/favicon.png",
      "link": "https://blog.chosunghyun.com/",
      "links": [
        {
          "href": "https://blog.chosunghyun.com/",
          "rel": "alternate",
          "type": "text/html"
        }
      ],
      "title": "Sunghyun Cho",
      "title_detail": {
        "base": "https://blog.chosunghyun.com/rss/",
        "language": "None",
        "type": "text/plain",
        "value": "Sunghyun Cho"
      }
    },
    "link": "https://blog.chosunghyun.com/",
    "links": [
      {
        "href": "https://blog.chosunghyun.com/",
        "rel": "alternate",
        "type": "text/html"
      },
      {
        "href": "https://blog.chosunghyun.com/rss/",
        "rel": "self",
        "type": "application/rss+xml"
      }
    ],
    "subtitle": "Sunghyun Cho's Blog",
    "subtitle_detail": {
      "base": "https://blog.chosunghyun.com/rss/",
      "language": "None",
      "type": "text/html",
      "value": "Sunghyun Cho's Blog"
    },
    "title": "Sunghyun Cho",
    "title_detail": {
      "base": "https://blog.chosunghyun.com/rss/",
      "language": "None",
      "type": "text/plain",
      "value": "Sunghyun Cho"
    },
    "ttl": "60",
    "href": "https://blog.chosunghyun.com/rss/",
    "namespaces": {
      "": "http://www.w3.org/2005/Atom",
      "content": "http://purl.org/rss/1.0/modules/content/",
      "dc": "http://purl.org/dc/elements/1.1/",
      "media": "http://search.yahoo.com/mrss/"
    },
    "status": 200,
    "version": "rss20"
  }
}

2. Creating Markdown Files from RSS Data

I only needed a few values from the self.feed.entries that RSSReader returns, so I made an MDCreator class to work through them.

class MDCreator:
    def __init__(self, rawData, blogDomain):
        self.rawData = rawData
        self.blogDomain = blogDomain

    def createFile(self, directory):
        try:
            os.makedirs(directory + "/" + self.rawData.title)
            print('Folder "' + self.rawData.title + '" Created ')
        except FileExistsError:
            print('Folder "' + self.rawData.title + '" already exists')

        self.directory = directory + "/" + self.rawData.title

        MDFile = codecs.open(self.directory + "/README.md", "w", "utf-8")
        MDFile.write(self.render())
        MDFile.close()

The blogDomain parameter is used later.

Here's all the code does.

1. Creating an MDCreator stores the blog address in self.blogDomain and the raw feed data in self.rawData. That raw data is the self.feed.entries from RSSReader's parse().
2. createFile() makes a folder per post inside the backup folder, named after the post's title. Each folder gets a README.md with the post inside.

Files are created through the codecs library so Windows uses Unicode instead of the CP949 codec. That's what makes emojis in the RSS show up correctly 🚀🥊

3. Adding Post Information to the Generated Markdown File

For the post information I went with Jekyll-style Front Matter. It seemed like the easiest way to check a post's title, tags, link, author, and the rest.

def render(self):
    try:
        postTitle = str(self.rawData.title)
    except AttributeError:
        postTitle = "Post Title Unknown"
        print("Post Title does not exist")

    try:
        postTags = str(self.getValueListOfDictList(self.rawData.tags, "term"))
    except AttributeError:
        postTags = "Post Tags Unknown"
        print("Post Tags does not exist")

    try:
        postLink = "Post Link Unknown"
        postLink = str(self.rawData.link)
    except AttributeError:
        print("Post Link does not exist")

    try:
        postID = str(self.rawData.id)
    except AttributeError:
        postID = "Post ID unknown"
        print("Post ID does not exist")

    try:
        postAuthors = str(self.rawData.authors)
    except AttributeError:
        postAuthors = "Authors Unknown"
        print("Authors does not exist")

    try:
        postPublished = str(self.rawData.published)
    except AttributeError:
        postPublished = "Published Date unknown"
        print("Published Date does not exist")

    self.renderedData = (
        "---\nlayout: post\ntitle: "
        + postTitle
        + "\ntags: "
        + postTags
        + "\nurl: "
        + postLink
        + "\nauthors: "
        + postAuthors
        + "\npublished: "
        + postPublished
        + "\nid: "
        + postID
        + "\n---\n"
    )

Here's all the code does.

1. It checks if the post's title, tags, link, ID, author names, and publication date are in the RSS, and puts what it finds into the Front Matter.
2. If a value is missing it writes ~ Unknown instead.

Tags take the odd code like self.getValueListOfDictList(self.rawData.tags, "term") because Ghost writes them in this format. Gatsby and WordPress do the same.

'tags': [{'label': None, 'scheme': None, 'term': 'English'},
         {'label': None, 'scheme': None, 'term': 'Code'},
         {'label': None, 'scheme': None, 'term': 'Apple'}],

def getValueListOfDictList(self, dicList, targetkey):
    arr = []
    for dic in dicList:
        for key, value in dic.items():
            if key == targetkey:
                arr.append(value)
    return arr

This pulls only the term out of tags and adds it to the Front Matter. Run it and you get Front Matter like this.

---
layout: post
title: Apple's Easter Egg
tags: ["English", "Code", "Apple"]
url: https://blog.chosunghyun.com/apples-easter-egg/
authors: [{ "name": "S Cho" }]
published: Sun, 19 Jan 2020 17:00:00 GMT
id: /_ Some Post ID _/
---

And this is how GitHub renders it.

4. Adding Post Summary and Content to the Generated Markdown File

Next the Summary and Content from the RSS go into renderedData.

self.renderedData += "\n\n# " + postTitle + "\n\n## Summary\n\n"
try:
    self.renderedData += self.rawData.summary
except AttributeError:
    self.renderedData += "RSS summary does not exist."

self.renderedData += "\n\n## Content\n\n"
try:
    for el in self.getValueListOfDictList(self.rawData.content, "value"):
        self.renderedData += "\n" + str(el)
except AttributeError:
    self.renderedData += "RSS content does not exist."

One interesting thing. Ghost and WordPress support both RSS Summary and Content, but Jekyll-based GitHub Pages and Tistory stuff the whole post into the Summary. (...) Ghost lets you set a post's Excerpt, and that becomes the Summary.

5. Adding Images to the Generated Markdown File

A backup is only whole if the images survive too. Unless an image is embedded in the HTML as base64, it's just an img tag with a src. Once the server goes down that src loads nothing, so every image has to be downloaded during the backup.

I leaned on How to Download All Images from a Web Page in Python by PythonCode for this part.

soup = bs(self.renderedData, features="html.parser")
for img in soup.findAll("img"):
    for imgsrc in ["src", "data-src"]:
        try:
            remoteFile = img[imgsrc]
            break
        except KeyError:
            continue

    if self.isDomain(remoteFile) != True:
        print("remoteFile", remoteFile, "is not a domain.")
        remoteFile = self.blogDomain + "/" + remoteFile
        print("Fixing it to", remoteFile)

    print('Trying to download "' + remoteFile + '" and save it at "' + self.directory + '/images"')
    self.download(remoteFile, self.directory + "/images")

    img["src"] = "images/" + remoteFile.split("/")[-1]
    img["srcset"] = ""
    print(img["src"])

self.renderedData = str(soup)
return self.renderedData

Here's all the code does.

1. It reads the renderedData string as HTML and finds every img tag.
2. It checks for a src or data-src attribute. data-src is there for WordPress compatibility.
3. It makes an images folder inside each post folder and saves the images there. The file name comes from the last part of the src, so https://blog.someone.com/images/example.png becomes images/example.png.
4. It rewrites the src to that relative path.
5. If there's a srcset, it removes it, for Gatsby compatibility.

def download(self, url, pathname):
    if not os.path.isdir(pathname):
        os.makedirs(pathname)

    response = requests.get(url, stream=True)
        file_size = int(response.headers.get("Content-Length", 0))
        filename = os.path.join(pathname, url.split("/")[-1])

        if filename.find("?") > 0:
            filename = filename.split("?")[0]

        progress = tqdm(
            response.iter_content(256),
            f"Downloading {filename}",
            total=file_size,
            unit="B",
            unit_scale=True,
            unit_divisor=1024,
        )

        with open(filename, "wb") as f:
            for data in progress:
                f.write(data)
                progress.update(len(data))

One problem is image addresses are all over the place. Some sites write the full domain, like <img src = "https://example.png/images/example.png">. Others start from a subdirectory, like <img src = "/images/example.png">. A few just write <img src = "example.png">. To cover as many of these as I could, I wrote an isDomain() function that spots the domain. Other libraries kept reading file extensions like .png as top-level domains like .com, so I added some exception handling.

def isDomain(self, string):
    if string.startswith("https://") or string.startswith("http://"):
        return True
    elif string.startswith("/"):
        return False
    else:
        return validators.domain(string.split("/")[0])

When an address can't be reached directly, like <img src = "/images/example.png">, the domain gets added in front. This is where self.blogDomain comes in.

Results

I tried it on this blog, a self-hosted Ghost blog. Just run main.py and the backup rolls on.

Testing says these services work. A post's format or layout may shift a little, but the backup itself does its job.

• Ghost
• WordPress
• Jekyll-based GitHub Pages
• Gatsby-based GitHub Pages
• Medium
• Tistory

Goal Achievement Evaluation

Main Goal

• Both posts and photos should be backed up. (3/3)

  Fully done. Videos don't get backed up, but videos usually come in through YouTube embeds anyway, so the chance of losing anything that way is much lower. That's why they were out of the goal from the start.

Bonus Goals

• It should be in a human-readable format. (Human-Readable Medium) (2/3)

  Compared to Ghost's own backup, the important bits sit right in the Front Matter and posts render almost like they do on the blog. Folders make posts and photos easy to find too. Editing is the awkward part. The file is Markdown but the body is HTML. It does just what Lots of copies keep stuff safe asks for.

• It should be clear which photo goes into which location of which post. (In preparation for restoring the blog) ★★★

  It is clear which photo goes into which location of which post.

• Backup should be convenient. (2/3)

  main.py needs to be executed manually. I'm thinking of automating it with crontab someday.

One catch comes from RSS itself. Only posts in the feed get backed up. Feeds usually carry just the latest posts to save bandwidth, and each blog has an option to adjust that. A Ghost blog puts its latest 15 posts in the feed by default, and you can't change that count inside the Ghost CMS. You have to touch the code of Ghost Core.

• It should be possible to create a replica outside the blog. (2/3)

  Download a lot of photos from a WordPress blog over and over, and access can get blocked for a while.

Future Plans

Sitting with the finished thing for a bit, I started thinking it could help people who want to move their blog but are stuck with a lot of piled-up data. The plan from here is to keep improving it in that direction, into a tool that helps with blog migration.

References

• How to Download All Images from a Web Page in Python by PythonCode
• Ghost Custom RSS Feed
• If the feed contains unsupported Unicode characters, the script won't run properly. An editor or blog error usually causes this, and a service like Feed Validator can find those characters so you can fix them and get things running again.