0%

Fun with GPG

This week, I was having making some great progress in understanding how GPG works either locally or through email. The original intention to do all this was because I would like my router to send me notification whenever the tranmission finish download the torrent. This may be simple as it sounds and it had been working correctly for me for several months since I create the initial script.

This week, however, I decided to do something special. I would like the router to sign/encrypt the message it sends to me. I am not sure why I need that, but anyways, I did get to learn a lot through the process.

Here is my original script, it simply uses the mailx program that installed on the router and sends the email through SMTP. Looks quite simple, it pretty much is the same as the script I have shown in the previous post:

1
2
3
4
5
6
7
8
#!/bin/sh

SMTP_SERVER=YOUR_EMAIL_SMTP
MESSAGE="Hello!\n\n \tThis is a notification from transmission, $TR_TORRENT_NAME has been completed on $TR_TIME_LOCALTIME\n\n Thanks!"
SENDER="YOUR_EMAIL_USER_NAME"
RECEPIENT="EMAIL_TO_RECEIVE"

printf "$MESSAGE"|mailx -vr $SENDER -s "[Transmission] Torrent Has Been Downloaded" -S smtp=$SMTP_SERVER -S smtp-use-starttls -S smtp-auth=login -S smtp-auth-user=$SENDER -S smtp-auth-password="YOUR_EMAIL_PASSWORD" -S ssl-verify=ignore $RECEPIENT

A little more explanation here, there are two variables preset by Transmission. $TR_TORRENT_NAME, is the name of the torrent that has just been finished. $TR_TIME_LOCALTIME is the time when the download was finished. There were several other environment variables set by transmission also. Here is a list of them^1
Note: The meaning of this variable are not explicitly documented in the wiki, and I guess the meaning based on my understanding.

Env Variable Name Meaning
TR_APP_VERSION The version of the transmission app.
TR_TIME_LOCALTIME The time when the current torrent has been downloaded.
TR_TORRENT_DIR The directory that the content of the torrent was downloaded to.
TR_TORRENT_HASH The hash value of the torrent.
TR_TORRENT_ID The ID of this torrent (in the download list for transmission bookeeping?).
TR_TORRENT_NAME The name of the torrent.

So my initial thought was that adding the GPG encryption or signing is as easy as adding a new pipeline that redirects the output to GPG. However, it turned out to be much more difficult than that. When the script is called by transmission, it doesn’t set the environment variable required by GPG and because of this GPG would failed to find the private key used to sign/encrypt the message and therefore failed to encrypt. After setting the environmental variable in the scripts the GPG encryption works correctly. Here is the working script with encryption and signing.

1
2
3
4
5
6
7
8
9
10
11
12
#!/bin/sh

SMTP_SERVER=YOUR_EMAIL_SMTP
MESSAGE="Hello!\n\n \tThis is a notification from transmission, $TR_TORRENT_NAME has been completed on $TR_TIME_LOCALTIME\n\n Thanks!"
SENDER="YOUR_EMAIL_USER_NAME"
RECEPIENT="EMAIL_TO_RECEIVE"
HOME="YOUR HOME DIRECTORY"
GPGHOME="YOUR .gpg DIRECTORY"
export HOME=$HOME
export GPGHOME=$GPGHOME

printf "$MESSAGE"|gpg --sign --encrypt --passphrase "your pass phrase" --batch --armor --encrypt -r recipient_pubkey_id |mailx -vr $SENDER -s "[Transmission] Torrent Has Been Downloaded" -S smtp=$SMTP_SERVER -S smtp-use-starttls -S smtp-auth=login -S smtp-auth-user=$SENDER -S smtp-auth-password="YOUR_EMAIL_PASSWORD" -S ssl-verify=ignore $RECEPIENT

From this week onward, I will be keeping notes for some of the issues I encountered during work or developing some of my personal works.

Here are notes from my development last week:

  • This might be a quite simple one, but I found it might be useful for future reference. In the development of TIA, previous developer used to disable the checkbox in a form. The result of this is the value of the disabled checkboxes were not submitted with the form. The previous developer opt to set the checkboxes separately which I don’t believe is an ideal solution. I thereby found another solution as follows:

    1
    <input type="checkbox" class="readonly" checked="checked" onclick="javascript:return false" />

    However, this is not enough in my case, I also need to have it gray out to make it looks like disabled. It turned out that this can be solved via simple CSS:

    1
    2
    3
    4
    5
    input.readonly {
    opacity : .50;
    filter : alpha(opacity=50); /* IE<9 */
    cursor : default;
    }

    Just make sure to give the checkbox proper class name.

  • This week, I was also asked to generate data from InterCAD. What I used to do was to copy the result of the query from database to a MS Excel worksheet and done the work there. This is especially easy when when there are not a lot of data. This time around, there were 16 days worth of data that I need to do statistics on. So, I decided to create a Bash script to do this task. The bash script is listed as follows, it is a pretty easy one and can be improved in the future:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    COUNTER=1
    TYPE=ACCIDENT
    while [ $COUNTER -lt 17 ]; do
    if [ $COUNTER -lt 10 ]
    then
    d='2017-05-0'
    else
    d='2017-05-'
    fi
    echo $d$COUNTER
    grep $d$COUNTER a.csv|grep $TYPE|wc -l
    let COUNTER=COUNTER+1
    done

    Basically, this time around, I exported the result of my query into a csv called a.csv and use grep to get what I want. This script can be improved in several ways:

    • Use awk instead of simple grep

    • Have the user input the type and the date from the command line.

      This will be for future improvements.

  • For my own project, I have set up a log watcher for the Flexget which I have installed on my Raspberry Pi 3. This log watcher runs 5 minutes after Flexget is ran and attach the error part of the log as an attachment and sent to my email address.

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    SMTP_SERVER=YOUR_EMAIL_SMTP
    RECEPIENT=EMAIL_TO_RECEIVE
    SENDER="YOUR_EMAIL_USER_NAME"
    TEMPLOG="TEMP_FILE_TO_STORE_ERROR_LOG" #Will be deleted email is sent
    DATE=$(date -I)
    MESSAGE=$(cat $HOME/flexget/flexget.log |
    grep -v INFO |
    grep -v VERBOSE |
    grep $DATE)
    CONTENT="Attached is the error log from today, please check!"
    if [ ! -z "$MESSAGE" ]
    then
    printf "$MESSAGE"|dd of=$HOME/errorlog
    printf "$CONTENT"|mailx -vr $SENDER -a $TEMPLOG -s "[TRANSMISSION ERROR] Transmission has encountered error" -S smtp=$SMTP_SERVER -S smtp-use-starttls -S smtp-auth=login -S smtp-auth-user=$SENDER -S smtp-auth-password="YOUR_EMAIL_PASSWORD" -S ssl-verify=ignore $RECEPIENT
    rm $TEMPLOG
    fi

    This script finds the error from flexget.log and write to TEMPLOG. If there is indeed errors exists in the log. The file will be attached and sent through the SMTP server to the address of your choice.

  • Over the weekend, I also started to learn React. React is a web library developed and used by Facebook. It is similar to Angular JS, but uses more straight rather than the TypeScript used by Angular JS. There were several notes that I would like to write down for future reference:

    • React are composed of components, however, there were several ways to declare a component in React.

      • The first way is the way that I would prefer, which is to use the ES6 class declaration, shown as follows:

        1
        2
        3
        4
        5
        6
        7
        8
        9
        import {Component} from 'react';
        export class SkiDayCount extends Component{
        constructor(props){
        super(props);
        }
        render(){
        return (<div></div>)
        }
        }
      • The second way is cleaner, it is a stateless way of declaring and declares the component as a function, it is shown as follows:

        1
        export const SkiDayCount = () => (<div></div>) 
      • The third way, however, is in the process of deprecation and is not recommended at this point, but it is still shown here:

        1
        2
        3
        4
        5
        6
        import React from 'react'
        export const SkiDayCount = React.createClass({
        render(){
        return (<div></div>)
        }
        })

        This will be deprecated in the future release and put here just for reference only.

    • To use the above mentioned newly created component in a page. You can use the following code:

      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      import React from 'react'
      import { render } from 'react-dom'
      //import the component, assuming it is in a folder called components and the source is called SkiDayCount.js
      import { SkiDayCount } from './components/SkiDayCount'

      window.React = React

      render(
      <SkiDayCount />,
      document.getElementById('react-container')
      )
  • These are the notes from last week, I will learn more about React this week and hopefully will come up with more notes for next week.

参考链接:

https://gist.github.com/sebleblanc/f5e4a635d0fc8b953df7

缘起

今天心血来潮,把之前部署在Github Pages上的Jekyll删掉了,完全换成了Hexo,并进行重做。主要还是想到自己在工作和平常生活中遇到的不少技术坑还是可以值得写下来记录一下。我的目标是每周一篇技术博客吧。
这周想要写的就是Flexget。身为一个海外党,在国外追新番还是比较不方便的:B站倒是能上,但在那上面看新番的时候缓存慢成狗也是蛮无语的。在跟龟速缓冲战斗了半年多之后,我决定转做一个下载党。我最初的想法是用Java自己造个轮子,后来发现自己水平太烂实在太难,就转向了找现成的方案。Google 出来不少自动追番的相关内容。大部分都提到了Flexget这个东西。在看了几篇介绍之后。我就决定使用Flexget来实现自动追番。由于Google上关于Flexget的文章大都是13年的,而Flexget这软件已经更新了很多次了,有些配置文件已经不在适用,为了方便后人折腾,我就决定将我折腾的过程记录下来。

准备

以下是安装和折腾 Flexget 的前提条件:

  1. 树莓派或任意 Linux 系统的主机。(推荐至少1GB内存,因为 Flexget 的依赖会比较多)
  2. 树莓派或 Linux 主机上已安装有至少一个 BT 下载工具。我自己用的是装在路由器上的Transmission,不想折腾的话,也可以采用这个方案。当然 Flexget 可以和很多下载工具整合,不光是 Transmission ,不过你需要自己查询它的英文文档。
  3. 树莓派或 Linux 主机上已安装有 Python 和 Pip。
  4. 懂一点 YAML。Flexget的配置文件全部是基于 YAML 格式的,这个格式学起来其实也不算复杂,所以懂一点就足够。
  5. 一个提供种子或者磁力链接的 RSS 源。这里我就不列举了,一般看动漫的都会知道哪能找到这样的资源。

折腾

这个 Flexget 折腾起来难度不是很大,下面是一个简单的教程

  • 安装

    • 官方的教程强烈推荐将 Flexget 安装到一个独立的 Python 环境中以免它的依赖跟本地依赖冲突。我个人也是这样做的:

    • 首先要做的是安装虚拟环境废话。这个只需要此命令即可实现:pip install virtualenv

    • 安装好 virtualenv 后可以在本地创建一个文件夹作为 Flexget 的独立环境:

      mkdir flextget && cd flexget

    • 将刚刚建立的 flexget 文件夹转成独立的 Python 环境:virtualenv .

    • 现在 Flexget 文件夹中就有了一个完整的 Python 环境,我们需要先激活这个环境,然后再开始安装Flexget:

      1
      2
      3
      source bin/activate #激活独立 Python 环境
      pip install flexget #安装flexget, 需要一段时间,因为依赖比较多
      deactivate #安装完成后,解除对当前独立 Python 环境的激活
    • 这个时候,我们已经将 Flexget 安装完成了,但由于现在还没有配置文件,启动 Flexget 时会报错

      1
      bin/flexget execute #运行Flexget, 会返回 Failed to find configuration file config.yml
  • 配置

    • Flexget 的配置文件采用的 YAML 语言,并且非常灵活,可以做到在一个文件中加载另一个文件。

    • 配置文件的核心是 config.yml, 下面是我自己的 config.yml 以及注释:

      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      16
      17
      18
      19
      20
      21
      22
      templates:
      global:
      transmission:
      host: 192.168.1.1
      port: 9091
      clean_transmission:
      host: 192.168.1.1
      port: 9091
      finished_for: 2 days
      enabled: Yes
      manipulate:
      - description:
      remove: yes

      tasks:
      bangumi:
      rss:
      url: 'https://example.com/rss.xml'
      urlfix: true
      include:
      - regexp.yml
      - notify.yml
    • 我的config.yml中包含了以下的部分:

      • 这里要注意的是flexget的config.yml是由一个个对插件的配置组成的。有个比较坑的地方是,配置文件中的缩进必须使用空格,同时,缩进必须为2的倍数而不能随机。

      • 这个配置文件中使用的transmission插件需要额外的依赖,可使用如下命令安装 (需先按上文的说明进入 flexget 的 Python 环境):

        1
        pip install transmissionrpc
      • templates 这个部分是模板插件,global 代表这个模板适用于所有 Flexget的下载任务。global 中的具体属性是用于设置 transmission 的。transmission部分中的host和port分别代表transmission所在服务器的IP和端口。clean_transmission插件可以自动清除已完成的transmission任务。这里的finished_for:2 days 意味着flexget会在一个BT下载任务完成两天后将其从tranmission的任务列表中清除。Manipulate 插件部分,我的设置是在匹配正则表达式钱将 RSS 条目中的description部分清除。这样可以能更精确的匹配。

      • template 之后是 tasks 插件的配置。tasks 插件是 flexget 的核心部分,在 flexget 中,每一个 task 代表了一个下载任务,可以使用不同的 RSS 源以及储存位置。上面的配置文件中只有一个名为 bangumi 的任务,这个任务使用了rss,urlfix,以及 include 插件。rss插件部分定义了用于订阅的 rss 文件的地址。urlfix插件可以修复部分无效url。include插件包含了两个配置文件,regexp.yml 是我用于储存新番正则表达式的配置文件,而 notify.yml 的配置文件则是用于flexget自动发送通知邮件。

    • regexp.yml 的内容如下:

      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      regexp:
      accept:
      - '.+喵萌茶会字幕组.+南镰仓高校女子自行车社.+\d{1,}.+720p'
      - '.+RH字幕組.+清恋.+\d{1,}.+720p'
      - '.+轻之国度X动漫国.+Urara迷路帖.+\d{1,}.+720p'
      - '.+極影字幕社.+廢天使加百列.+\d{1,}.+720p'
      - '.+澄空学园.+人渣的本心.+第\d{1,}话.+720p'
      - '.+動漫國字幕組.+政宗君的復仇.+\d{1,}.+720P.+繁體'
      - '.+動漫國字幕組.+為美好的世界獻上祝福.+\d{1,}.+720P.+繁體'
      from: title
      • 此文件中主要包含了regexp插件的配置,独立出来的目的是方便更新。请注意此文件中第一行中的 regexp 有一定的缩进,这主要是为了符合前文提到的 flexget 的规则。
      • accept 部分是 flexget 会接受的正则表达式,from:title 则表示当且仅当 title 匹配正则表达式的时候才会匹配。
    • notify.yml 的内容如下:

      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      notify:    
      task:
      via:
      - email:
      from:
      to:
      smtp_host:
      smtp_port:
      smtp_tls:
      smtp_username: your_username
      smtp_password: your_password
      • notify 插件同样是按任务分类,在默认的情况下,会在种子匹配被添加到 transmission 的下载里时才会发送一封邮件。
  • 运行

    • 在以上配置完成之后,建议先使用 Flexget 自带的验证功能验证配置文件的格式,毕竟配置文件中坑比较多。可使用如下命令:flexget check 。如果验证有错的话,该功能会具体的指出错误出在哪一行,读者可自行参考错误信息完成对配置文件的更正。

    • 验证完配置的格式之后,如果需保证正确,flexget 还提供了一种 test execute 的方式来测试是否真的能从rss文件提取,该运行方式不会将匹配的种子发送到 BT 下载器,也不会发送邮件,但会告诉你如果真的运行命令,会出现的结果,这对于检验 rss 文件的有效性非常有用。运行方式如下:

      1
      flexget --test execute
    • 如果以上命令的运行结果和你所想的一样,那么你就可以正式的运行flexget了。只需去掉 –test 部分即可:

      1
      flexget execute
    • 此命令如果成功运行则意味着我们可以想办法让flexget定期运行,以实现自动追番的目的。我这里采用的是 Linux 自带的 crontab 的方法。

      • 首先,运行 crontab -e , 此命令会在默认修改器中打开crontab文件

      • 之后,需要在文件的最后加入一下内容:

        1
        00 8,12,17,19,21 * * * /path/to/flexget/bin/flexget --cron -c /path/to/flexget/config.yml execute

        我目前的配置文件是在每天的8点,12点,17点,19点和21点的时候自动执行flexget任务。由于cron的特殊性,这里必须使用完整的flexget路径并使用 –cron 告诉flexget使用cron模式以及通过 -c 指定配置文件的位置。