Kevin Jiang's Technical Blog

As a developer, you will sometimes face weird problems, it is important to come up with reliable and repeatable ways to solve this problems, so when such problems come up again, you would be able to find a solution easier. As for myself, one of the tools that I found most useful on Unix-like system is jq, which is a tool to process json files. Let me demonstrate how I use this tool to solve some problems I encountered during work.

Problem: Convert a JSON file to CSV

Example JSON

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

[
  {
    "title": "This is a song",
    "artist": "This is an artist",
    "album": "This is an album",
    "year": 1989
  },
  {
    "title": "This is a song",
    "artist": "This is an artist",
    "album": "This is an album",
    "year": 1989
  },
  {
    "title": "This is a song",
    "artist": "This is an artist",
    "album": "This is an album",
    "year": 1989
  }
]

JQ code to generate csv:

1

jq -r '(.[0] | keys_unsorted) as $keys | ([$keys] + map([.[ $keys[] ]])) [] | @csv'

Resulting CSV:

1
2
3
4

"title","artist","album","year"
"This is a song","This is an artist","This is an album",1989
"This is a song","This is an artist","This is an album",1989
"This is a song","This is an artist","This is an album",1989

Problem: Aggregate JSON object.

Example JSON:

1
2
3
4
5

{
  "A": [{ "Name": "A1" }, { "Name": "A2" }],
  "B": [{ "Name": "B1" }, { "Name": "B2" }],
  "C": [{ "Name": "C" }]
}

The goal is to produce something like below:

1

{ "A": ["A1", "A2"], "B": ["B1", "B2"], "C": ["C"] }

It transform the object and aggregate (or compress?) them by “Name” property, I know this can be easily done with JavaScript, but jq and bash seems like more widely used and will come in handy when JavaScript is not available.

The jq code I came up with is as follows:

1

jq '[keys_unsorted[] as $k|{($k): [.[$k][]|.Name]}]|add'

References:

• https://stackoverflow.com/a/32967407

What would be the easiest way to remove all but one word from the end of each line in Vim?

Today, I found a challenge on vimgolf and thought it might be interesting to share my solutions.

Here is the original file

1
2
3
4

abcd 1 erfg 7721231
acd 2 erfgasd 324321
acd 3 erfgsdasd 23432
abcd 5 erfgdsad 123556

Here is the desired result:

1
2
3
4

7721231
324321
23432
123556

The challenge is quite straightforward, delete all but the last characters in the file. I found several ways to tackle this challenge, so let me show you all:

Take 1: Use macro only

Record a macro, and play it back, so the keystrokes would be

1
2
3
4
5
6
7
8

qa $bd0j q 4@a <cr>

qa starts recording macro to register a
$ move cursor to the end of the line
d0 means *d*elete to the beginning of the line
j move cursor down
q finish macro
4@a repeat the macro in register a 4 times

Take 2: Use regex and %norm

It’s quite obvious that all we want to keep from the original files are the numbers. So the regex would be simple to come up with, like something as simple as /\d\+$<cr> will do. Once you type this into vim, all the numbers at the end of the line will be highlighted. Next you can do:

1
2
3
4
5

:%norm dn <cr>

% means applying to the whole file,
norm means in execute following command in normal mode
dn means *d*elete to *n*ext match

Take 3: No regex pure %norm

This is the fastest way I can come up with, still, not as fast as the top answers on Vimgolf but it is decent in my opinion. Being slightly different from the option above, it is still using %norm though:

1
2
3
4
5

%norm $bd0 <cr>  

% means applying to the whole file,
$ move cursor to the end of the line
d0 means *d*elete to the beginning of the line

Takeaways:

• norm is a quite powerful, and can be used to achieve complex stuff that can be otherwise achieved through a macro.
• d the delete command is useful in many unexpected ways, besides dn and d0 command mentioned above which deletes to the next match and deletes to the beginning of the line respectively. An additional useful variation of d command is d4/x where 4/x meaning 4th occurrence of x.

This week, I was tasked with creating basic infrastructure for one of our newwebsites. We use Fastly for our CDN and New Relic as a log aggregation tool, most of our infrastructure is setup using Terraform, which is a popular infrastructure as code (IaC) platform. Terraform supports Fastly through a custom plugin, which also has the capability of New Relic. We need to customize the format of New Relic so that we can find the logs easily in New Relic. As expected, the plugin actually provide a pretty straight forward to accomplish this – what you need to do is to come up with a proper json string that align with the format that Fastly provided in their document. This seemingly straightforward task ended up took me some time to debug.

In terraform, you can create objects, and an object can be converted to string by using jsonencode function. This is where the pitfall comes in: Fastly’s formatting document said they have a option called “%>s” which represents the final status of the fastly request. From my perspective, it would definitely be helpful if we can include this when we send the log to Fastly. So I added this to my formatting object, and then jsonencode the object. To my suprise, I get an error saying that Fastly failed to parse the formatting option I created, which is quite strange. I then started to debug, I export TF_LOG=”DEBUG”, which asked Terraform to give me all the debug logs. To my suprise, I found that the “%>s” was jsonencode to “%\u003es” by terraform, and thus causing the error. How come “>” sign is encoded in Terraform? It turned out it’s for backward compatibility with ealier edition of Terraform. And according to their documentation

When encoding strings, this function escapes some characters using Unicode escape sequences: replacing <, >, &, U+2028, and U+2029 with \u003c, \u003e, \u0026, \u2028, and \u2029. This is to preserve compatibility with Terraform 0.11 behavior.

they encode much more characters for you without you realizing it. I really hope they can provide a way to opt-out of this feature or make this an opt-in, it will make things way easier for developers.

Recently, I have been eager to get the data saver on Android to work automatically for me. My goal is to have Tasker automatically enable the built-in data saver when my data is low. Before I came up with this idea, I have search the web to see if there is any solutions exist already so I don’t have to reinventing the wheel. Unfortunately, I couldn’t find anything on this topic so I have to create my own solution. Upon 1 week of testing, things to work as I expected so I decided to share in this article how I did it.

Prerequisite

• An rooted Android phone – Unfortunately, the solution I came up with only works on rooted Android phone, I’m running Android 9.0.
• A way to get to know you current data usage – Some carriers allow you to get you data usage through SMS, some might only allow you to use USSD to get data usage. I am only covering the case of using SMS here (my case).

How-to

The data saver in Android is basically something that limits all the background data usage. After some digging on the Internet, I found that you can turn the data saver on with the following command cmd netpolicy set restrict-background true (requires root). Then it is pretty easy to have the data saver turned on automatically when the data is low. My way of doing this is as follows:

• Set a global variable as threshold for triggering the data saver.
• Send an SMS to my carrier every morning at 8 AM, parsing the SMS to get my remaining data.
• If my data remaining is lower than the threshold then turn on the data saver whenever mobile data is in use. This is done by using the run shell action and run the command mentioned above.
• Turn data saver off automatically when I am connected to WiFi (Not necessary but add it just in case).

It is quite easy to do this when you carrier allows you to query data usage using SMS, with USSD however, things is not that easy and unfortunately, I haven’t figure out a way yet.

Recently, I decided to convert my QEMU based virtual machine installed on my Manjaro Linux to the VirtualBox format. The reason behind this is that I would like to be able to use the same VM across different host system (specifically Manjaro Linux and Windows 7). It is not a easy thing to do, so I decided to document it for future references.

Prerequisite?

• An existing image created using QEMU (My VM file end with .img, for example)
• VirtualBox

How to?

First thing first, you would need to convert the QEMU image (extension img) to raw format, this can be done by issuing the following command:

qemu-img convert Windows7-compressed.img -O raw Windows7.raw

This will generate a raw image. Note that this newly generated file might be a lot larger than the file it based on, this is because the img file allocates space on as-is basis.

After you get the raw image, it’s time to convert it to VDI format (which is used by VirtualBox). You can do this by running:

VBoxManage convertfromraw Windows7.raw --format vdi Windows7.vdi

Then, it is recommended to compact the image:

VBoxManage modifyhd Windows7.vdi --compact

So after the previous step, you will have a working VirtualBox image, but if you boot it from VirtualBox, it might not work.

Gotchas!

In my case, what I was trying to convert was a Windows 7 VM, and when I finished the above steps and try to boot the VM, I got a BSOD. My feeling is that there were some default that QEMU used that doesn’t work for a newly created machine in VirtualBox. I tweak the following settings in the newly created VirtualBox VM:

• Delete the auto created SATA controller and change it IDE controller.

It turned out, after doing that, everything works as expected.

Why Tasker?

This is probably an article that is long overdue to me personally. I have been an Android user since 2011, started with my Nexus S that I bought for use in college. It seems to me, the app Tasker had been very famous in the Andorid community, especially among users who know how to program.
For those who have never heard of this app, this is a powerful app that allows you to automate almost anything that you can think of on the Android system.For me,It was not until August of last year when I bought the tasker app that I started to realize how powerful this app is. I am really not a big fan of its old school UI and design so I didn’t start using it until this year. After uisng it for a while, I did come to realize that there is no other apps close to it, if I were to move to iOS, this is probably among one the apps that I will miss. In this article I will explain how I used takser to automate things for myself, it is some boring stuff, but it did become something that I’m using everyday.

What can you use it for?

I will list one of the most used Tasks/Profiles that I created and used in this app:

Turn off/on ADB when using certain apps.

This is one of the easy ones that I found very useful. It is not uncommon nowadays for some apps to require you to turn off your ADB (Android Debugging Bridge) settings when you are using them, which to me, is quite annoying. So naturally, I created a Tasker profile together with a task to automate this. The trick here is you need to have root access, otherwise you are pretty much out of luck for this particular example. To create such a task, assuming you already have proper root access, go to the TASKS page in the app and click on the add icon, choose a name you like, and then you can create your first task! Think of a task as the things you want Tasker to do for you. In this particular case, my goal is simple: turn off ADB if it was not already off, turn it on if it was off. This way we can have one task that a turns off the ADB when you turns on the app; when you turns the app off, ADB will be switch back to on.

It is clear that we will need a global variable that holds the status of the adb status, to do this add an action: click on the button add icon while you are in the “Task Edit” page, and filter based on Shell, you will see a “Run Shell” as a result. Click on it, and in the “Command” input, enter “settings get global adb_enabled” and in “Store Output In” input, choose a global variable name you would like to use to hold the status of the current ADB status, just remember that you need to use all caps for the name for tasker to know it is an global variable. Also remember to check the “Use Root” checkbox. After this step, things will be simple, just add the if and else condition like I mentioned before, set adb_enabled to 0 if it is 1 and set it to 1 if it is 0, and after that don’t forget to set the global variable again.

TL;DR Here is the XML, that you can directly import to your Tasker if you don’t want to create it yourself:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61

<TaskerData sr="" dvi="1" tv="5.2.bf">
    <Task sr="task3">
        <cdate>1526421645228</cdate>
        <edate>1529892193595</edate>
        <id>3</id>
        <nme>adb_auto</nme>
        <pri>1006</pri>
        <Action sr="act0" ve="7">
            <code>123</code>
            <Str sr="arg0" ve="3">settings get global adb_enabled</Str>
            <Int sr="arg1" val="0"/>
            <Int sr="arg2" val="1"/>
            <Str sr="arg3" ve="3">%ADB_STATUS</Str>
            <Str sr="arg4" ve="3"/>
            <Str sr="arg5" ve="3"/>
        </Action>
        <Action sr="act1" ve="7">
            <code>37</code>
            <ConditionList sr="if">
                <Condition sr="c0" ve="3">
                    <lhs>%ADB_STATUS</lhs>
                    <op>0</op>
                    <rhs>1</rhs>
                </Condition>
            </ConditionList>
        </Action>
        <Action sr="act2" ve="7">
            <code>123</code>
            <Str sr="arg0" ve="3">settings put global adb_enabled 0</Str>
            <Int sr="arg1" val="0"/>
            <Int sr="arg2" val="1"/>
            <Str sr="arg3" ve="3">%ADB_STATUS</Str>
            <Str sr="arg4" ve="3"/>
            <Str sr="arg5" ve="3"/>
        </Action>
        <Action sr="act3" ve="7">
            <code>43</code>
        </Action>
        <Action sr="act4" ve="7">
            <code>123</code>
            <Str sr="arg0" ve="3">settings put global adb_enabled 1</Str>
            <Int sr="arg1" val="0"/>
            <Int sr="arg2" val="1"/>
            <Str sr="arg3" ve="3">%ADB_STATUS</Str>
            <Str sr="arg4" ve="3"/>
            <Str sr="arg5" ve="3"/>
        </Action>
        <Action sr="act5" ve="7">
            <code>38</code>
        </Action>
        <Action sr="act6" ve="7">
            <code>123</code>
            <Str sr="arg0" ve="3">settings get global adb_enabled</Str>
            <Int sr="arg1" val="0"/>
            <Int sr="arg2" val="1"/>
            <Str sr="arg3" ve="3">%ADB_STATUS</Str>
            <Str sr="arg4" ve="3"/>
            <Str sr="arg5" ve="3"/>
        </Action>
    </Task>
</TaskerData>

To import, you need to save the above snippet to XML on your phone. In tasker app, click and hold the “TASKS” tab header, select import and choose the file

To run the task based on your predefined conditions, go to “PROFILES” tab, click on add button in the button, choose the application you want, and then, chose the task you imported and you are good to go! Now the app won’t say you can’t use it because you have ADB turned on, Tasker will turned adb off when you are using the app, and turn it on when you are not using it! :)

What is Google App Scripts?

Before explaining what I did with Google App Scripts, let me explain what is Google App Scripts. It is basically a scripting engine that let you access all kinds of Google App features (including Gmail, Google Calendar, Google Docs, etc.). There are other ways to access these features but Google App Script might be the easist way out there. It comes with an online editor (you can work offline as well), and Google can trigger your scripts at certain time based on your need. Oh, forgot to mention, Google App Script uses JavaScript which is one of the most popular languages right now (Sadly, it doesn’t support ES6 syntax as of writing).

Why use Google App Scripts?

I have a habit of archiving my Emails every week on Mondays. It is easy enough to do in the gmail Web interface but I do need to remember to do it. That make me think if I can find a good way to automate this. I would always think of using some kind of API first, but it seems to me that it wouldn’t worth it to spend time to make a app just for the purpose of archiving emails. Then I came across Google App Scripts, which turned out to be exactly what I need to automate this.

What I came up with

Here is what I came up with to archive emails that are one-week-old every Monday at midnight:

1
2
3
4
5
6
7
8
9
10
11

function gmailAutoarchive() {
  var maxDate = new Date();
  // Archive all threads in inbox whose last message date is older than today.
  var threads = GmailApp.getInboxThreads();
  for(var i in threads){
    var thread = threads[i];
    if (thread.getLastMessageDate()<maxDate){
      thread.moveToArchive();
    }
  }
}

You will be able to use this script by putting it into the google script editor located here.

To have Google auto run your scripts, you need to add the trigger which can be found under Edit –> Current project’s triggers.

Variation

After creating this script, I created another script as a variation to this script that move messages with certain labels to trash.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

function gmailAutoRemove() {
  var maxDate = new Date();
  maxDate.setDate(maxDate.getDate() - 30); // what was the date at that time?
  var label = [GmailApp.getUserLabelByName("label1"), GmailApp.getUserLabelByName("label2"), GmailApp.getUserLabelByName("label3")];
  label.forEach(removeByLabel(maxDate));
}


function removeByLabel(maxDate){
  return function(label){
   var threads = label.getThreads(0, 100);
    for(var i in threads){
      var thread = threads[i];
      if (thread.getLastMessageDate() < maxDate){
        thread.moveToTrash();
      }
    }
  }
}

References: Auto archive emails in Gmail after 2 days

Fun with GPG

This week, I was having making some great progress in understanding how GPG works either locally or through email. The original intention to do all this was because I would like my router to send me notification whenever the tranmission finish download the torrent. This may be simple as it sounds and it had been working correctly for me for several months since I create the initial script.

This week, however, I decided to do something special. I would like the router to sign/encrypt the message it sends to me. I am not sure why I need that, but anyways, I did get to learn a lot through the process.

Here is my original script, it simply uses the mailx program that installed on the router and sends the email through SMTP. Looks quite simple, it pretty much is the same as the script I have shown in the previous post:

1
2
3
4
5
6
7
8

#!/bin/sh

SMTP_SERVER=YOUR_EMAIL_SMTP
MESSAGE="Hello!\n\n \tThis is a notification from transmission, $TR_TORRENT_NAME has been completed on $TR_TIME_LOCALTIME\n\n Thanks!"
SENDER="YOUR_EMAIL_USER_NAME"
RECEPIENT="EMAIL_TO_RECEIVE"

printf "$MESSAGE"|mailx -vr $SENDER -s "[Transmission] Torrent Has Been Downloaded" -S smtp=$SMTP_SERVER -S smtp-use-starttls -S smtp-auth=login -S smtp-auth-user=$SENDER -S smtp-auth-password="YOUR_EMAIL_PASSWORD" -S ssl-verify=ignore $RECEPIENT

A little more explanation here, there are two variables preset by Transmission. $TR_TORRENT_NAME, is the name of the torrent that has just been finished. $TR_TIME_LOCALTIME is the time when the download was finished. There were several other environment variables set by transmission also. Here is a list of them^1
Note: The meaning of this variable are not explicitly documented in the wiki, and I guess the meaning based on my understanding.

So my initial thought was that adding the GPG encryption or signing is as easy as adding a new pipeline that redirects the output to GPG. However, it turned out to be much more difficult than that. When the script is called by transmission, it doesn’t set the environment variable required by GPG and because of this GPG would failed to find the private key used to sign/encrypt the message and therefore failed to encrypt. After setting the environmental variable in the scripts the GPG encryption works correctly. Here is the working script with encryption and signing.

1
2
3
4
5
6
7
8
9
10
11
12

#!/bin/sh

SMTP_SERVER=YOUR_EMAIL_SMTP
MESSAGE="Hello!\n\n \tThis is a notification from transmission, $TR_TORRENT_NAME has been completed on $TR_TIME_LOCALTIME\n\n Thanks!"
SENDER="YOUR_EMAIL_USER_NAME"
RECEPIENT="EMAIL_TO_RECEIVE"
HOME="YOUR HOME DIRECTORY"
GPGHOME="YOUR .gpg DIRECTORY"
export HOME=$HOME
export GPGHOME=$GPGHOME

printf "$MESSAGE"|gpg --sign --encrypt --passphrase "your pass phrase" --batch --armor --encrypt  -r recipient_pubkey_id |mailx -vr $SENDER -s "[Transmission] Torrent Has Been Downloaded" -S smtp=$SMTP_SERVER -S smtp-use-starttls -S smtp-auth=login -S smtp-auth-user=$SENDER -S smtp-auth-password="YOUR_EMAIL_PASSWORD" -S ssl-verify=ignore $RECEPIENT

From this week onward, I will be keeping notes for some of the issues I encountered during work or developing some of my personal works.

Here are notes from my development last week:

• This might be a quite simple one, but I found it might be useful for future reference. In the development of TIA, previous developer used to disable the checkbox in a form. The result of this is the value of the disabled checkboxes were not submitted with the form. The previous developer opt to set the checkboxes separately which I don’t believe is an ideal solution. I thereby found another solution as follows:

  1	<input type="checkbox" class="readonly" checked="checked" onclick="javascript:return false" />

  However, this is not enough in my case, I also need to have it gray out to make it looks like disabled. It turned out that this can be solved via simple CSS:

  1 2 3 4 5

  input.readonly { opacity : .50; filter : alpha(opacity=50); /* IE<9 */ cursor : default; }

  Just make sure to give the checkbox proper class name.

• This week, I was also asked to generate data from InterCAD. What I used to do was to copy the result of the query from database to a MS Excel worksheet and done the work there. This is especially easy when when there are not a lot of data. This time around, there were 16 days worth of data that I need to do statistics on. So, I decided to create a Bash script to do this task. The bash script is listed as follows, it is a pretty easy one and can be improved in the future:

  1 2 3 4 5 6 7 8 9 10 11 12 13

  COUNTER=1 TYPE=ACCIDENT while [ $COUNTER -lt 17 ]; do if [ $COUNTER -lt 10 ] then d='2017-05-0' else d='2017-05-' fi echo $d$COUNTER grep $d$COUNTER a.csv|grep $TYPE|wc -l let COUNTER=COUNTER+1 done

  Basically, this time around, I exported the result of my query into a csv called a.csv and use grep to get what I want. This script can be improved in several ways:

  • Use awk instead of simple grep

  • Have the user input the type and the date from the command line.

    This will be for future improvements.

• For my own project, I have set up a log watcher for the Flexget which I have installed on my Raspberry Pi 3. This log watcher runs 5 minutes after Flexget is ran and attach the error part of the log as an attachment and sent to my email address.

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

  SMTP_SERVER=YOUR_EMAIL_SMTP RECEPIENT=EMAIL_TO_RECEIVE SENDER="YOUR_EMAIL_USER_NAME" TEMPLOG="TEMP_FILE_TO_STORE_ERROR_LOG" #Will be deleted email is sent DATE=$(date -I) MESSAGE=$(cat $HOME/flexget/flexget.log | grep -v INFO | grep -v VERBOSE | grep $DATE) CONTENT="Attached is the error log from today, please check!" if [ ! -z "$MESSAGE" ] then printf "$MESSAGE"|dd of=$HOME/errorlog printf "$CONTENT"|mailx -vr $SENDER -a $TEMPLOG -s "[TRANSMISSION ERROR] Transmission has encountered error" -S smtp=$SMTP_SERVER -S smtp-use-starttls -S smtp-auth=login -S smtp-auth-user=$SENDER -S smtp-auth-password="YOUR_EMAIL_PASSWORD" -S ssl-verify=ignore $RECEPIENT rm $TEMPLOG fi

  This script finds the error from flexget.log and write to TEMPLOG. If there is indeed errors exists in the log. The file will be attached and sent through the SMTP server to the address of your choice.

• Over the weekend, I also started to learn React. React is a web library developed and used by Facebook. It is similar to Angular JS, but uses more straight rather than the TypeScript used by Angular JS. There were several notes that I would like to write down for future reference:

  • React are composed of components, however, there were several ways to declare a component in React.

    • The first way is the way that I would prefer, which is to use the ES6 class declaration, shown as follows:

      1 2 3 4 5 6 7 8 9

      import {Component} from 'react'; export class SkiDayCount extends Component{ constructor(props){ super(props); } render(){ return (<div></div>) } }

    • The second way is cleaner, it is a stateless way of declaring and declares the component as a function, it is shown as follows:

      1	export const SkiDayCount = () => (<div></div>)

    • The third way, however, is in the process of deprecation and is not recommended at this point, but it is still shown here:

      1 2 3 4 5 6

      import React from 'react' export const SkiDayCount = React.createClass({ render(){ return (<div></div>) } })

      This will be deprecated in the future release and put here just for reference only.

  • To use the above mentioned newly created component in a page. You can use the following code:

    1 2 3 4 5 6 7 8 9 10 11

    import React from 'react' import { render } from 'react-dom' //import the component, assuming it is in a folder called components and the source is called SkiDayCount.js import { SkiDayCount } from './components/SkiDayCount' window.React = React render( <SkiDayCount />, document.getElementById('react-container') )

• These are the notes from last week, I will learn more about React this week and hopefully will come up with more notes for next week.

参考链接：

https://gist.github.com/sebleblanc/f5e4a635d0fc8b953df7

缘起

今天心血来潮，把之前部署在Github Pages上的Jekyll删掉了，完全换成了Hexo，并进行重做。主要还是想到自己在工作和平常生活中遇到的不少技术坑还是可以值得写下来记录一下。我的目标是每周一篇技术博客吧。
这周想要写的就是Flexget。身为一个海外党，在国外追新番还是比较不方便的：B站倒是能上，但在那上面看新番的时候缓存慢成狗也是蛮无语的。在跟龟速缓冲战斗了半年多之后，我决定转做一个下载党。我最初的想法是用Java自己造个轮子，后来发现自己水平太烂实在太难，就转向了找现成的方案。Google 出来不少自动追番的相关内容。大部分都提到了Flexget这个东西。在看了几篇介绍之后。我就决定使用Flexget来实现自动追番。由于Google上关于Flexget的文章大都是13年的，而Flexget这软件已经更新了很多次了，有些配置文件已经不在适用，为了方便后人折腾，我就决定将我折腾的过程记录下来。

准备

以下是安装和折腾 Flexget 的前提条件：

1. 树莓派或任意 Linux 系统的主机。（推荐至少1GB内存，因为 Flexget 的依赖会比较多)
2. 树莓派或 Linux 主机上已安装有至少一个 BT 下载工具。我自己用的是装在路由器上的Transmission，不想折腾的话，也可以采用这个方案。当然 Flexget 可以和很多下载工具整合，不光是 Transmission ，不过你需要自己查询它的英文文档。
3. 树莓派或 Linux 主机上已安装有 Python 和 Pip。
4. 懂一点 YAML。Flexget的配置文件全部是基于 YAML 格式的，这个格式学起来其实也不算复杂，所以懂一点就足够。
5. 一个提供种子或者磁力链接的 RSS 源。这里我就不列举了，一般看动漫的都会知道哪能找到这样的资源。

折腾

这个 Flexget 折腾起来难度不是很大，下面是一个简单的教程

• 安装

  • 官方的教程强烈推荐将 Flexget 安装到一个独立的 Python 环境中以免它的依赖跟本地依赖冲突。我个人也是这样做的:

  • 首先要做的是安装虚拟环境废话。这个只需要此命令即可实现：pip install virtualenv

  • 安装好 virtualenv 后可以在本地创建一个文件夹作为 Flexget 的独立环境:

    mkdir flextget && cd flexget

  • 将刚刚建立的 flexget 文件夹转成独立的 Python 环境：virtualenv .

  • 现在 Flexget 文件夹中就有了一个完整的 Python 环境，我们需要先激活这个环境，然后再开始安装Flexget：

    1 2 3

    source bin/activate #激活独立 Python 环境 pip install flexget #安装flexget, 需要一段时间，因为依赖比较多 deactivate #安装完成后，解除对当前独立 Python 环境的激活

  • 这个时候，我们已经将 Flexget 安装完成了，但由于现在还没有配置文件，启动 Flexget 时会报错

    1	bin/flexget execute #运行Flexget, 会返回 Failed to find configuration file config.yml

• 配置

  • Flexget 的配置文件采用的 YAML 语言，并且非常灵活，可以做到在一个文件中加载另一个文件。

  • 配置文件的核心是 config.yml, 下面是我自己的 config.yml 以及注释：

    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22

    templates: global: transmission: host: 192.168.1.1 port: 9091 clean_transmission: host: 192.168.1.1 port: 9091 finished_for: 2 days enabled: Yes manipulate: - description: remove: yes tasks: bangumi: rss: url: 'https://example.com/rss.xml' urlfix: true include: - regexp.yml - notify.yml

  • 我的config.yml中包含了以下的部分：

    • 这里要注意的是flexget的config.yml是由一个个对插件的配置组成的。有个比较坑的地方是，配置文件中的缩进必须使用空格，同时，缩进必须为2的倍数而不能随机。

    • 这个配置文件中使用的transmission插件需要额外的依赖，可使用如下命令安装 (需先按上文的说明进入 flexget 的 Python 环境)：

      1	pip install transmissionrpc

    • templates 这个部分是模板插件，global 代表这个模板适用于所有 Flexget的下载任务。global 中的具体属性是用于设置 transmission 的。transmission部分中的host和port分别代表transmission所在服务器的IP和端口。clean_transmission插件可以自动清除已完成的transmission任务。这里的finished_for:2 days 意味着flexget会在一个BT下载任务完成两天后将其从tranmission的任务列表中清除。Manipulate 插件部分，我的设置是在匹配正则表达式钱将 RSS 条目中的description部分清除。这样可以能更精确的匹配。

    • template 之后是 tasks 插件的配置。tasks 插件是 flexget 的核心部分，在 flexget 中，每一个 task 代表了一个下载任务，可以使用不同的 RSS 源以及储存位置。上面的配置文件中只有一个名为 bangumi 的任务，这个任务使用了rss，urlfix，以及 include 插件。rss插件部分定义了用于订阅的 rss 文件的地址。urlfix插件可以修复部分无效url。include插件包含了两个配置文件，regexp.yml 是我用于储存新番正则表达式的配置文件，而 notify.yml 的配置文件则是用于flexget自动发送通知邮件。

  • regexp.yml 的内容如下：

    1 2 3 4 5 6 7 8 9 10

    regexp: accept: - '.+喵萌茶会字幕组.+南镰仓高校女子自行车社.+\d{1,}.+720p' - '.+RH字幕組.+清恋.+\d{1,}.+720p' - '.+轻之国度X动漫国.+Urara迷路帖.+\d{1,}.+720p' - '.+極影字幕社.+廢天使加百列.+\d{1,}.+720p' - '.+澄空学园.+人渣的本心.+第\d{1,}话.+720p' - '.+動漫國字幕組.+政宗君的復仇.+\d{1,}.+720P.+繁體' - '.+動漫國字幕組.+為美好的世界獻上祝福.+\d{1,}.+720P.+繁體' from: title

    • 此文件中主要包含了regexp插件的配置，独立出来的目的是方便更新。请注意此文件中第一行中的 regexp 有一定的缩进，这主要是为了符合前文提到的 flexget 的规则。
    • accept 部分是 flexget 会接受的正则表达式，from：title 则表示当且仅当 title 匹配正则表达式的时候才会匹配。

  • notify.yml 的内容如下：

    1 2 3 4 5 6 7 8 9 10 11

    notify: task: via: - email: from: to: smtp_host: smtp_port: smtp_tls: smtp_username: your_username smtp_password: your_password

    • notify 插件同样是按任务分类，在默认的情况下，会在种子匹配被添加到 transmission 的下载里时才会发送一封邮件。

• 运行

  • 在以上配置完成之后，建议先使用 Flexget 自带的验证功能验证配置文件的格式，毕竟配置文件中坑比较多。可使用如下命令：flexget check 。如果验证有错的话，该功能会具体的指出错误出在哪一行，读者可自行参考错误信息完成对配置文件的更正。

  • 验证完配置的格式之后，如果需保证正确，flexget 还提供了一种 test execute 的方式来测试是否真的能从rss文件提取，该运行方式不会将匹配的种子发送到 BT 下载器，也不会发送邮件，但会告诉你如果真的运行命令，会出现的结果，这对于检验 rss 文件的有效性非常有用。运行方式如下:

    1	flexget --test execute

  • 如果以上命令的运行结果和你所想的一样，那么你就可以正式的运行flexget了。只需去掉 –test 部分即可：

    1	flexget execute

  • 此命令如果成功运行则意味着我们可以想办法让flexget定期运行，以实现自动追番的目的。我这里采用的是 Linux 自带的 crontab 的方法。

    • 首先，运行 crontab -e , 此命令会在默认修改器中打开crontab文件

    • 之后，需要在文件的最后加入一下内容：

      1	00 8,12,17,19,21 * * * /path/to/flexget/bin/flexget --cron -c /path/to/flexget/config.yml execute

      我目前的配置文件是在每天的8点，12点，17点，19点和21点的时候自动执行flexget任务。由于cron的特殊性，这里必须使用完整的flexget路径并使用 –cron 告诉flexget使用cron模式以及通过 -c 指定配置文件的位置。