The challenge is quite straightforward, delete all but the last characters in the file. I found several ways to tackle this challenge, so let me show you all:
Take 1: Use macro only
Record a macro, and play it back, so the keystrokes would be
1 2 3 4 5 6 7 8
qa $bd0j q 4@a <cr>
qa starts recording macro to register a $ move cursor to the end of the line d0 means *d*elete to the beginning of the line j move cursor down q finish macro 4@a repeat the macro in register a 4 times
Take 2: Use regex and %norm
It’s quite obvious that all we want to keep from the original files are the numbers. So the regex would be simple to come up with, like something as simple as /\d\+$<cr> will do. Once you type this into vim, all the numbers at the end of the line will be highlighted. Next you can do:
1 2 3 4 5
:%norm dn <cr>
% means applying to the whole file, norm means in execute following command in normal mode dn means *d*elete to *n*ext match
Take 3: No regex pure %norm
This is the fastest way I can come up with, still, not as fast as the top answers on Vimgolf but it is decent in my opinion. Being slightly different from the option above, it is still using %norm though:
1 2 3 4 5
%norm $bd0 <cr>
% means applying to the whole file, $ move cursor to the end of the line d0 means *d*elete to the beginning of the line
Takeaways:
norm is a quite powerful, and can be used to achieve complex stuff that can be otherwise achieved through a macro.
d the delete command is useful in many unexpected ways, besides dn and d0 command mentioned above which deletes to the next match and deletes to the beginning of the line respectively. An additional useful variation of d command is d4/x where 4/x meaning 4th occurrence of x.
This week, I was tasked with creating basic infrastructure for one of our newwebsites. We use Fastly for our CDN and New Relic as a log aggregation tool, most of our infrastructure is setup using Terraform, which is a popular infrastructure as code (IaC) platform. Terraform supports Fastly through a custom plugin, which also has the capability of New Relic. We need to customize the format of New Relic so that we can find the logs easily in New Relic. As expected, the plugin actually provide a pretty straight forward to accomplish this – what you need to do is to come up with a proper json string that align with the format that Fastly provided in their document. This seemingly straightforward task ended up took me some time to debug.
In terraform, you can create objects, and an object can be converted to string by using jsonencode function. This is where the pitfall comes in: Fastly’s formatting document said they have a option called “%>s” which represents the final status of the fastly request. From my perspective, it would definitely be helpful if we can include this when we send the log to Fastly. So I added this to my formatting object, and then jsonencode the object. To my suprise, I get an error saying that Fastly failed to parse the formatting option I created, which is quite strange. I then started to debug, I export TF_LOG=”DEBUG”, which asked Terraform to give me all the debug logs. To my suprise, I found that the “%>s” was jsonencode to “%\u003es” by terraform, and thus causing the error. How come “>” sign is encoded in Terraform? It turned out it’s for backward compatibility with ealier edition of Terraform. And according to their documentation
When encoding strings, this function escapes some characters using Unicode escape sequences: replacing <, >, &, U+2028, and U+2029 with \u003c, \u003e, \u0026, \u2028, and \u2029. This is to preserve compatibility with Terraform 0.11 behavior.
they encode much more characters for you without you realizing it. I really hope they can provide a way to opt-out of this feature or make this an opt-in, it will make things way easier for developers.
Recently, I have been eager to get the data saver on Android to work automatically for me. My goal is to have Tasker automatically enable the built-in data saver when my data is low. Before I came up with this idea, I have search the web to see if there is any solutions exist already so I don’t have to reinventing the wheel. Unfortunately, I couldn’t find anything on this topic so I have to create my own solution. Upon 1 week of testing, things to work as I expected so I decided to share in this article how I did it.
Prerequisite
An rooted Android phone – Unfortunately, the solution I came up with only works on rooted Android phone, I’m running Android 9.0.
A way to get to know you current data usage – Some carriers allow you to get you data usage through SMS, some might only allow you to use USSD to get data usage. I am only covering the case of using SMS here (my case).
How-to
The data saver in Android is basically something that limits all the background data usage. After some digging on the Internet, I found that you can turn the data saver on with the following command cmd netpolicy set restrict-background true (requires root). Then it is pretty easy to have the data saver turned on automatically when the data is low. My way of doing this is as follows:
Set a global variable as threshold for triggering the data saver.
Send an SMS to my carrier every morning at 8 AM, parsing the SMS to get my remaining data.
If my data remaining is lower than the threshold then turn on the data saver whenever mobile data is in use. This is done by using the run shell action and run the command mentioned above.
Turn data saver off automatically when I am connected to WiFi (Not necessary but add it just in case).
It is quite easy to do this when you carrier allows you to query data usage using SMS, with USSD however, things is not that easy and unfortunately, I haven’t figure out a way yet.
Recently, I decided to convert my QEMU based virtual machine installed on my Manjaro Linux to the VirtualBox format. The reason behind this is that I would like to be able to use the same VM across different host system (specifically Manjaro Linux and Windows 7). It is not a easy thing to do, so I decided to document it for future references.
Prerequisite?
An existing image created using QEMU (My VM file end with .img, for example)
VirtualBox
How to?
First thing first, you would need to convert the QEMU image (extension img) to raw format, this can be done by issuing the following command:
qemu-img convert Windows7-compressed.img -O raw Windows7.raw
This will generate a raw image. Note that this newly generated file might be a lot larger than the file it based on, this is because the img file allocates space on as-is basis.
After you get the raw image, it’s time to convert it to VDI format (which is used by VirtualBox). You can do this by running:
So after the previous step, you will have a working VirtualBox image, but if you boot it from VirtualBox, it might not work.
Gotchas!
In my case, what I was trying to convert was a Windows 7 VM, and when I finished the above steps and try to boot the VM, I got a BSOD. My feeling is that there were some default that QEMU used that doesn’t work for a newly created machine in VirtualBox. I tweak the following settings in the newly created VirtualBox VM:
Delete the auto created SATA controller and change it IDE controller.
It turned out, after doing that, everything works as expected.
This is probably an article that is long overdue to me personally. I have been an Android user since 2011, started with my Nexus S that I bought for use in college. It seems to me, the app Tasker had been very famous in the Andorid community, especially among users who know how to program. For those who have never heard of this app, this is a powerful app that allows you to automate almost anything that you can think of on the Android system.For me,It was not until August of last year when I bought the tasker app that I started to realize how powerful this app is. I am really not a big fan of its old school UI and design so I didn’t start using it until this year. After uisng it for a while, I did come to realize that there is no other apps close to it, if I were to move to iOS, this is probably among one the apps that I will miss. In this article I will explain how I used takser to automate things for myself, it is some boring stuff, but it did become something that I’m using everyday.
What can you use it for?
I will list one of the most used Tasks/Profiles that I created and used in this app:
Turn off/on ADB when using certain apps.
This is one of the easy ones that I found very useful. It is not uncommon nowadays for some apps to require you to turn off your ADB (Android Debugging Bridge) settings when you are using them, which to me, is quite annoying. So naturally, I created a Tasker profile together with a task to automate this. The trick here is you need to have root access, otherwise you are pretty much out of luck for this particular example. To create such a task, assuming you already have proper root access, go to the TASKS page in the app and click on the add icon, choose a name you like, and then you can create your first task! Think of a task as the things you want Tasker to do for you. In this particular case, my goal is simple: turn off ADB if it was not already off, turn it on if it was off. This way we can have one task that a turns off the ADB when you turns on the app; when you turns the app off, ADB will be switch back to on.
It is clear that we will need a global variable that holds the status of the adb status, to do this add an action: click on the button add icon while you are in the “Task Edit” page, and filter based on Shell, you will see a “Run Shell” as a result. Click on it, and in the “Command” input, enter “settings get global adb_enabled” and in “Store Output In” input, choose a global variable name you would like to use to hold the status of the current ADB status, just remember that you need to use all caps for the name for tasker to know it is an global variable. Also remember to check the “Use Root” checkbox. After this step, things will be simple, just add the if and else condition like I mentioned before, set adb_enabled to 0 if it is 1 and set it to 1 if it is 0, and after that don’t forget to set the global variable again.
TL;DR Here is the XML, that you can directly import to your Tasker if you don’t want to create it yourself:
<TaskerDatasr=""dvi="1"tv="5.2.bf"> <Tasksr="task3"> <cdate>1526421645228</cdate> <edate>1529892193595</edate> <id>3</id> <nme>adb_auto</nme> <pri>1006</pri> <Actionsr="act0"ve="7"> <code>123</code> <Strsr="arg0"ve="3">settings get global adb_enabled</Str> <Intsr="arg1"val="0"/> <Intsr="arg2"val="1"/> <Strsr="arg3"ve="3">%ADB_STATUS</Str> <Strsr="arg4"ve="3"/> <Strsr="arg5"ve="3"/> </Action> <Actionsr="act1"ve="7"> <code>37</code> <ConditionListsr="if"> <Conditionsr="c0"ve="3"> <lhs>%ADB_STATUS</lhs> <op>0</op> <rhs>1</rhs> </Condition> </ConditionList> </Action> <Actionsr="act2"ve="7"> <code>123</code> <Strsr="arg0"ve="3">settings put global adb_enabled 0</Str> <Intsr="arg1"val="0"/> <Intsr="arg2"val="1"/> <Strsr="arg3"ve="3">%ADB_STATUS</Str> <Strsr="arg4"ve="3"/> <Strsr="arg5"ve="3"/> </Action> <Actionsr="act3"ve="7"> <code>43</code> </Action> <Actionsr="act4"ve="7"> <code>123</code> <Strsr="arg0"ve="3">settings put global adb_enabled 1</Str> <Intsr="arg1"val="0"/> <Intsr="arg2"val="1"/> <Strsr="arg3"ve="3">%ADB_STATUS</Str> <Strsr="arg4"ve="3"/> <Strsr="arg5"ve="3"/> </Action> <Actionsr="act5"ve="7"> <code>38</code> </Action> <Actionsr="act6"ve="7"> <code>123</code> <Strsr="arg0"ve="3">settings get global adb_enabled</Str> <Intsr="arg1"val="0"/> <Intsr="arg2"val="1"/> <Strsr="arg3"ve="3">%ADB_STATUS</Str> <Strsr="arg4"ve="3"/> <Strsr="arg5"ve="3"/> </Action> </Task> </TaskerData>
To import, you need to save the above snippet to XML on your phone. In tasker app, click and hold the “TASKS” tab header, select import and choose the file
To run the task based on your predefined conditions, go to “PROFILES” tab, click on add button in the button, choose the application you want, and then, chose the task you imported and you are good to go! Now the app won’t say you can’t use it because you have ADB turned on, Tasker will turned adb off when you are using the app, and turn it on when you are not using it! :)
Before explaining what I did with Google App Scripts, let me explain what is Google App Scripts. It is basically a scripting engine that let you access all kinds of Google App features (including Gmail, Google Calendar, Google Docs, etc.). There are other ways to access these features but Google App Script might be the easist way out there. It comes with an online editor (you can work offline as well), and Google can trigger your scripts at certain time based on your need. Oh, forgot to mention, Google App Script uses JavaScript which is one of the most popular languages right now (Sadly, it doesn’t support ES6 syntax as of writing).
Why use Google App Scripts?
I have a habit of archiving my Emails every week on Mondays. It is easy enough to do in the gmail Web interface but I do need to remember to do it. That make me think if I can find a good way to automate this. I would always think of using some kind of API first, but it seems to me that it wouldn’t worth it to spend time to make a app just for the purpose of archiving emails. Then I came across Google App Scripts, which turned out to be exactly what I need to automate this.
What I came up with
Here is what I came up with to archive emails that are one-week-old every Monday at midnight:
1 2 3 4 5 6 7 8 9 10 11
functiongmailAutoarchive() { var maxDate = newDate(); // Archive all threads in inbox whose last message date is older than today. var threads = GmailApp.getInboxThreads(); for(var i in threads){ var thread = threads[i]; if (thread.getLastMessageDate()<maxDate){ thread.moveToArchive(); } } }
You will be able to use this script by putting it into the google script editor located here.
To have Google auto run your scripts, you need to add the trigger which can be found under Edit –> Current project’s triggers.
Variation
After creating this script, I created another script as a variation to this script that move messages with certain labels to trash.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
functiongmailAutoRemove() { var maxDate = newDate(); maxDate.setDate(maxDate.getDate() - 30); // what was the date at that time? var label = [GmailApp.getUserLabelByName("label1"), GmailApp.getUserLabelByName("label2"), GmailApp.getUserLabelByName("label3")]; label.forEach(removeByLabel(maxDate)); }
functionremoveByLabel(maxDate){ returnfunction(label){ var threads = label.getThreads(0, 100); for(var i in threads){ var thread = threads[i]; if (thread.getLastMessageDate() < maxDate){ thread.moveToTrash(); } } } }
This week, I was having making some great progress in understanding how GPG works either locally or through email. The original intention to do all this was because I would like my router to send me notification whenever the tranmission finish download the torrent. This may be simple as it sounds and it had been working correctly for me for several months since I create the initial script.
This week, however, I decided to do something special. I would like the router to sign/encrypt the message it sends to me. I am not sure why I need that, but anyways, I did get to learn a lot through the process.
Here is my original script, it simply uses the mailx program that installed on the router and sends the email through SMTP. Looks quite simple, it pretty much is the same as the script I have shown in the previous post:
1 2 3 4 5 6 7 8
#!/bin/sh
SMTP_SERVER=YOUR_EMAIL_SMTP MESSAGE="Hello!\n\n \tThis is a notification from transmission, $TR_TORRENT_NAME has been completed on $TR_TIME_LOCALTIME\n\n Thanks!" SENDER="YOUR_EMAIL_USER_NAME" RECEPIENT="EMAIL_TO_RECEIVE"
A little more explanation here, there are two variables preset by Transmission. $TR_TORRENT_NAME, is the name of the torrent that has just been finished. $TR_TIME_LOCALTIME is the time when the download was finished. There were several other environment variables set by transmission also. Here is a list of them^1 Note: The meaning of this variable are not explicitly documented in the wiki, and I guess the meaning based on my understanding.
Env Variable Name
Meaning
TR_APP_VERSION
The version of the transmission app.
TR_TIME_LOCALTIME
The time when the current torrent has been downloaded.
TR_TORRENT_DIR
The directory that the content of the torrent was downloaded to.
TR_TORRENT_HASH
The hash value of the torrent.
TR_TORRENT_ID
The ID of this torrent (in the download list for transmission bookeeping?).
TR_TORRENT_NAME
The name of the torrent.
So my initial thought was that adding the GPG encryption or signing is as easy as adding a new pipeline that redirects the output to GPG. However, it turned out to be much more difficult than that. When the script is called by transmission, it doesn’t set the environment variable required by GPG and because of this GPG would failed to find the private key used to sign/encrypt the message and therefore failed to encrypt. After setting the environmental variable in the scripts the GPG encryption works correctly. Here is the working script with encryption and signing.
1 2 3 4 5 6 7 8 9 10 11 12
#!/bin/sh
SMTP_SERVER=YOUR_EMAIL_SMTP MESSAGE="Hello!\n\n \tThis is a notification from transmission, $TR_TORRENT_NAME has been completed on $TR_TIME_LOCALTIME\n\n Thanks!" SENDER="YOUR_EMAIL_USER_NAME" RECEPIENT="EMAIL_TO_RECEIVE" HOME="YOUR HOME DIRECTORY" GPGHOME="YOUR .gpg DIRECTORY" export HOME=$HOME export GPGHOME=$GPGHOME
From this week onward, I will be keeping notes for some of the issues I encountered during work or developing some of my personal works.
Here are notes from my development last week:
This might be a quite simple one, but I found it might be useful for future reference. In the development of TIA, previous developer used to disable the checkbox in a form. The result of this is the value of the disabled checkboxes were not submitted with the form. The previous developer opt to set the checkboxes separately which I don’t believe is an ideal solution. I thereby found another solution as follows:
However, this is not enough in my case, I also need to have it gray out to make it looks like disabled. It turned out that this can be solved via simple CSS:
Just make sure to give the checkbox proper class name.
This week, I was also asked to generate data from InterCAD. What I used to do was to copy the result of the query from database to a MS Excel worksheet and done the work there. This is especially easy when when there are not a lot of data. This time around, there were 16 days worth of data that I need to do statistics on. So, I decided to create a Bash script to do this task. The bash script is listed as follows, it is a pretty easy one and can be improved in the future:
1 2 3 4 5 6 7 8 9 10 11 12 13
COUNTER=1 TYPE=ACCIDENT while [ $COUNTER -lt 17 ]; do if [ $COUNTER -lt 10 ] then d='2017-05-0' else d='2017-05-' fi echo$d$COUNTER grep $d$COUNTER a.csv|grep $TYPE|wc -l let COUNTER=COUNTER+1 done
Basically, this time around, I exported the result of my query into a csv called a.csv and use grep to get what I want. This script can be improved in several ways:
Use awk instead of simple grep
Have the user input the type and the date from the command line.
This will be for future improvements.
For my own project, I have set up a log watcher for the Flexget which I have installed on my Raspberry Pi 3. This log watcher runs 5 minutes after Flexget is ran and attach the error part of the log as an attachment and sent to my email address.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
SMTP_SERVER=YOUR_EMAIL_SMTP RECEPIENT=EMAIL_TO_RECEIVE SENDER="YOUR_EMAIL_USER_NAME" TEMPLOG="TEMP_FILE_TO_STORE_ERROR_LOG"#Will be deleted email is sent DATE=$(date -I) MESSAGE=$(cat $HOME/flexget/flexget.log | grep -v INFO | grep -v VERBOSE | grep $DATE) CONTENT="Attached is the error log from today, please check!" if [ ! -z "$MESSAGE" ] then printf"$MESSAGE"|dd of=$HOME/errorlog printf"$CONTENT"|mailx -vr $SENDER -a $TEMPLOG -s "[TRANSMISSION ERROR] Transmission has encountered error" -S smtp=$SMTP_SERVER -S smtp-use-starttls -S smtp-auth=login -S smtp-auth-user=$SENDER -S smtp-auth-password="YOUR_EMAIL_PASSWORD" -S ssl-verify=ignore $RECEPIENT rm $TEMPLOG fi
This script finds the error from flexget.log and write to TEMPLOG. If there is indeed errors exists in the log. The file will be attached and sent through the SMTP server to the address of your choice.
Over the weekend, I also started to learn React. React is a web library developed and used by Facebook. It is similar to Angular JS, but uses more straight rather than the TypeScript used by Angular JS. There were several notes that I would like to write down for future reference:
React are composed of components, however, there were several ways to declare a component in React.
The first way is the way that I would prefer, which is to use the ES6 class declaration, shown as follows:
This will be deprecated in the future release and put here just for reference only.
To use the above mentioned newly created component in a page. You can use the following code:
1 2 3 4 5 6 7 8 9 10 11
import React from'react' import { render } from'react-dom' //import the component, assuming it is in a folder called components and the source is called SkiDayCount.js import { SkiDayCount } from'./components/SkiDayCount'