Solving Repeatative Problem with Vim

This is the third article about NeoVim tips, if you are interested in other part of this, please check here and here

Motivation: The need for a clean and organized environment file.

Quite often in a project, a developer will have to deal with a lot of environment variables and/or environment files. However, they could also be the part that's least maintained in the entire codebase. As an example, sometimes, a feature flag could be added to the environment file, but when the feature goes live, people would fail to remove them. As time gets by, there will be A LOT of this kind of variables in the environment file.

As it is the end of the year, and I really couldn't finish much dev work with the amount of time left for the new year, I opt to deal with this issue with the intention to create a clean and organized environment file.

Overview: What's in an environment file

Before I get into the problem I'm trying to solve, it is important to understand the problem we are trying to solve. For the project I'm working on, we have two kinds of environment files, one of which is only used locally, the other one is used by the Kubernetes platform and is used at runtime. Here's an example of what they look like:

local_env_file.env

A_SERVICE               = http://abcdefg.tld
PORT                    = 53100
B_SERVICE               = http://bcdefg.tld
FEATURE_FLAG_AAA        = true
C_SERVICE               = http://cdefg.tld
FEATURE_FLAG_CCC_UNUSED = true
D_SERVICE               = http://defg.tld

helm_values.env.yaml

- name: A_SERVICE
  value: http://abcdefg.tld
- name: PORT
  value: 53100
- name: B_SERVICE
  value: http://bcdefg.tld
- name: FEATURE_FLAG_AAA
  value: true
- name: C_SERVICE
  value: http://cdefg.tld
- name: FEATURE_FLAG_CCC_UNUSED
  value: true
- name: D_SERVICE
  value: http://defg.tld

This is only a short snippet of the file, the whole file would be around 200 lines. Obviously, this kind of long file is not maintainable and it could also become source of unexpected bugs/failures if not dealt with carefully. To avoid this kind of issues, I have decided to make things cleaner and more organized, here are the expected end result after I've done the clean up:

local_env_file_CLEANED.env

PORT                    = 53100
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+                              SERVICES                               +
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
A_SERVICE               = http://abcdefg.tld
B_SERVICE               = http://bcdefg.tld
C_SERVICE               = http://cdefg.tld
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+                            FEATURE_FLAGS                            +
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
FEATURE_FLAG_AAA        = true

helm_values.env.yaml

- name: PORT
  value: 53100
- name: A_SERVICE
  value: http://abcdefg.tld
- name: B_SERVICE
  value: http://bcdefg.tld
- name: C_SERVICE
  value: http://cdefg.tld

- name: FEATURE_FLAG_AAA
  value: true

You might not agree with the format I did, but it is obviously much more organized than its previous state. And also note the unused feature flags and services are both removed.

Implementation

Once I've defined the end state, I went and started implementing it. To be honest, it look much difficult than it looked. I will break things by steps, so it's easier to explain.

Organizing services and feature flags

This might be the easiest task of all. You would first need to find all the lines that contains FEATURE_FLAGS or SERVICES. Then you would need to copy them all to a single register and paste them all out together.

What I did, for the .env file: clear the b register first with qbq, then do :g/FEATURE_FLAG/:norm "Bdd, then go to the end of the file with G and paste with "Bp

Similarly, for the helm_values file, you can slightly modify the command used above to: :g/FEATURE_FLAG/:norm "B2dd. This is because in the yaml file we have, the name and value are on separate lines.

Removing unused services and feature flags

This is the most tedious part. It would be much ideal if this could be done as part of the regular release or clean up process. However, I still opt to do it with some not-so-intuitive VIM scripting.

To understand how to solve this, we need to clearly understand an unused variable is a variable that's not appeared in the code base. To me, this means you won't be able to find any usage for the variable by doing a code search. Based on this criteria, we can come up with some Vim script:

• For a env file:

  :g!/^#/let x = system('rg -t "js" -t "ts" -l --count-matches ' . shellescape(expand("<cword>")) .. '|grep "" -c')|if x==0 |delete|endif

It looks complex, but it does what it meant to do. Let's break it down to see what it does step-by-step:

• :g this is the typical Vim :global command where it would apply a series of ex commands on lines that matches the following pattern.
• !/^#/ this is the regex we are looking for. In this case, it means any lines that do not start with a # or hash sign.
• let x = system('xxxxx') this is defining a vim variable called x, and assign the output of the command xxxx to variable x.
• rg -t "js" -t "ts" -l --count-matches ' . shellescape(expand("<cword>")) .. ' this is running rg command in a shell, you can actually replace this with grep but I find rg to be much more efficient.
  • For rg, I used the following flags:
    • -t argument would tell rg which type of file to look for
    • -l argument means 'Print only the paths with at least one match and suppress match contents'.
    • --count-matches means suppresses normal output and shows the number of individual matches of the given patterns for each file searched.
  • Notice the . immediately following the single quote, that is used for concatenating commands.
  • shellescape() function is used for escaping {string} for use as a shell command argument.
  • expand() function, is used to expand expand wildcards and the following special keywords in {string}. When combined with expand(<cword>), it would expand to word under the cursor.
  • |grep "" -c this character that follows removes the new line or empty character from the rg output, note this is still part of the shell command sequence.
  • |if x==0 |delete|endif, this is the final part of the command, it's straightforward: it's checking whether x is equal to 0. x is the variable we used to store the count of matches, so if the match is equal to 0, we delete the current line and end the if statement.

I have to admit, this is a relatively counter-intuitive command, but it's quite convenient to use. Depending on the size of your file and CPU speed of your computer, your VIM might freeze for a short while, as it is executing some heavy I/O command under the hood.

• For a yaml file:

  :g/name: \([A-Z].*\)/let x = system('rg -t "js" -t "ts" -l --count-matches ' . shellescape(matchstr(getline('.'), '\<\u.\+\>')) .. '|grep "" -c')|if x==0 |delete 2|endif

  Again, I will explain this one as well, most of the things are the same.
• For the pattern part, I switched to use name: \([A-Z].*\), this would match the line that contains name and also capture whatever comes after the name part.
• Inside the shellescape function, I switched to use matchstr() function which returns the matched string. In this case, it matching getline('.') against pattern '\<\u.\+\>' which is any upper case character.
• All the other part of the command remain the same, except delete 2, this is because for the yaml file, I would like to delete both the name and value which consists of two lines.

Conclusion

In this article, I explore the use of shell command in combination with Vim to solve some the repetitive task of finding and deleting unused variables in a file inside a project.

---

References:

• man rg(1)
• VIM:
  • :help :g
  • :help system()
  • :help shellescape()
  • :help expand()
  • :help matchstr()