Month: January 2018

Dockerized ronniechung.com with minor problems. Nothing really to do with Docker itself though. The site is a WordPress blog and after I migrated it to Docker it was working fine. That is until I forced HTTPS with Let’s Encrypt and Nginx. The site would load but the CSS and JS wouldn’t, still being served over HTTP. So I thought this was a problem with the Nginx configuration, kept looking at it but didn’t see anything that would make this happen. Tried to login to the WordPress admin and got a “too many redirects” error. Searched for Nginx SSL too many redirects but didn’t seem to be the problem. Then I remembered I had to do something with WordPress when I first got it forced over SSL. Turns out the problem is with WordPress itself and the fix is to add the following to your wp-config.php file under the WP_DEBUG line:

if ( $_SERVER[‘HTTP_X_FORWARDED_PROTO’] == ‘https’ )
{
   $_SERVER[‘HTTPS’]       = ‘on’;
   $_SERVER[‘SERVER_PORT’] = 443;
}

Apparently this is the same fix for CloudFlare too.

Docker is all the rage right now. Well containers in general. I’ve looked into Docker before for work before and it just didn’t fit with what the company needed at the time. I always wanted to take a look at Docker in particular for my own personal projects and recently I got the chance to do just that. Late one night I decided it would be a good time to do some cleanup of own of my project servers, namely the one that hosts the Reddit Media Scraper (RMS). As the name implies, it scrapes Reddit… for media… yeah… Anyway, this is just a personal project to teach myself how to access restful APIs (it uses not only Reddit’s API but also Imgur’s, Gfycat, and some others too), web scraping and functional Python programming. Well, as I was cleaning out files in my less than alert state, I accidentally destroyed the entire RMS directory. Yarp.

Since I had to restore all the app files, I decided to just wrap it all up into a Docker container and learn something in the process. I learned a little about Docker the first time after looking into the matter. Just had to learn how to build a container using a Dockerfile, which didn’t seem very hard to do. Looked up the Dockerfile reference documentation, started up Sublime Text, and in 5 mins I had it done. It was pretty basic. Used the official ubuntu image, installed python3 and python3-pip, copied the code, installed requirements, and set the command to run on container start. Tested it out and it worked! Well… it sort of worked.

I had the title of every link that was proceeded printed out to stdout and some titles had non-ASCII characters which would cause an UnicodeEncodeError to occur when ran inside the container. Ok, not a problem I’ll just convert it. So that’s what I tried first… only to realize that the problem is with the container not supporting UTF-8 and not a problem with the RMS or Python 3. Some quick googling revealed that by default Docker containers are set for ASCII. Suggestions for changing the container from ASCII to UTF-8 by setting the ENV LANG=UTF-8:

docker container run —name rms -e LANG=UTF-8 rms

Which didn’t work for me… after some more googling and testing I eventually found that I had to set LANG=C.UTF-8 for it to work. Updated the Dockerfile to set that ENV and gave it a test. It worked! Took me about 2 hours total from getting things working manually (i.e. running /bin/bash in the Ubuntu container), writing the Dockerfile, and finally getting everything working and deployed successfully. Pretty cool, I can see why developers like Docker so much.

Well I lied when I said there would be more pictures of pretty ladies posted soon after the server was redone. Been keeping busy with work, lots of changes being made to our company infrastructure. I’ve also been keeping busy with some personal projects. One thing I set out to do this year was to really learn Javascript. Ew, I know. But it’s a language that is in high demand and isn’t going away any time soon.

Some background first, the languages that I learned in chronological order is: Visual Basic 6, Visual Basic .NET, ASP, ASP .NET, C++, Java, C, x86 ASM, Scheme (I HATE SCHEME!), Haskell (I LOVE Haskell), and finally Python. For the last 2 years or so, I’ve been coding almost exclusively in Python. It’s a great elegant language especially compared to the languages I learned before. It should also be mentioned that I barely recall how to write any code in the above listed languages except Java and some basic C/C++, it’s been too long!

Now going from a beautiful language like Python, which is white space delimited and lacks semicolons and brackets, to trying to learn something that most programmers would describe as fugly at best aka Javascript is… difficult. The goal was to not just learn Javascript for use in websites but ultimately to learn Node.JS and Angular.JS. Without going into the details, the transition was an exercise in patience. I did “re-learn” a few things such as the use of closures which was completely forgotten until Javascript reminded me that it existed. The last two weeks was spent building something in Javascript/Node/Angular. The result is an image preview/viewer for photo selection. Eh? Let me explain.

Over the years, I’ve been building programs that help me with photography. Like every photographer, I have a preferred way of doing things. There was a need to find an easy way to delivery photos to people and like most people at the time I used FTP (this was before Dropbox was a thing). Unfortunately, some people didn’t know how to use FTP and would get confused. Fortunately though, Dropbox was available and had just released the share public link feature. But I didn’t like having to pay Dropbox for storage (2GB wasn’t enough for me). At the same time, Amazon Web Services (AWS) had just announced their new Simple Storage Service (S3).

I looked for all the information I could find on how S3 worked. After some research, I set out to write myself a S3 clone or something similar so I wouldn’t need to use Dropbox anymore. The progress was slow, especially never having done any kind of web app before. Since Java was what I knew at the time, the Bucket File Sharing System (BFS System) was written in Java Server Pages (JSP). While researching how to implement the basic features of the BFS, I found that JSP is considered obsolete and instead should be using Servlets. *Sigh* Started over learning Servlets. I also learned that US laws regarding encryption is stupid and one should not attempt to implement your own AES and BCrypt algorithms. Long story short, version 1 of the BFS System was written in Java and custom written Javascript/CSS. It was ugly… really ugly but it worked and did the job.

Fast forward a year or so and Python was becoming all the rage. So of course I had to learn Python too. This was a very easy and pleasant transition. Python was awesome. The next logical step was to redo the BFS System using Django! So that’s what I did. And in the process replaced all the ugly custom written CSS with Bootstrap. The end result was still ugly… but better.

Fast forward another year or so and I finally got the chance to see how S3 worked first hand. The company was going to build their new mobile infrastructure in AWS and I was the one who was going to do it. To my surprise, the BFS System came pretty damn close to how S3 worked and by this time (I kept adding features) almost had feature parity! If you can’t tell from this post, the BFS System is one of my proudest pet projects 🙂

At the same time, I couldn’t find an image viewer that would do what I wanted, how I wanted. So… it was a good time to learn PyQT. I set out to write my own image viewer that was a cross between Adobe’s Bridge and Photo Mechanic, just much uglier and slower :p After about 6 months, I had a working image viewer that fit my workflow. It would spit out JSON which could then be processed by another Python script I had written that would pull the images that I had selected for processing out from the photo storage drive into the working directory of the temp drive. It would do basic duplicate detection and renaming too. The image viewer is now the most used program (i.e. used by me) I have ever written.

Now something that’s been annoying me for a long time. During events where I could end up taking up to 1000+ pictures, finding a way to allow multiple people to select photos that they want me to process was difficult. The usual way I dealt with this was by outputting low resolution jpegs, send as a zip to the various people via the BFS, and have them email me the filenames of the photos they wanted. Not very efficient. How to solve this? Node and Angular to the rescue! Perfect time to learn Javascript eh? So that’s what I did, I built a photo selection app using Node/Angular that people could just go to via a link I send them, select which photos they want and the app would save a JSON output of their selection that could then be passed to my Python script mentioned above. Cool right?! Damn straight…

TL;DR: I hate Scheme, Javascript sucks, I make ugly shit, learning is fun.

Till next time…