Ronnie’s Epoch

Dockerized ronniechung.com with minor problems. Nothing really to do with Docker itself though. The site is a WordPress blog and after I migrated it to Docker it was working fine. That is until I forced HTTPS with Let’s Encrypt and Nginx. The site would load but the CSS and JS wouldn’t, still being served over HTTP. So I thought this was a problem with the Nginx configuration, kept looking at it but didn’t see anything that would make this happen. Tried to login to the WordPress admin and got a “too many redirects” error. Searched for Nginx SSL too many redirects but didn’t seem to be the problem. Then I remembered I had to do something with WordPress when I first got it forced over SSL. Turns out the problem is with WordPress itself and the fix is to add the following to your wp-config.php file under the WP_DEBUG line:

if ( $_SERVER[‘HTTP_X_FORWARDED_PROTO’] == ‘https’ )
{
   $_SERVER[‘HTTPS’]       = ‘on’;
   $_SERVER[‘SERVER_PORT’] = 443;
}

Apparently this is the same fix for CloudFlare too.

Docker is all the rage right now. Well containers in general. I’ve looked into Docker before for work before and it just didn’t fit with what the company needed at the time. I always wanted to take a look at Docker in particular for my own personal projects and recently I got the chance to do just that. Late one night I decided it would be a good time to do some cleanup of own of my project servers, namely the one that hosts the Reddit Media Scraper (RMS). As the name implies, it scrapes Reddit… for media… yeah… Anyway, this is just a personal project to teach myself how to access restful APIs (it uses not only Reddit’s API but also Imgur’s, Gfycat, and some others too), web scraping and functional Python programming. Well, as I was cleaning out files in my less than alert state, I accidentally destroyed the entire RMS directory. Yarp.

Since I had to restore all the app files, I decided to just wrap it all up into a Docker container and learn something in the process. I learned a little about Docker the first time after looking into the matter. Just had to learn how to build a container using a Dockerfile, which didn’t seem very hard to do. Looked up the Dockerfile reference documentation, started up Sublime Text, and in 5 mins I had it done. It was pretty basic. Used the official ubuntu image, installed python3 and python3-pip, copied the code, installed requirements, and set the command to run on container start. Tested it out and it worked! Well… it sort of worked.

I had the title of every link that was proceeded printed out to stdout and some titles had non-ASCII characters which would cause an UnicodeEncodeError to occur when ran inside the container. Ok, not a problem I’ll just convert it. So that’s what I tried first… only to realize that the problem is with the container not supporting UTF-8 and not a problem with the RMS or Python 3. Some quick googling revealed that by default Docker containers are set for ASCII. Suggestions for changing the container from ASCII to UTF-8 by setting the ENV LANG=UTF-8:

docker container run —name rms -e LANG=UTF-8 rms

Which didn’t work for me… after some more googling and testing I eventually found that I had to set LANG=C.UTF-8 for it to work. Updated the Dockerfile to set that ENV and gave it a test. It worked! Took me about 2 hours total from getting things working manually (i.e. running /bin/bash in the Ubuntu container), writing the Dockerfile, and finally getting everything working and deployed successfully. Pretty cool, I can see why developers like Docker so much.

Well I lied when I said there would be more pictures of pretty ladies posted soon after the server was redone. Been keeping busy with work, lots of changes being made to our company infrastructure. I’ve also been keeping busy with some personal projects. One thing I set out to do this year was to really learn Javascript. Ew, I know. But it’s a language that is in high demand and isn’t going away any time soon.

Some background first, the languages that I learned in chronological order is: Visual Basic 6, Visual Basic .NET, ASP, ASP .NET, C++, Java, C, x86 ASM, Scheme (I HATE SCHEME!), Haskell (I LOVE Haskell), and finally Python. For the last 2 years or so, I’ve been coding almost exclusively in Python. It’s a great elegant language especially compared to the languages I learned before. It should also be mentioned that I barely recall how to write any code in the above listed languages except Java and some basic C/C++, it’s been too long!

Now going from a beautiful language like Python, which is white space delimited and lacks semicolons and brackets, to trying to learn something that most programmers would describe as fugly at best aka Javascript is… difficult. The goal was to not just learn Javascript for use in websites but ultimately to learn Node.JS and Angular.JS. Without going into the details, the transition was an exercise in patience. I did “re-learn” a few things such as the use of closures which was completely forgotten until Javascript reminded me that it existed. The last two weeks was spent building something in Javascript/Node/Angular. The result is an image preview/viewer for photo selection. Eh? Let me explain.

Over the years, I’ve been building programs that help me with photography. Like every photographer, I have a preferred way of doing things. There was a need to find an easy way to delivery photos to people and like most people at the time I used FTP (this was before Dropbox was a thing). Unfortunately, some people didn’t know how to use FTP and would get confused. Fortunately though, Dropbox was available and had just released the share public link feature. But I didn’t like having to pay Dropbox for storage (2GB wasn’t enough for me). At the same time, Amazon Web Services (AWS) had just announced their new Simple Storage Service (S3).

I looked for all the information I could find on how S3 worked. After some research, I set out to write myself a S3 clone or something similar so I wouldn’t need to use Dropbox anymore. The progress was slow, especially never having done any kind of web app before. Since Java was what I knew at the time, the Bucket File Sharing System (BFS System) was written in Java Server Pages (JSP). While researching how to implement the basic features of the BFS, I found that JSP is considered obsolete and instead should be using Servlets. *Sigh* Started over learning Servlets. I also learned that US laws regarding encryption is stupid and one should not attempt to implement your own AES and BCrypt algorithms. Long story short, version 1 of the BFS System was written in Java and custom written Javascript/CSS. It was ugly… really ugly but it worked and did the job.

Fast forward a year or so and Python was becoming all the rage. So of course I had to learn Python too. This was a very easy and pleasant transition. Python was awesome. The next logical step was to redo the BFS System using Django! So that’s what I did. And in the process replaced all the ugly custom written CSS with Bootstrap. The end result was still ugly… but better.

Fast forward another year or so and I finally got the chance to see how S3 worked first hand. The company was going to build their new mobile infrastructure in AWS and I was the one who was going to do it. To my surprise, the BFS System came pretty damn close to how S3 worked and by this time (I kept adding features) almost had feature parity! If you can’t tell from this post, the BFS System is one of my proudest pet projects 🙂

At the same time, I couldn’t find an image viewer that would do what I wanted, how I wanted. So… it was a good time to learn PyQT. I set out to write my own image viewer that was a cross between Adobe’s Bridge and Photo Mechanic, just much uglier and slower :p After about 6 months, I had a working image viewer that fit my workflow. It would spit out JSON which could then be processed by another Python script I had written that would pull the images that I had selected for processing out from the photo storage drive into the working directory of the temp drive. It would do basic duplicate detection and renaming too. The image viewer is now the most used program (i.e. used by me) I have ever written.

Now something that’s been annoying me for a long time. During events where I could end up taking up to 1000+ pictures, finding a way to allow multiple people to select photos that they want me to process was difficult. The usual way I dealt with this was by outputting low resolution jpegs, send as a zip to the various people via the BFS, and have them email me the filenames of the photos they wanted. Not very efficient. How to solve this? Node and Angular to the rescue! Perfect time to learn Javascript eh? So that’s what I did, I built a photo selection app using Node/Angular that people could just go to via a link I send them, select which photos they want and the app would save a JSON output of their selection that could then be passed to my Python script mentioned above. Cool right?! Damn straight…

TL;DR: I hate Scheme, Javascript sucks, I make ugly shit, learning is fun.

Till next time…

Lately there’s been a lot of fuss about serverless. But what the hell does serverless even mean? Let’s see if I can provide a brief explanation.

The last two years at work, I’ve been gradually moving our on-premise infrastructure up into “the cloud” (AWS and DO mainly). At this point, about 85% of all the company’s infrastructure is now in “the cloud”. Which brings us to AWS Lambda and serverless. How the hell could something be serverless? Well what they really mean is that you don’t have to worry about the server. You write some code and the code will be triggered to run and you don’t have to provision a server, install the dependencies, etc. Sounds cool, right? Yes. And more importantly to the company, it can help save a lot of money.

There’s a few things that I could transition to AWS Lambda. Until recently though, Lambda didn’t support Python 3.x and that meant it wasn’t something that I wanted to deal with. But that has changed now and Lambda supports Python 3.6. So in the next month or so, I’ll be playing around with it and hopefully can migrate some stuff over and save the company some more money. Should be a fun side project.

Till next time…

I’m often working on little personal projects alongside the projects for work. Personal projects though, gives me a chance to try new things, experiment, learn new techs, etc. I’m also much less careful so end up doing really stupid things (in hindsight). So I thought it’ll be nice to share some of those… adventures. Keep in mind, I’m no pro… so if you know of better ways of doing something, let me know in the comments!

Working on a trouble ticket system the other day, calling it Tick-IT ha! Get it????? From experience, whenever I write projects that uses a database I always end up adding new features that require modifications to the table schema. The changes usually involves one of two options: 1) adding a new table that shares a key with the existing tables in the best case 2) requiring the adding of a new column which means migrating all existing data in the table into a temporary table, deleting the original, then creating the new table, moving the data from the temp table to the newly created one. A total pain in the fucking ass.

In an attempt to be smart, I decided that I would only have 3 columns in the tickets table which holds all ticket information: ID (unique ticket ID), ticket number, and ticket data (JSON blob that stores all other information on the ticket). Brilliant right? Need to add new fields? Just add to the JSON object! Nope… See, the problem that I didn’t think about was the JSON data isn’t searchable using SQL. It would require a full text search. So… Not so smart after all. 

Till next time…

Disclaimer: I wrote all this on my phone while “shopping”. Please excuse any misspellings or grammar problems.

It’s a new year and a new start for the server that runs ronniechung.com. The process took far FAR longer than I originally expected. This was because I was learning while doing. So what changed?

First off, ronniechung.com no longer runs on bare bones. What I mean is that the server that hosts the site is now a VM. The physical server is the VM host, it runs Debian. The hypervisor as you can probably guess is KVM. The VM that runs the site is running Ubuntu still because I didn’t want any distro changes to affect my scripts, custom web apps, etc. The process of getting all this setup was actually easier than expected. Only took a few days to get it all configured and ready for deployment. 

What really took a long time was getting the VM setup. I wanted to learn configuration management using either Puppet or Ansible. Puppet was the obvious choice being that it was so popular, but as I quickly learned it wasn’t the right solution for me. The time spent learning Puppet though (about a week) was worth it for sure. The biggest problem with learning something new while attempting to apply it to a “production” system at the same time is you’re almost guaranteed to make mistakes or do things that aren’t exactly “best practices” first time through. And that’s exactly what happened. 

A few days ago, everything was set and all tests were passing. I switched the DNS back thinking it was time to bring things back to normal. I was wrong… First I wanted Ansible to handle everything including restoring files (like the pics of the beautiful ladies ;)). While writing the playbook for that, I decided to update everything to follow “best practices” which I learned the same day. Finally late into New Year’s Eve, I had everything updated and finished. Ran one more test to make sure I didn’t accidentally screw something up (I did) before going to sleep. Checked in the morning and well, I made a couple mistakes in regards to restoring of files. Simple fixes but still annoying. 

As you can see today though, everything is fully operational and all managed by Ansible. Best part is if I ever needed to rebuild its just a few steps and less than 30 mins to do. There’s still a couple things to do, I don’t have VM backups setup yet for instance, but all the major work is finished. Quite happy with the setup right now but if you know me you’ll know I like to keep changing things. I get bored real easy 🙂

Till next time… 

EDIT: Fixed a spelling error.

I’m a tinkerer. Never one to leave well enough alone. So sometime this month, I’ll be rebuilding the server again for at least a couple days. I’m SURE you all have questions!

Why rebuild again?! Because I want to learn new shit.

What are you learning?! Uh… quite a few things 🙂

What’s changing?! Nothing that you’ll notice dear reader… maybe…

Are you going to fuck up and end up losing all these pics of beautiful models?! No… hopefully not. Assuming I remember to back everything up first :p

Why don’t you have backups?! I’m lazy…

Will the new rebuild include a backup system?! Hell YES!

When will this happen?! When I’m not lazy. Currently aiming for next week, but we’ll see.

When will you post new pics of pretty ladies?! Soon after the rebuild… If I’m not lazy 🙂

Don’t know if anyone even looks at this blog now but in case there is, you’ll know. 

The beautiful Roxana Popescu allowed me to test out my new toys and lighting on her. The following are the best pictures from the shoot.

I’ve created an Instagram for all my photography related stuff. Follow me @ronniecphotography

Till next time…

 

 

Did a photo shoot with the lovely Roxxana on Monday. It was a pretty simple shoot, some head shots, some full body. Had a good time and got some good images which I will be posting in the coming week. For now though, here are some behind the scenes pictures.

 

Stay tuned for the final images. Till next time…