The main idea behind Software Architecture Methodologies such as Clean Architecture and Hexagonal Architecture is to create loosely coupled components that can be organized into layers. This way of writing code leverages the separation of concerns design principle and makes our application easier to maintain, i.e. we can easily …
I had two weeks of vacation planned around PyCon, but canceled it since it didn't make sense to take time off when I couldn't go anywhere because of COVID-19.
I was feeling pretty down when the conference start date passed. I had spent the past few months looking forward to …
Need to access a private Git repo inside of a Dockerfile during the
docker build process.
The main idea is to use HTTP Basic Authentication while downloading files using the
git command line client.
Create a Personal Access Token with repo scope using your Git provider's interface …
Yesterday I gave an abridged version of my Data Science Workflows using Docker Containers talk at my employer's Biweekly Engineering Lunch session.
It had been a minute since I last gave this talk so I ran through the examples to make sure everything worked. Having all my examples containerized is …
As a learning exercise, I migrated all my side projects to Kubernetes. I was using Fluent Bit to forward application container logs to my ElasticSearch instance.
Everything was working as expected, but the cluster was starting to get expensive. DigitalOcean Kubernetes is competitively priced, but paying $80/month to run …
The paradox of choice in Python packaging tools makes dependency management more complex than it needs to be. In this Quick Hit, we will explore Python dependency management and demonstrate an easy-to-implement workflow to generate reproducible environments.
Note: this article assumes familiarity with virtual environments.
We use dependency management …
Podcasts allowed me to immerse myself in the tech industry during the earlly stages of my career change.
Talk Python, PythonBytes,
Test & Code,
Software Engineering Radio,
and the Atwood-Spolsky era Stack Overflow Podcast
were instrumental in leveling up my skills
and helping me get my first job.
I started my first tech job at a large corporation. We have lots of internal tools and systems that allow us to solve large-scale data problems. It's been an interesting change from startup life where platforms weren't as mature, where systems weren't as integrated into workflows and processes.
While I'm …
Programming is writing code to solve problems. Software Engineering is the practice of using a structured process to solve problems. As engineers, we want to have a codebase we can change, extend, and refactor as required. Tests ensure our program works as intended and that changes to the codebase do …
I recently purchased a new computer and needed to transfer files from the old computer to the new one. Tried a USB drive, no luck. Tried copying files over the network via the macOS Finder, but the sheer number of files (~500,000) made this an exercise in futility.
Apispec's design made it easy to extend core functionality for a specific use case. I extended the
apispec.BasePlugin class, overrode a couple of methods, and …
In this Quick Hit, I will show the Pythonic way of sorting objects by multiple attributes.
Note: This example is leveraging Python 3.7 dataclasses for convenience. Objects can be of any type.
Earlier this week, I went through the fantastic PyCon 2018 tutorial, Building a Search Engine with Python + Elasticsearch. This required me to have a local installation of Elasticsearch and Kibana.
Local installation? Time to spin up Docker, more specifically, Docker Compose.
Docker-Compose is a tool that allows us to define …
The journey of a thousand miles begins with a single step.
Everything in life that's worth having requires effort. In tech, it also requires convincing somebody you are good enough to take a chance on.
A few months ago, I pivoted from data into web development because I wanted to …
(Note: This essay was written for my Foundations of Leadership course.)
Software Engineering is the discipline of Computer Science concerned with “developing and maintaining software systems that behave reliably and efficiently, are affordable to develop and maintain, and satisfy all the requirements that customers have defined for them” . Creating …
Being a Software Engineer isn't just about being effective in a specific programming language, it's about being able to solve any given problem using the tools at hand.
This week at work I have to extend the functionality of a WordPress plug-in so it can fit into our microservices-based backend …
In this post we will explore
2017 will forever be etched in our memories as the year Python overtook R to become the leading language for Data Science. There are many factors that play into this: Python's simple syntax, the fantastic PyData ecosystem, and of course buy-in from Python's BDFL.
PEP 465 introduced the
@ infix operator that is designated to be used for matrix multiplication. The acceptance and implementation of this proposal in Python 3.5 was a signal to the scientific community that Python is taking its role as a numerical computation language very seriously.
Testing is one of my favourite things about programming. There is nothing like the piece of mind that comes from being able to modify code without having to worry about breaking something.
Dynamic typing refers to programs checking variable types at runtime versus compile time. By not having to define the variable type as we write code, we can work faster and take advantage of duck typing.
On the other hand, we do miss out on some benefits that types bring. Similiar to how the principles of test-driven development force us to write smaller, more testable functions, thinking about types improves software design by making us cognizant of the values our variables hold as well as the values our functions can return.
Here’s to the new year. May she be a damn sight better than the old one, and may we all be home before she’s over.
End of year is the perfect time to reflect on the past and to plan for the future.
- Became more public …
In Python, functions are objects. This means we can return them from other functions.
In this Quick Hit, we will use this property of functions to mock out an external API with fake data that can be used to test our internal application logic.
Note: I previously used Python functions to simulate the behavior of a case statement
In this post, I will describe how to use BSON-NumPy to pull data out of Mongo and into pandas. While this library is still in the prototype stage, it's hard to to ignore the 10x speed improvement that comes from reading BSON documents directly into NumPy.
For this example, I will be analyzing a collection of my tweets
I was floored when I discovered I could use tuples (and namedtuples) as Python dictionary keys.
Turns out this is not totally correct.
Started reading Fluent Python and learned that tuples are not always immutable. For instance, they can contain list elements...
The startup I work for is making a transition into a microservices-based architecture. I'm working with a Senior Engineer to design a new backend platform based off our current monolith.
We plan on having a bunch of REST API endpoints in containers that DevOps will orchestrate using Kubernetes. The current …
Truly wonderful the mind of a child is
It was an action packed week at work: we migrated our infrastructure to the Google Cloud Platform and into Kubernetes. Pods, nodes, deployments. Oh my!
Not that I helped much with the migration; I'm new and still learning the ropes.
I know …
The best laid schemes o' mice an' men
Gang aft agley
I started a new job as a Mathematician / Software Engineer at a Chicago-based healthcare startup in October. It's my first proper development job and I could not be more excited.
I get paid to write code. It's pretty awesome …
Even though my development folders are well organized, I still have to feel my way around the filesystem when I'm looking for a project directory. It's like trying to find a lightswitch in a dark room: a
cd here, an
ls there, maybe a
find when I'm stuck.
I always …
ChiPy hosts Python Project Night on the 3rd Thursday of every month. Groups of four are given an hour to work through a structured exercise with each person spending 15 minutes at the keyboard writing code. At the end of the hour, one group is selected to present their solution.
2017 is the year I became deliberate about my approach to programming. Like most newbie developers, I hacked together spaghetti code around chunks of answers I found on StackOverflow (Praise Be).
I didn't care how I solved the problem, only that it was solved enough to cross off my todo …
A stack is a linear data structure that holds a collection of items. We can define operations performed in a stack from the point of view of a user:
.push()- add item to top
.pop()- remove and return item from top
.peek()- return top most element
Trueif stack …
I love using the Command Line; it's my favourite part of programming. It wasn't always this way. I used to worry that the commands I entered would destory the delicate equilibrium where everything just worked. Would this be the
pip install that ended it all?
Enter virtual environments.
A virtual …
As part of my degree requirements, I enrolled in Information System Acquisition and Lifecycle with Dr. Ray Robinson (TA: Eric Abbott) for the Summer 2017 quarter.
The course name didn't have me expecting much. I thought I'd put off learning the material until the absolute last moment, procrastating with Python …
- Explore Plotly's new Dash library
- Discuss how to structure Dash apps using MVC
- Build interactive dashboard to display historical soccer results
The best way to learn something is to teach someone else.
Similar to how writing about side projects on this blog helped develop my …
Ever since Anaconda started getting in my way during this year's PyCon sprints, I have been moving my …
withstatements and the context manager protocol
- Implement context manager class to query MongoDB
withblock and increase code readability
I recently read Steve McConnell's Code Complete to level up my software development skill-set. The book has helped me become more deliberate about programming and problem solving in general. Before I sit down to write a single line of code, I take some time to plan out the work I am going to do versus code by the seat of my pants
Was reading the Python design FAQs and saw that immutable elements can be used as dictionary keys.
Wait a minute.... tuples are immutable, can they be used as dictionary keys? YES!
- Gather historical Premier League data
- Analyze results to generate standings at any given time
- Find teams that held the same position in the league table for longer than Manchester United's 104 days in 6th place (2016-17 PL season)
One of the biggest stories of the 2016-17 Premier League season has been the
(Note: This post is part of my reddit-scraper series)
- Running Scrapy spider as a script
- Scheduling script to run on PythonAnywhere cloud
Previously on Siv Scripts, we created a web scraping pipeline to pull Top Posts from Reddit and store them in a MongoDB collection. At this stage, we …
(Note: This post is part of my reddit-scraper series)
- Web frameworks intro
- Explore the Flask microframework
- Understand the Model-View-Controller (MVC) pattern
- Build Flask web app
Last time we started our web application adventure by learning how to generate dynamic HTML webpages from data stored in MongoDB using MongoEngine and …
(Note: This post is part of my reddit-scraper series)
- Overview of MongoDB
- Discussion of Object-Relational Mapping (ORM)
- Use MongoEngine to get items out of MongoDB
- Render HTML pages using Jinja2
- Interact with REST API to send emails with Requests
Previously on Siv Scripts, we implemented a web scraping pipeline …
- Use Python to display location data on interactive map
- Cluster points in a heatmap to visualize usage
One of my favorite things about the Python programming language is that I can always import a library to do the heavy lifting and focus my attention on application logic. It's a …
- Discuss advantages of using Scrapy framework
- Create Reddit spider and scrape top posts from list of subreddits
- Implement Scrapy pipeline to send scraped data into MongoDB
Wouldn't it be great if every website had a free API we could poll to get the data we wanted?
Sure, we could …