Solving Problems Using Code
Posted by Aly Sivji in Deep Dives
The main idea behind Software Architecture Methodologies such as Clean Architecture and Hexagonal Architecture is to create loosely coupled components that can be organized into layers. This way of writing code leverages the separation of concerns design principle and makes our application easier to maintain, i.e. we can easily …
I had two weeks of vacation planned around PyCon, but canceled it since it didn't make sense to take time off when I couldn't go anywhere because of COVID-19.
I was feeling pretty down when the conference start date passed. I had spent the past few months looking forward to …
Posted by Aly Sivji in Quick Hits
This canonical URL for this blog is alysivji.github.io. I have also set up nginx to forward requests to alysivji.com and sivji.com to this blog.
This is all well and good when are you running nginx out of a VM, but what if you are using the …
Posted by Aly Sivji in Quick Hits
Need to access a private Git repo inside of a Dockerfile during the docker build process.
The main idea is to use HTTP Basic Authentication while downloading files using the git command line client.
Create a Personal Access Token with repo scope using your Git provider's interface …
I started taking classes at The Second City Training Center in January this year. I was only able to complete Level A before the world shut down because of COVID-19.
It was a great experience and I was looking forward to more classes, but that wasn't in the cards for …
Posted by Aly Sivji in Quick Hits
Yesterday I gave an abridged version of my Data Science Workflows using Docker Containers talk at my employer's Biweekly Engineering Lunch session.
It had been a minute since I last gave this talk so I ran through the examples to make sure everything worked. Having all my examples containerized is …
As a learning exercise, I migrated all my side projects to Kubernetes. I was using Fluent Bit to forward application container logs to my ElasticSearch instance.
Everything was working as expected, but the cluster was starting to get expensive. DigitalOcean Kubernetes is competitively priced, but paying $80/month to run …
Posted by Aly Sivji in Quick Hits
The paradox of choice in Python packaging tools makes dependency management more complex than it needs to be. In this Quick Hit, we will explore Python dependency management and demonstrate an easy-to-implement workflow to generate reproducible environments.
Note: this article assumes familiarity with virtual environments.
We use dependency management …
Podcasts allowed me to immerse myself in the tech industry during the earlly stages of my career change.
Talk Python, PythonBytes,
Test & Code,
Podcast.__init__,
Software Engineering Radio,
and the Atwood-Spolsky era Stack Overflow Podcast
were instrumental in leveling up my skills
and helping me get my first job.
In …
Posted by Aly Sivji in Quick Hits
I started my first tech job at a large corporation. We have lots of internal tools and systems that allow us to solve large-scale data problems. It's been an interesting change from startup life where platforms weren't as mature, where systems weren't as integrated into workflows and processes.
While I'm …
Posted by Aly Sivji in Deep Dives
Programming is writing code to solve problems. Software Engineering is the practice of using a structured process to solve problems. As engineers, we want to have a codebase we can change, extend, and refactor as required. Tests ensure our program works as intended and that changes to the codebase do …
Posted by Aly Sivji in Quick Hits
I recently purchased a new computer and needed to transfer files from the old computer to the new one. Tried a USB drive, no luck. Tried copying files over the network via the macOS Finder, but the sheer number of files (~500,000) made this an exercise in futility.
Googling …
I started a new job in October and bought a 2018 MacBook Pro to celebrate. Purchased the 13" Touchbar with (mostly) maxed out components: 2.7GHz i7, 16GB RAM, and 1TB SSD. It's a huge step up from my late-2013 model; ~6x according to Moore's Law.
As soon as I …
Posted by Aly Sivji in Quick Hits
Bandit is a static analysis tool that can find common security issues in Python code. The command line utility scans .py files and generates a report detailing issues by confidence and severity.
I ran Bandit on a few of my repositories and found I was using md5 to hash passwords …
Posted by Aly Sivji in Quick Hits
In July, I released my first open source project. It's an apispec plugin that generates OpenAPI Specification (aka Swagger docs) for Falcon web applications.
Apispec's design made it easy to extend core functionality for a specific use case. I extended the apispec.BasePlugin class, overrode a couple of methods, and …
Posted by Aly Sivji in Quick Hits
A couple of weeks ago, I stumbled across the Python Sorting HOW TO. Learned that Python can sort objects by multiple attributes using operator.attrgetter and operator.itemgetter.
In this Quick Hit, I will show the Pythonic way of sorting objects by multiple attributes.
Note: This example is leveraging Python 3.7 dataclasses for convenience. Objects can be of any type.
Posted by Aly Sivji in Quick Hits
Earlier this week, I went through the fantastic PyCon 2018 tutorial, Building a Search Engine with Python + Elasticsearch. This required me to have a local installation of Elasticsearch and Kibana.
Local installation? Time to spin up Docker, more specifically, Docker Compose.
Docker-Compose is a tool that allows us to define …
Posted by Aly Sivji in Quick Hits
Python 3.7 was released a few weeks back. Lots of new features in this minor release: data classes, new breakpoint keyword, dicts preserving insertion order, amongst others.
One feature that hasn't got as much press, or really any press, is an update to the unittest.mock module in the Standard Library. Starting in Python 3.7, instances of Mock
The journey of a thousand miles begins with a single step.
Everything in life that's worth having requires effort. In tech, it also requires convincing somebody you are good enough to take a chance on.
A few months ago, I pivoted from data into web development because I wanted to …
(Note: This essay was written for my Foundations of Leadership course.)
Software Engineering is the discipline of Computer Science concerned with “developing and maintaining software systems that behave reliably and efficiently, are affordable to develop and maintain, and satisfy all the requirements that customers have defined for them” [1]. Creating …
Posted by Aly Sivji in Quick Hits
Being a Software Engineer isn't just about being effective in a specific programming language, it's about being able to solve any given problem using the tools at hand.
This week at work I have to extend the functionality of a WordPress plug-in so it can fit into our microservices-based backend …
A Python 3.6 feature that has flown under the radar is the revamped email package.
In this post we will explore email and walk through an example of creating and sending an email using the Python Standard Library.
emailThe email package is comprised of three major components …
@ Python's Matrix Multiplication Operator
Posted by Aly Sivji in Quick Hits
2017 will forever be etched in our memories as the year Python overtook R to become the leading language for Data Science. There are many factors that play into this: Python's simple syntax, the fantastic PyData ecosystem, and of course buy-in from Python's BDFL.
PEP 465 introduced the @ infix operator that is designated to be used for matrix multiplication. The acceptance and implementation of this proposal in Python 3.5 was a signal to the scientific community that Python is taking its role as a numerical computation language very seriously.
Posted by Aly Sivji in Quick Hits
Testing is one of my favourite things about programming. There is nothing like the piece of mind that comes from being able to modify code without having to worry about breaking something.
Like most Python programmers, I started testing using the PyUnit framework (aka unittest) included in Python's Standard Library. PyUnit is part of the xUnit
Posted by Aly Sivji in Quick Hits
Dynamic typing refers to programs checking variable types at runtime versus compile time. By not having to define the variable type as we write code, we can work faster and take advantage of duck typing.
On the other hand, we do miss out on some benefits that types bring. Similiar to how the principles of test-driven development force us to write smaller, more testable functions, thinking about types improves software design by making us cognizant of the values our variables hold as well as the values our functions can return.
Here’s to the new year. May she be a damn sight better than the old one, and may we all be home before she’s over.
End of year is the perfect time to reflect on the past and to plan for the future.
Posted by Aly Sivji in Quick Hits
In Python, functions are objects. This means we can return them from other functions.
In this Quick Hit, we will use this property of functions to mock out an external API with fake data that can be used to test our internal application logic.
Note: I previously used Python functions to simulate the behavior of a case statement
Posted by Aly Sivji in Data Analysis
In this post, I will describe how to use BSON-NumPy to pull data out of Mongo and into pandas. While this library is still in the prototype stage, it's hard to to ignore the 10x speed improvement that comes from reading BSON documents directly into NumPy.
For this example, I will be analyzing a collection of my tweets
Posted by Aly Sivji in Quick Hits
I was floored when I discovered I could use tuples (and namedtuples) as Python dictionary keys.
Turns out this is not totally correct.
Started reading Fluent Python and learned that tuples are not always immutable. For instance, they can contain list elements...
Posted by Aly Sivji in Deep Dives
The startup I work for is making a transition into a microservices-based architecture. I'm working with a Senior Engineer to design a new backend platform based off our current monolith.
We plan on having a bunch of REST API endpoints in containers that DevOps will orchestrate using Kubernetes. The current …
Truly wonderful the mind of a child is
It was an action packed week at work: we migrated our infrastructure to the Google Cloud Platform and into Kubernetes. Pods, nodes, deployments. Oh my!
Not that I helped much with the migration; I'm new and still learning the ropes.
I know …
The best laid schemes o' mice an' men
Gang aft agley
I started a new job as a Mathematician / Software Engineer at a Chicago-based healthcare startup in October. It's my first proper development job and I could not be more excited.
I get paid to write code. It's pretty awesome …
Posted by Aly Sivji in Quick Hits
Even though my development folders are well organized, I still have to feel my way around the filesystem when I'm looking for a project directory. It's like trying to find a lightswitch in a dark room: a cd here, an ls there, maybe a find when I'm stuck.
I always …
Posted by Aly Sivji in Quick Hits
ChiPy hosts Python Project Night on the 3rd Thursday of every month. Groups of four are given an hour to work through a structured exercise with each person spending 15 minutes at the keyboard writing code. At the end of the hour, one group is selected to present their solution.
2017 is the year I became deliberate about my approach to programming. Like most newbie developers, I hacked together spaghetti code around chunks of answers I found on StackOverflow (Praise Be).
I didn't care how I solved the problem, only that it was solved enough to cross off my todo …
Posted by Aly Sivji in Quick Hits
A stack is a linear data structure that holds a collection of items. We can define operations performed in a stack from the point of view of a user:
.push() - add item to top.pop() - remove and return item from top.peek() - return top most element.is_empty() - True if stack …I love using the Command Line; it's my favourite part of programming. It wasn't always this way. I used to worry that the commands I entered would destory the delicate equilibrium where everything just worked. Would this be the pip install that ended it all?
Enter virtual environments.
A virtual …
As part of my degree requirements, I enrolled in Information System Acquisition and Lifecycle with Dr. Ray Robinson (TA: Eric Abbott) for the Summer 2017 quarter.
The course name didn't have me expecting much. I thought I'd put off learning the material until the absolute last moment, procrastating with Python …
Summary
- Explore Plotly's new Dash library
- Discuss how to structure Dash apps using MVC
- Build interactive dashboard to display historical soccer results
I spent a good portion of 2014-15 learning JavaScript to create interactive, web-based dashboards for a work project. I wrapped D3.js with Angular directives to create modular …
The best way to learn something is to teach someone else.
With this in mind, I started creating and presenting technical talks on a variety of subjects: Big Data in Healthcare, Python language features, graph algorithms, etc.
Similar to how writing about side projects on this blog helped develop my …
Summary¶
- Explore
withstatements and the context manager protocol- Implement context manager class to query MongoDB
- Convert
try...finallyblock towithblock and increase code readability
I recently read Steve McConnell's Code Complete to level up my software development skill-set. The book has helped me become more deliberate about programming and problem solving in general. Before I sit down to write a single line of code, I take some time to plan out the work I am going to do versus code by the seat of my pants
Posted by Aly Sivji in Quick Hits
Was reading the Python design FAQs and saw that immutable elements can be used as dictionary keys.
Wait a minute.... tuples are immutable, can they be used as dictionary keys? YES!
Posted by Aly Sivji in Data Analysis
Summary¶
- Gather historical Premier League data
- Analyze results to generate standings at any given time
- Find teams that held the same position in the league table for longer than Manchester United's 104 days in 6th place (2016-17 PL season)
One of the biggest stories of the 2016-17 Premier League season has been the rise
(Note: This post is part of my reddit-scraper series)
Summary
- Running Scrapy spider as a script
- Scheduling script to run on PythonAnywhere cloud
Previously on Siv Scripts, we created a web scraping pipeline to pull Top Posts from Reddit and store them in a MongoDB collection. At this stage, we …
(Note: This post is part of my reddit-scraper series)
Summary
- Web frameworks intro
- Explore the Flask microframework
- Understand the Model-View-Controller (MVC) pattern
- Build Flask web app
Last time we started our web application adventure by learning how to generate dynamic HTML webpages from data stored in MongoDB using MongoEngine and …
(Note: This post is part of my reddit-scraper series)
Summary
- Overview of MongoDB
- Discussion of Object-Relational Mapping (ORM)
- Use MongoEngine to get items out of MongoDB
- Render HTML pages using Jinja2
- Interact with REST API to send emails with Requests
Previously on Siv Scripts, we implemented a web scraping pipeline …
Posted by Aly Sivji in Data Analysis
Summary
- Use Python to display location data on interactive map
- Cluster points in a heatmap to visualize usage
One of my favorite things about the Python programming language is that I can always import a library to do the heavy lifting and focus my attention on application logic. It's a …
Summary
- Discuss advantages of using Scrapy framework
- Create Reddit spider and scrape top posts from list of subreddits
- Implement Scrapy pipeline to send scraped data into MongoDB
Wouldn't it be great if every website had a free API we could poll to get the data we wanted?
Sure, we could …