Solving Problems Using Code
Posted by Aly Sivji in Deep Dives
The main idea behind Software Architecture Methodologies such as Clean Architecture and Hexagonal Architecture is to create loosely coupled components that can be organized into layers. This way of writing code leverages the separation of concerns design principle and makes our application easier to maintain, i.e. we can easily …
Posted by Aly Sivji in Quick Hits
The paradox of choice in Python packaging tools makes dependency management more complex than it needs to be. In this Quick Hit, we will explore Python dependency management and demonstrate an easy-to-implement workflow to generate reproducible environments.
Note: this article assumes familiarity with virtual environments.
We use dependency management …
Podcasts allowed me to immerse myself in the tech industry during the earlly stages of my career change.
Talk Python, PythonBytes,
Test & Code,
Podcast.__init__,
Software Engineering Radio,
and the Atwood-Spolsky era Stack Overflow Podcast
were instrumental in leveling up my skills
and helping me get my first job.
In …
Posted by Aly Sivji in Quick Hits
I started my first tech job at a large corporation. We have lots of internal tools and systems that allow us to solve large-scale data problems. It's been an interesting change from startup life where platforms weren't as mature, where systems weren't as integrated into workflows and processes.
While I'm …
Posted by Aly Sivji in Deep Dives
Programming is writing code to solve problems. Software Engineering is the practice of using a structured process to solve problems. As engineers, we want to have a codebase we can change, extend, and refactor as required. Tests ensure our program works as intended and that changes to the codebase do …
Posted by Aly Sivji in Quick Hits
Bandit is a static analysis tool that can find common security issues in Python code. The command line utility scans .py files and generates a report detailing issues by confidence and severity.
I ran Bandit on a few of my repositories and found I was using md5 to hash passwords …
Posted by Aly Sivji in Quick Hits
In July, I released my first open source project. It's an apispec plugin that generates OpenAPI Specification (aka Swagger docs) for Falcon web applications.
Apispec's design made it easy to extend core functionality for a specific use case. I extended the apispec.BasePlugin class, overrode a couple of methods, and …
Posted by Aly Sivji in Quick Hits
A couple of weeks ago, I stumbled across the Python Sorting HOW TO. Learned that Python can sort objects by multiple attributes using operator.attrgetter and operator.itemgetter.
In this Quick Hit, I will show the Pythonic way of sorting objects by multiple attributes.
Note: This example is leveraging Python 3.7 dataclasses for convenience. Objects can be of any type.
Posted by Aly Sivji in Quick Hits
Python 3.7 was released a few weeks back. Lots of new features in this minor release: data classes, new breakpoint keyword, dicts preserving insertion order, amongst others.
One feature that hasn't got as much press, or really any press, is an update to the unittest.mock module in the Standard Library. Starting in Python 3.7, instances of Mock
A Python 3.6 feature that has flown under the radar is the revamped email package.
In this post we will explore email and walk through an example of creating and sending an email using the Python Standard Library.
emailThe email package is comprised of three major components …
@ Python's Matrix Multiplication Operator
Posted by Aly Sivji in Quick Hits
2017 will forever be etched in our memories as the year Python overtook R to become the leading language for Data Science. There are many factors that play into this: Python's simple syntax, the fantastic PyData ecosystem, and of course buy-in from Python's BDFL.
PEP 465 introduced the @ infix operator that is designated to be used for matrix multiplication. The acceptance and implementation of this proposal in Python 3.5 was a signal to the scientific community that Python is taking its role as a numerical computation language very seriously.
Posted by Aly Sivji in Quick Hits
Testing is one of my favourite things about programming. There is nothing like the piece of mind that comes from being able to modify code without having to worry about breaking something.
Like most Python programmers, I started testing using the PyUnit framework (aka unittest) included in Python's Standard Library. PyUnit is part of the xUnit
Posted by Aly Sivji in Quick Hits
Dynamic typing refers to programs checking variable types at runtime versus compile time. By not having to define the variable type as we write code, we can work faster and take advantage of duck typing.
On the other hand, we do miss out on some benefits that types bring. Similiar to how the principles of test-driven development force us to write smaller, more testable functions, thinking about types improves software design by making us cognizant of the values our variables hold as well as the values our functions can return.
Posted by Aly Sivji in Quick Hits
In Python, functions are objects. This means we can return them from other functions.
In this Quick Hit, we will use this property of functions to mock out an external API with fake data that can be used to test our internal application logic.
Note: I previously used Python functions to simulate the behavior of a case statement
Posted by Aly Sivji in Data Analysis
In this post, I will describe how to use BSON-NumPy to pull data out of Mongo and into pandas. While this library is still in the prototype stage, it's hard to to ignore the 10x speed improvement that comes from reading BSON documents directly into NumPy.
For this example, I will be analyzing a collection of my tweets
Posted by Aly Sivji in Quick Hits
I was floored when I discovered I could use tuples (and namedtuples) as Python dictionary keys.
Turns out this is not totally correct.
Started reading Fluent Python and learned that tuples are not always immutable. For instance, they can contain list elements...
Posted by Aly Sivji in Quick Hits
ChiPy hosts Python Project Night on the 3rd Thursday of every month. Groups of four are given an hour to work through a structured exercise with each person spending 15 minutes at the keyboard writing code. At the end of the hour, one group is selected to present their solution.
Posted by Aly Sivji in Quick Hits
A stack is a linear data structure that holds a collection of items. We can define operations performed in a stack from the point of view of a user:
.push() - add item to top.pop() - remove and return item from top.peek() - return top most element.is_empty() - True if stack …I love using the Command Line; it's my favourite part of programming. It wasn't always this way. I used to worry that the commands I entered would destory the delicate equilibrium where everything just worked. Would this be the pip install that ended it all?
Enter virtual environments.
A virtual …
Summary
- Explore Plotly's new Dash library
- Discuss how to structure Dash apps using MVC
- Build interactive dashboard to display historical soccer results
I spent a good portion of 2014-15 learning JavaScript to create interactive, web-based dashboards for a work project. I wrapped D3.js with Angular directives to create modular …
Summary¶
- Explore
withstatements and the context manager protocol- Implement context manager class to query MongoDB
- Convert
try...finallyblock towithblock and increase code readability
I recently read Steve McConnell's Code Complete to level up my software development skill-set. The book has helped me become more deliberate about programming and problem solving in general. Before I sit down to write a single line of code, I take some time to plan out the work I am going to do versus code by the seat of my pants
Posted by Aly Sivji in Quick Hits
Was reading the Python design FAQs and saw that immutable elements can be used as dictionary keys.
Wait a minute.... tuples are immutable, can they be used as dictionary keys? YES!
Posted by Aly Sivji in Data Analysis
Summary¶
- Gather historical Premier League data
- Analyze results to generate standings at any given time
- Find teams that held the same position in the league table for longer than Manchester United's 104 days in 6th place (2016-17 PL season)
One of the biggest stories of the 2016-17 Premier League season has been the rise
(Note: This post is part of my reddit-scraper series)
Summary
- Running Scrapy spider as a script
- Scheduling script to run on PythonAnywhere cloud
Previously on Siv Scripts, we created a web scraping pipeline to pull Top Posts from Reddit and store them in a MongoDB collection. At this stage, we …
(Note: This post is part of my reddit-scraper series)
Summary
- Web frameworks intro
- Explore the Flask microframework
- Understand the Model-View-Controller (MVC) pattern
- Build Flask web app
Last time we started our web application adventure by learning how to generate dynamic HTML webpages from data stored in MongoDB using MongoEngine and …
(Note: This post is part of my reddit-scraper series)
Summary
- Overview of MongoDB
- Discussion of Object-Relational Mapping (ORM)
- Use MongoEngine to get items out of MongoDB
- Render HTML pages using Jinja2
- Interact with REST API to send emails with Requests
Previously on Siv Scripts, we implemented a web scraping pipeline …
Posted by Aly Sivji in Data Analysis
Summary
- Use Python to display location data on interactive map
- Cluster points in a heatmap to visualize usage
One of my favorite things about the Python programming language is that I can always import a library to do the heavy lifting and focus my attention on application logic. It's a …
Summary
- Discuss advantages of using Scrapy framework
- Create Reddit spider and scrape top posts from list of subreddits
- Implement Scrapy pipeline to send scraped data into MongoDB
Wouldn't it be great if every website had a free API we could poll to get the data we wanted?
Sure, we could …