This is an excerpt from Learn Python the Hard Way, 5th Edition exercise 55. I felt this exercise was so generally useful that I should copy it to the blog so people can refer to it if they're struggling with documentation. Keep in mind that I don't think Pandas' docs are bad, they just need a cohesive course to bring it all together.
This exercise is going to teach two very important skills. First, you'll learn about Pandas and its DataFrame
construct. This is the most common way to work with data in the Python Data Science world. Second, you're going to learn how to read typical programmer documentation. This is a far more useful skill as it applies to every single programming topic you will ever encounter. In fact, you should think of this exercise as using Pandas to teach you how to read documentation.
There's a concept in painting called "the gestalt." The gestalt of a painting is how all of the parts of a painting fit together to create a single cohesive experience. Imagine I paint a portrait of you and create the most perfect mouth, eyes, nose, ears, and hair you've ever seen. You see each part is perfect and then you pull back, and when placed together...they're all wrong. The eyes are too close together, the nose is too dark compared to everything else, and the ears are different sizes. On their own, they're perfect, but when combined into a finished work of art they're awful because I didn't also pay attention to the gestalt of the painting.
For something to have high quality you have to pay attention to the qualities of each individual piece, and how those pieces fit together. Programmer documentation is frequently like this awful portrait with perfect features that don't fit together. Programmers will very clearly and accurately describe every single function, the nuances of every option to those functions, and every class they made. Then completely ignore any documentation that describes how those pieces fit together or how to use them to do anything.
This kind of documentation is everywhere. Look at Python's original sqlite3 documentation then compare it to the latest version that finally has how to use placeholders. That's a fairly important topic you need for good security and it's...just casually ignored for about a decade?
Learning from this documentation requires a particular style of reading that's more active. That's what you will learn in this exercise.
I won't force you to suffer through really bad documentation. Instead you'll take a baby step and learn how to read documentation using the Pandas documentation. The Pandas documentation is good. It at least has a quick start guide to get you going, cookbooks, how-to guides, an API reference, and lots of examples. Everything is clearly described, but when you read it you're still kind of lost because it's a lot of documentation spread all over with no clear curriculum.
This is where active reading comes into play, and it's something I've had you do for this entire course by making you type in code and change it. To read programmer documentation actively means you have to type in the code as you read, change the code to find more, and apply what you learn to your own problems to learn how to use what you learn. Your goal with this process is to find the gestalt the programmers ignored.
The very first thing you should do is find the docs. You might laugh but sometimes that's a difficult first step. Important questions to ask:
Let's go through the Pandas documentation and answer each of these questions:
DataFrame
so any documents that cover that are useful. If you wanted to process many .csv
files then you'd look for documents explaining loading and saving .csv
files.What do you do if most of these have "No" answers? What if the project only has auto-generated API docs and not a single document or example explaining how to use the API? First, do you have to use this pile of garbage? Life's too short to use software that not even the developers care about, so maybe just don't use it. If you really want to use it or have to use it, then you have two complimentary strategies:
If the project has everything you need then you have a couple different strategies:
These options are not mutually exclusive. Start with one option, and if it's not working, switch to another. Keep doing this until you understand enough to use the project or study further.
In the Pandas example we have everything we need except an overall curriculum telling us where to go, so that's why you need a strategy. I have three complimentary strategies in this situation:
This will seem counter-intuitive but when reading programmer documentation you will have more success if you start with the code, then read about it. This works because the code is something you can experience and that experience gives you better understanding of what's being said in the documentation.
Let's look at the 10 minute guide to Pandas as an example. Right away there's this code:
import numpy as np
import pandas as pd
s = pd.Series([1, 3, 5, np.nan, 6, 8])
# this prints it in Jupyter
s
dates = pd.date_range("20130101", periods=6)
# print it in Jupyter
dates
This code is spread across multiple short descriptions about the code, so you type each example in first. Once it's working, change it around and then read the descriptions. This will make the descriptions easier to understand.
However, if you read the descriptions first this is what you read:
Customarily, we import as follows.
Creating a Series by passing a list of values, letting
pandas create a default RangeIndex.
Creating a DataFrame by passing a NumPy array with a
datetime index using date_range() and labeled columns:
Those on their own or with a quick glance at the code make almost no sense. After you get the code working these sentences help fill in gaps in your understanding. They also link to more documentation on what you just used.
After you get a piece of code working take the time to break it so you can see how errors are handled. One massive blocker for beginners is deciphering the convoluted error messages programming languages produce. There's almost a weird art to reading them and using Google to find the answer. One of the ways to learn the "language of terrible errors" is to expose yourself to as many errors as possible on purpose so you can study them.
The second thing to do is ask if you can do something and then try to do it. You'll ask, "How do I give a Series a different index?" Or, you might ask, "How can I pass a Series to a DataFrame?" The kinds of changes you want to focus on are combinations of things you just learned.
A key aspect of learning to code (or anything) is explaining what you've learned back to yourself. The best way to do this while you're working is to have a notes.txt
file in the directory where you're putting the code you write. In this notes.txt
write down questions you have, things you discover, and comments about what you're learning.
Another important part of the notes.txt
is links. You should be recording links to what you read or what you need to read as you work. This will help you later when you need to remember where you read about something.
The entire purpose of this last module is to move you from someone who knows Python to someone who can use Python to express their own ideas. After you feel you have enough understanding of the project you should try to make something of any size with it. This is when you will switch to relying more on API reference than the other documentation.
If you're stuck and can't think of anything to create, then take an example from the cookbook or how-to documents documentation and modify it to do something new. Maybe you have it load the data from a SQL database or change the data used.
When I think of painting, writing, and programming I think of them as mediums for articulating my automatic thoughts, experiences, and feelings so I can consciously understand them. Painting helps me understand what I see. Writing helps me understand what I know and feel. Programming helps me understand how to do something.
I spend all day using my eyes to see the world, but it's only when I try to paint what I see that I start to consciously understand what I'm seeing. Painting forces me to consciously understand the automatic way my visual system processes the world.
Programming forces me to structure my understanding of how something works into logical steps and structures. After I turn a process or idea into code I understand how it could actually work.
Writing helps me organize my almost random thoughts into a coherent conscious structure. The act of organizing all of my thoughts into an essay that makes sense and flows naturally helps me further understand my ideas.
More importantly, each of these mediums--painting, programming, and prose--force me to explore what I don't know. Externalizing my knowledge in these ways gives me a glimpse into my brain. I can look at a painting and say, "Well it looks like I have no idea what this flower actually looks like." I can study code and see, "I clearly have no idea how this algorithm is supposed to work." I can read through an essay and see, "I really don't know how to explain what I'm feeling about this topic."
This is why you should write about what you've learned. You don't have to show it to anyone or be a good writer. Your writing doesn't have to be original. I'll tell you, 99% of all writing is not original. The point is not to impress other people with how clever a writer you are. The point is to explain to yourself what you know so you can see if you actually learned something.
For Step 7 I want you to write at least 8-10 paragraphs teaching someone else what you've just learned about DataFrames
. How would you explain the DataFrame
to someone who knows Python? What is your best advice on how to use it? Are there any things to avoid when using it?
Another option is to write your own curriculum to learn the Pandas DataFrame
. If you were to write a curriculum for someone else what links should they read in order to best understand it. For each link describe what they learn at that stage, and how it relates to the previous link they studied.
A final option is to use Jupyter to create a Notebook that demonstrates and explains everything someone else would need to learn. I suggest first write a short version of the curriculum idea, and then turn that into a structured notebook that follows the curriculum.
The final step in this process is to ask yourself, "What's the big picture for this project?" This is a more abstract step and should fall out naturally from your writings and notes, but being able to summarize the project will give you a mental framework to hold everything else you learn.
Your understanding of the project might be different from the authors, but your description of the project is more for you than a general statement for everyone.
If I were to summarize the purpose of Pandas I might have several "gestalt statements":
Am I right? What did you come up with? Does it help you understand Pandas better?
While it is useful for you to learn how to read documentation and devise your own curriculum, I also feel you might need one provided by me. The problem is, projects frequently change and I want this course to last longer than the next version of Pandas. To solve this problem, I'll have a dedicated page of LPTHW errata that will also include my idea for a Pandas curriculum you can follow.
]]>If you play a game like Go or Chess you know the rules are fairly simple, yet the games they enable are extremely complex. Really good games have this unique quality of simple rules with complex interactions. Programming is also a game with a few simple rules that create complex interactions, and in this exercise we're going to learn what those rules are.
Before we do that, I need to stress that you most likely won't use these rules directly when you code. There are languages that do utilize these rules directly, and your CPU uses them too, but in daily programming you'll rarely use them. If that's the case then why learn the rules?
Because these rules are everywhere, and understanding them will help you understand the code you write. It'll help you debug the code when it goes wrong. If you ever want to know how the code works you'll be able to "disassemble" it down to its basic rules and really see how it works. These rules are a cheat code. Pun totally intended.
I'm also going to warn you that you are not expected to totally understand this right away. Think of this exercise as setting you up for the rest of the exercises in this module. You're expected to study this exercise deeply, and when you get stuck, move on to the next exercises as a break. You want to bounce between this one and the next ones until the concepts "click" and it starts to make sense. You should also study these rules as deeply as you can, but don't get stuck here. Struggle for a few days, move on, come back, and keep trying. As long as you keep trying you can't actually "fail".
All programs are a sequence of instructions which tell a computer to do something. You've seen Python doing this already when you type code like this:
x = 10
y = 20
z = x + y
This code starts at line 1, goes to line 2, and so on until the end. That's a sequence of instructions, but inside Python these 3 lines are converted into another sequence of instructions that look like this:
LOAD_CONST 0 (10) # load the number 10
STORE_NAME 0 (x) # store that in x
LOAD_CONST 1 (20) # load the number 20
STORE_NAME 1 (y) # store that in y
LOAD_NAME 0 (x) # loads x (which is 10)
LOAD_NAME 1 (y) # loads y (which is 20)
BINARY_ADD # adds those
STORE_NAME 2 (z) # store the result in z
That looks totally different from the Python version, but I bet you could probably figure out what this sequence of instructions is doing. I've added comments to explain each instruction, and you should be able to connect it back to the Python code above.
I'm not joking. Take some time right now to connect each line of the Python code to the lines of this "byte code". Using the comments I provided I'm positive you can figure it out, and doing so might turn on a light in your head about the Python code.
It's not necessary to memorize this or even understand each of these instructions. What you should realize is your Python code is being translated into a sequence of simpler instructions that tell the computer to do something. This sequence of instructions is called a "byte code" because it's usually stored in a file as a sequence of numbers a computer understands. The output you see above is usually called an "assembly language" because it's a human "readable" (barely) version of those bytes.
These simpler instructions are processed starting at the top, do one small thing at a time, and go to the end when the program exits. That's just like your Python code but with a simpler syntax of INSTRUCTION OPTIONS
. Another way to look at this is each part of x = 10
might become its own instructions in this "byte code."
That's the first rule of The Game of Code: Everything you write eventually becomes a sequence of bytes fed to a computer as instructions for what the computer should do.
To get this output yourself, you use a module called dis which stands for "disassemble." This kind of code is traditionally called "byte code" or "assembly language", so dis
means to "dis-assemble." To use dis
you can import it and use the dis()
function like this:
from dis import dis
dis('''
x = 10
y = 20
z = x + y
''')
In this Python code I'm doing the following:
dis()
function from the dis
module.dis()
function, but I give it a multi-line string using '''
.dis()
function with ''')
.When you run this in Jupyter you'll see it dump the byte code like I have above, but maybe with some extras we'll cover in a minute.
When you run Python (version 3) these bytes are stored in a directory named __pycache__
. If you put this code into a ex19.py
file and then run it with python ex19.py
you should see this directory.
Looking in this directory you should see a bunch of files ending in .pyc
with names similar to the code that generated them. These .pyc
files contain your compiled Python code as bytes.
When you run dis()
you're printing a human readable version of the numbers in the .pyc
file.
A sequence of simple instructions like LOAD_CONST 10
is not very useful. Yay! You can load the number 10! Amazing! Where code starts to become useful is when you add the concept of the "jump" to make this sequence non-linear. Let's look at a new piece of Python code:
while True:
x = 10
To understand this code we have to foreshadow a later exercise where you learn about the while-loop
. The code while True:
simply says "Keep running the code under me x = 10
while True
is True
." Since True
will always be True
this will loop forever. If you run this in Jupyter it will never end.
What happens when you dis()
this code? You see the new instruction JUMP_ABSOLUTE
:
dis("while True: x = 10")
0 LOAD_CONST 1 (10)
2 STORE_NAME 0 (x)
4 JUMP_ABSOLUTE 0 (to 0)
You saw the first two instructions when we covered the x = 10
code, but now at the end we have JUMP_ABSOLUTE 0
. Notice there's numbers 0
, 2
, and 4
to the left of these instructions? In the previous code I cut them out so you wouldn't be distracted, but here they're important because they represent locations in the sequence where each instruction lives. All JUMP_ABSOLUTE 0
does is tell Python to "jump to the instruction at position 0" which is LOAD_CONST 1 (10)
.
With this simple instruction we now have turned boring straight line code into a more complex loop that's not straight anymore. Later we'll see how jumps combine with tests to allow even more complex movements through the sequence of bytes.
You may have noticed that the Python code reads as "while True is True set x equal to 10" but the dis()
output reads more like "set x equal to 10, jump to do it again." That's because of Rule #1 which says we have to produce a sequence of bytes only. There is no nested structures, or any syntax more complex than INSTRUCTION OPTIONS
allowed.
To follow this rule Python has to figure out how to translate its code into a sequence of bytes that produce the desired output. That means, moving the actual repetition part to the end of the sequence so it will be in a sequence. You'll find this "backwards" nature comes up often when looking at byte codes and assembly language.
Yes, technically a JUMP instruction is simply telling the computer to process a different instruction in the sequence. It can be the next one, a previous one, or one in the future. The way this works is the computer keeps track of the "index" of the current instruction, and it simply increments that index.
When you JUMP you're telling the computer to change this index to a new location in the code. In the code for our while loop (below) the JUMP_ABSOLUTE
is at index 4
(see the 4 to the left). After it runs, the index changes to 0
where the LOAD_CONST
is located, so the computer runs that instruction again. This loops forever.
0 LOAD_CONST 1 (10)
2 STORE_NAME 0 (x)
4 JUMP_ABSOLUTE 0 (to 0)
A JUMP is useful for looping, but what about making decisions? A common thing in programming is to ask questions like:
"If x is greater than 0 then set y to 10."
If we write this out in simple Python code it might look like this:
if x > 0:
y = 10
Once again, this is foreshadowing something you'll learn later, but this is simple enough to figure out:
x
is greater than >
0.y = 10
.if x > 0:
? That is called a "block" and Python uses indentation to say "this indented code is part of the code above it."x
is NOT greater than 0
then Python will JUMP over the y = 10
line to skip it.To do this with our Python byte code we need a new instruction that implements the testing part. We have the JUMP. We have variables. We just need a way to compare two things and then a JUMP based on that comparison.
Let's take that code and dis()
it to see how Python does this:
dis('''
x = 1
if x > 0:
y = 10
''')
0 LOAD_CONST 0 (1) # load 1
2 STORE_NAME 0 (x) # x = 1
4 LOAD_NAME 0 (x) # load x
6 LOAD_CONST 1 (0) # load 0
8 COMPARE_OP 4 (>) # compare x > 0
10 POP_JUMP_IF_FALSE 10 (to 20) # jump if false
12 LOAD_CONST 2 (10) # not false, load 10
14 STORE_NAME 1 (y) # y = 10
16 LOAD_CONST 3 (None) # done, load None
18 RETURN_VALUE # exit
# jump here if false
20 LOAD_CONST 3 (None) # load none
22 RETURN_VALUE # exit
The key part of this code is the COMPARE_OP
and POP_JUMP_IF_FALSE
:
4 LOAD_NAME 0 (x) # load x
6 LOAD_CONST 1 (0) # load 0
8 COMPARE_OP 4 (>) # compare x > 0
10 POP_JUMP_IF_FALSE 10 (to 20) # jump if false
Here's what this code does:
LOAD_NAME
to load the x
variable.LOAD_CONST
to load the 0
constant.COMPARE_OP
which does the >
comparison and leaves a True
or False
result for later.POP_JUMP_IF_FALSE
makes the if x > 0
work. It "pops" the True
or False
value to get it, and if it reads False
it will JUMP
to instruction 20.y
if the comparison is False
, but if the comparison is True
then Python just runs the next instruction which starts the y = 10
sequence.Take some time walking through this to try to understand it. If you have a printer, try printing it out and set x
to different values manually, then trace through how the code works. What happens when you set x = -1
.
In the above code I'm skipping over exactly how Python "pops" the value to read it, but it's storing it in something called a "stack." For now just think of it as a temporary storage place that you "push" values into, and then "pop" them off. You really don't need to go much deeper than that at this stage in your learning. Just understand the effect is to get the result of the last instruction.
COMPARE_OP
used in loops too?Yes, and you could probably figure out how that works right now based on what you know. Try to write a while-loop
and see if you can get it to work with what you know now. Don't worry if you can't though as we'll be covering this in later exercises.
You need some way to keep track of changing data while the code operates, and this is done with "storage." Usually this storage is in the computer's memory and you create names for the data you're storing in memory. You've been doing this when you write code like this:
x = 10
y = 20
z = x + y
In each of the above lines we're making a new piece of data and storing it in memory. We're also giving these pieces of memory the names x
, y
, and z
. We can then use these names to "recall" those values from memory, which is what we do in z = x + y
. We're just recalling the value of x
and y
from memory to add them together.
That's the majority of the story, but the important part of this little rule is that you almost always use memory to control tests.
Sure, you can write code like this:
if 1 < 2:
print("but...why?")
That's pointless though since it's just running the second line after a pointless test. 1
is always less than 2
so it's useless.
Where tests like COMPARE_OP
shine is when you use variables to make the tests dynamic based on calculations. That's why I consider this a "rule of the Game of Code" because code without variables isn't really playing the game.
Take the time to go back through the previous examples and identify the places where LOAD
instructions are used to load values, and STORE
instructions are used to store values into memory.
The final rule of the Game of Code is how your code interacts with the outside world. Having variables is great, but a program that only has data you've typed into the source file isn't very useful. What you need is input and output.
Input is how you get data into your code from things like files, the keyboard, or the network. You've already used open()
and input()
to do that in the last module. You accessed input every time you opened a file, read the contents, and did something with them. You also used input when you've use...input()
to ask the user a question.
Output is how you save or transmit the results of your program. Output can be to the screen with print()
, to a file with file.write()
, or even over a network.
The only problem with input and output at this point is the byte code output is a little more complicated. Let's look at a simple one:
from dis import dis
dis("input('Yes? ')")
0 LOAD_NAME 0 (input)
2 LOAD_CONST 0 ('Yes? ')
4 CALL_FUNCTION 1
6 RETURN_VALUE
This dis()
run doesn't help much because now we're getting into an advanced topic we'll cover later called "functions", so let's stop there and pull these rules together.
Taking the 5 Rules we have the following Game of Code:
While this seems simple these little rules create very complicated software. Video games are a great example of very complicated software that does this. A video game reads your controller or keyboard as input, updates variables that control the models in the scene, and uses advanced instructions that render the scene to your screen as output.
Your next step is to study the Python that uses all of these rules.
The exercises after this will then reference the concepts here to explain the concepts of while-loops
, if-statements
, boolean logic, and the more advanced applications of these rules. By learning the rules it's hopefully going to make it easier to understand how each thing works, but you tell me. Did this make sense?
Did you know when you sign a contract with a publisher you have to update your books? Neither did I! I'm mostly joking but I've had enough demands and complaints from readers of Learn Python the Hard Way that it was time for an update, but I was too deep in JavaScript land to have bandwidth for it. Then last month my Publisher started bothering me for updates as well, so now I'm on the hook for a new edition.
I was reluctant to work on anything new related to Python due its stagnation in the web development space, but a few recent events have changed my mind: Codon and the popularity of Data Science.
I'm really excited about Codon and I'll be playing with it in the near future. I have a couple fun projects in mind that specifically leverage Codon's abilities, and I'll hopefully have a few articles about Codon in practice. Mostly I'm interested in how Codon compiles Python, and it's ability to interface with C fairly easily. It also seems to be really well designed and apparently it can embed the cpython interpreter for those cases where you absolutely have to run Python.
Here's their example showing the @python
decorator embedding the Python interpreter when you need it:
@python
def scipy_eigenvalues(i: List[List[float]]) -> List[float]:
# Code within this block is executed by the Python interpreter,
# so it must be valid Python code.
import scipy.linalg
import numpy as np
data = np.array(i)
eigenvalues, _ = scipy.linalg.eig(data)
return list(eigenvalues)
print(scipy_eigenvalues([[1.0, 2.0], [3.0, 4.0]])) # [-0.372281, 5.37228]
What's amazing about this design is it's combined with very a easy C FFI interface due to Codon's use of LLVM as the backend:
from C import pow(float, float) -> float
pow(2.0, 2.0) # 4.0
# Import and rename function
# cobj is a C pointer (void*, char*, etc.)
# None can be used to represent C's void
from C import puts(cobj) -> None as print_line
print_line("hello".c_str()) # prints "hello"; c_str() converts Codon str to C string
You can even inline the LLVM IR directly in your code for the rare cases when the compiler needs a little help:
@llvm
def popcnt(n: int) -> int:
declare i64 @llvm.ctpop.i64(i64)
%0 = call i64 @llvm.ctpop.i64(i64 %n)
ret i64 %0
print(popcnt(42)) # 3
I have a few projects in mind that could use this in the future, but I will need to fully review it and I do have some reservations about its license. More on that later.
Codon is awesome, and it's definitely getting me interested in Python again, but the real winner in the Python world is Data Science. Right now AI, Data Science, and Machine Learning are hot, and they're the primary thing Python is being used for. I think most of the students who contact me wanting to learn Python are interested in the world of Data Science and not web development or "backend" programming. I think languages like Go, Rust, and JavaScript have largely supplanted Python for general systems programming, and there's some evidence from Github that shows this trend.
Here's a list of the top 20 Python projects on Github by stars. Do you notice something?
Name | Stars | Category |
---|---|---|
public-apis | 241137 | Scraping |
system-design-primer | 220942 | Systems |
awesome-python | 169124 | Education |
TheAlgorithms/Python | 159025 | Education |
Python-100-Days | 136201 | Education |
Auto-GPT | 135364 | ML/DS |
youtube-dl | 120574 | Scraping |
transformers | 101767 | ML/DS |
stable-diffusion-webui | 78293 | ML/DS |
thefuck | 77571 | Systems |
django | 71010 | Web |
HelloGitHub | 69243 | Education |
pytorch | 67235 | ML/DS |
flask | 63061 | Web |
home-assistant/core | 60641 | Systems |
awesome-machine-learning | 58946 | ML/DS |
keras | 58428 | ML/DS |
fastapi | 58363 | Web |
ansible | 57471 | Systems |
scikit-learn | 54360 | ML/DS |
cpython | 53404 | Python |
manim | 51485 | Graphing |
funNLP | 50741 | ML/DS |
requests | 49698 | Scraping |
face_recognition | 48357 | ML/DS |
yt-dlp | 47975 | Scraping |
PayloadsAllTheThings | 47941 | Security |
you-get | 47412 | Scraping |
scrapy | 47303 | Scraping |
localstack | 47235 | Systems |
If we count the projects by their categories we have the following breakdown:
Project Type | Count |
---|---|
ML/DS | 9 |
Scraping | 6 |
Education | 4 |
Systems | 4 |
Web | 3 |
Security | 1 |
Graphing | 1 |
Python | 1 |
It's almost entirely data science projects, especially if you consider things like Graphing and Scraping being something primarily used in Data Science. If you do that then 80% of the top most popular projects on Github are related to Data Science. This fits with the wild success of Data Science, AI, and Machine Learning in the last five years, and the relative lack of innovation in Python's other use cases such as web development and systems management.
Now, if you think this isn't a fair analysis of popularity I want to stress that everyone is also quoting this as a measure of Python's general popularity. You aren't allowed to rave about Python climbing to the top of the Github stars chart and then balk at the suggestion that, actually, it's Data Science that's popular. Either stars are meaningless and Python's not popular, or stars are important and Python Data Science is popular.
Learn Python the Hard Way has always been focused on Pre-Beginners in that it assumes nothing and aims at building the knowledge someone needs to eventually learn the topic. My approach is not to teach someone to be a master of the subject, but to teach them all the things other writers assume "beginners" already know. If you've ever read a book that starts with print("Hello World")
then jumps to "a monad is just a monoid in the category of endorfunctors" then my book teaches you what that author assumes you know.
Focusing on Data Science in my style means that I won't teach you the entire world of Data Science, since that's already covered by many more qualified people than me. My goal in the new Learn Python the Hard Way is to teach you everything about Python programming that those courses assume you already know. When you're done with my book you'll have the skills you need to then understand other books.
A secondary goal in the new book is to get you familiar with the basic tools used in Data Science, like Jupyter, Pandas, Anaconda, and low level topics like data munging, testing, and graphing. I won't go extremely deep into these topics, but having a familiarity with them will make other books easier to understand.
Finally, I'm going to target the new book at a secondary audience of people who are knowledgeable of Data Science, but maybe they feel their Python skills are lacking. This would be anyone who has impostor syndrome when they write Python code and who wants to feel more confident in their basic Python knowledge. I want to "upgrade" people from strictly using Jupyter to creating full Python projects with automated testing for repeatable results in addition to detailed explanations of basic Python topics.
I've submitted the following outline to my publisher, but I'll be changing this as I work through the exercises using Jupyter. Remember that the goal of this course is not to craft a grand master of Python Data Science, but to teach a Pre-Beginner the basics of Python most other books assume you have.
First I start off with the usual first set of exercises to get people into controlling a computer with language, but I'll be using Anaconda and Jupyter exclusively to get people started.
Then I move on to simple I/O but focused on how to use Jupyter to create the files and open them. It's at this point that I'll start "weening" people off Jupyter and start making little scripts using a simple external text editor. This will help when they want to move their work into an external project to share, or start adding more traditional Python resources such as automated testing, deployment, and package sharing.
It's at this point I can start introducing simple functional programming and data structures. There's some people who hang out on Stack Overflow yelling at beginners that think you should start with OOP right away, but there's a significant problem with this belief:
You can construct all of Object Oriented Programming from just functions and dicts. You can't construct functions and dicts from objects and classes without first explaining functions and dicts.
With that in mind I'll teach functions and functional programming first so that later I can show them how to build their own Object Oriented System from first principles.
With functions covered I can then get into deeper into strings and the basics of simple data types:
After learning an introductory level of these basic data structures, and the previous information on jumps and functions, it's time to get into boolean logic, loops, and if-statements
. Once again, if you know about jumps, and you know about boolean tests, then you can understand if-statements
. If you understand jumps and if-statements
then you can figure out basic looping. After that it's a process of combining data structures with more advanced loops like for-loops
:
It's at this point that I've taught the fundamental parts of how programming works, so everything after this is either practicing those concepts or adding on concepts that use those fundamentals.
Object Oriented Programming is an example of something that's far easier to teach once someone knows about dict
and functions, so we get into this here. In the past I tried to "sneak" in an understanding of OOP with a weird method, but my JavaScript course has taught me that it's easier to teach people how to build their own basic OOP system with dict
and closures, then show how that "maps" to the built-in OOP of the language:
Once they reach this point they're probably ready to move off Jupyter and learn how to create a regular Python project with automated testing. This will cover more traditional developer tools, and I might throw in an exercise that has a CLI crash course right here rather than as an appendix.
Finally, this is a book about getting someone ready to study other Data Science books, so I'll spend the final exercises lightly touching on various data science topics. Things like Data Munging, DataFrames, Graphing and simple analysis. I might add in a bit of SQL but I'm not sure if I could cover enough SQL in a few exercises to be useful.
That's the plan so far. If you have feedback on this list of topics based on what you do as a Data Scientist then feel free to contact me @lzsthw on Twitter. My only warning is, if you're looking to get me to teach people that one thing you found annoying at your last job or to turn them into Python true believers, then don't bother. I don't indoctrinate people. I create independent learners who question what they learn and form their own opinions.
Inflation is kicking everyone's grapes and I'm no different, so the price on the finished course will be $59 going forward. However, I will offer an upgrade price for the difference if you already bought my previous version, and I'll give a free upgrade to anyone who buys (or has bought) the current version of Learn Python the Hard Way after April 2023.
This means if you bought it April 30th, 2023 you can pay $20 for an upgrade. If you bought it after May 1, 2023 you'll get a free upgrade. You'll also get early access to the content as I work on it and access to my Discord for help and feedback just like with Learn JavaScript the Hard Way.
]]>I tested Youtube's streaming in preparation for the JavaScript Level 1 live training and I have to say, it's awful. I'm actually kind of shocked how a company with this much money can be so bad at simple UI problems After this test I'm convinced it's going to be easier to stream on Twitch for you and me, but I want to lay out exactly what the problems are so people understand why.
Here is my, "Ten Reasons Youtube's Streaming is Awful":
The vast majority of the problems in this list all come from Youtube not understanding or figuring out what the ideal workflow should be for a streamer. Currently, most people do this:
When you stream to Youtube directly then you'd end up with two videos with basically the same content on the platform, both competing for viewers. As far as I can tell, this is Youtube's imagined workflow for streamers:
Obviously you could save locally and edit just like with Twitch, but then...why bother using Youtube if you're doing that anyway? This is why many people who stream to Youtube seem to have a "VODs" channel, where they put the fairly boring unedited streams so it doesn't compete with their main channel.
This is such a weird design that I have to think they just kind of bolted streaming on, and the rest of the criticisms in this list confirm this theory.
Something I could never figure out about Youtube streams is why they leave huge "Stream Starting Soon..." chunks at the start of their published video. Then I saw this Video by the Spiffing Britt on "How to Edge Youtube" where he effectively starts announcing a live stream but never starts it. Youtube apparently will show your soon to be live stream to tons of people in the first hour before a stream starts, so if you constantly bump the starting time then you trick Youtube into showing your video to more and more people. You really should watch the video to see him do it.
So, all those videos you see with 2 hour "Starting Soon..." times are probably doing this, and then the Youtube editor has so many problems that people just don't bother trimming that off so they don't lose all those impressions. Now, you can trim the stream, but the UI is really bad, has problems just playing the video, and then can't actually edit the stream.
What? Yes, Youtube's editor can't actually edit the stream. You see, Youtube comes from Google, and Google is famous for claiming that "real programmers don't delete." They spent about a decade trying to convince everyone that never deleting something is the way the world should work, and most of their infrastructure has this problem. Remember that debacle where Google+ let Developers access 52 million accounts. One thing I remember from this debacle was that because Google famously refused to delete anything people had access to everyone's deleted messages too. Feel free to correct me if I'm wrong, but I very much remember this being one of the big complaints.
Why does this matter? Well, that design "philosophy" seems to be present at Youtube, because if you trim a stream you just uploaded then Youtube won't edit that stream. No, you have to Save a Copy. That saved copy is then put into your Channels videos as if you uploaded it. So now you have two videos competing for viewers instead of one video that's a bit cleaned up.
Why would they do this? Oh, obviously the team that works on Youtube feels they're competing with the team working on Live streams, so they refuse to make it possible to trim videos so that they're getting juicy video uploads for the stats they need to get next year's bonuses. You laugh but I bet that's why, and it's a hilariously bad decision with huge costs.
Costs? Yes, if I edge Youtube with a 1 hour lead up to my stream, and then save that video, it's in Youtube's best interest to make me trim my video. They can simply hold the video until I've trimmed it and then process it. With their current system they have to process a full length video with garbage on both ends and a copy of that video I just uploaded that's significantly different.
Remember, this is the company that won't bother converting your video to vp9 if it has less than 1 million views. If they're that cost conscientious when it comes to encoding then the only way they'd double encode videos is if there's some huge internal politics involved.
I have to laugh at the possible idea of some passive aggressive developers at Youtube pissing at each other over code repos. When you do trim a video and save a copy, that copy is encoded as 720p max. I definitely uploaded a 1080p video. You just know there's some team at Google that gets bonuses based on total number of bytes uploaded, but the team in charge of encodings is probably penalized for costs, so when they get told to double encode they throw in a subtle middle finger of "720p that garbage."
Normally this probably isn't a big deal, but when the stream has text 720p really only works if you can control the encoder very closely. When I do 720p I'm setting very specific settings in ffmpeg to make sure that I max out the quality at that size and bitrate, plus tunings that work well for text and simple graphics. My talking head may get pixelated randomly, but the text usually comes out very clear with my settings.
Given all this, to use Youtube live streaming I'd have to do something like this:
The problem is, I would lose all the viewers on the live streams, and that means Youtube would probably knock my videos down in the rankings. Imagine if I stream to 1000 people, well I'd want to keep that video. Like Spiffing Brit showed, if you edge the live stream you get tons of viewers waiting, and then Youtube thinks that video is popular. If I then delete that live stream then I lose all that Juice. If I then upload it again Youtube will most likely ignore it.
The other solution is to not live stream at all, but instead to do a "premiere" which is just dumb. The whole point is to give people live education so they can interact with me...live...because it's live...a premiere is not live.
A minor annoyance is that Youtube's setup is far more difficult than Twitch's configuration, but I actually understand why Youtube is doing this. With Twitch you set a few options that most people leave alone, and then stream. If you set it up right you may never need to change your Twitch configuration and can just stream right away.
Youtube's configuration involves setting everything for every stream. You have to declare the language, whether it's for kids, tags, descriptions, etc. and then you can stream. I know why they're doing this, since they have gone completely insane trying to exploit children for ad revenue they are very interested in these settings. They have to make sure you never say a bad word in the first 8 minutes of a video (while they show you clips from Nazis and cam girls simulating sex) so that checks out.
Twitch has a very good phone application that lets you easily moderate your chat (to a limit of course). I use it to view chat messages and if someone's spamming or being an ass I just click their name once, and say block. I can also edit the stream's settings and a few other things that are handy.
Youtube's app...is Youtube. There's no phone app I could find, and the "Creator's Studio" is only for the video side of with no live stream support. There's no chat you can view, no way to edit your stream, nothing. Not sure how a trillion dollar company can't figure this out, but if Youtube can't make this kind of app then there's no hope for any of us.
So now I have to use the Youtube chat, which is reasonable but moderation is still far too difficult. Twitch seems to have gone through more growing pains when it comes to moderation so it seems to be better at it, and there's a lot of services you can employ to make it easier. With Youtube I struggled to figure it out before giving up. I'm sure moderation is there, but it didn't seem as smooth as Twitch's features.
One thing I want to work on is integration with Youtube, so I went to check out the docs and they're totally wrong. First, let's repeat the docs here:
After this it's all garbage because you don't need Oauth2, and you can't even get to that from this mysterious "Go to credentials." I had to search for "Credentials" in the search bar, browse through different screens until finally I could make an API key. The docs are completely wrong and I'm not sure how they don't know this, but it gets worse.
The API demo for JavaScript was written in 2018 but reads like it's from 1999. It uses var
and function
, but more importantly it uses the above stupid Oauth2 configuration that's entirely unecessary for someone getting started. Oauth2 is infinitely more complex than a simple API key, and most platforms have moved away from Oauth2 unless it involves getting permission from a user. This demo doesn't need any of that, and it ends up confusing the issue. Here's my version that works and is modern:
let fs = require('fs');
let {google} = require('googleapis');
const getChannel = (key) => {
let service = google.youtube('v3');
service.channels.list({
key,
part: 'snippet,contentDetails,statistics',
forUsername: 'GoogleDevelopers'
}, (err, response) => {
if (err) {
console.log('The API returned an error: ' + err);
} else {
console.debug(JSON.stringify(response.data, null, 4));
}
});
}
let api_key = fs.readFileSync("api_key.txt").toString();
getChannel(api_key);
This is all you need once you get an API key that you put into api_key.txt
. I have no idea why
they haven't updated the example in 5 years, but that was kind of the last straw.
I hate to keep pulling the rug on people, but I think the most optimal way to do this is to stream to Twitch where it's easier for everyone to view and easier for me to stream, then upload a nicely edited video to Youtube that people on Youtube can watch. You should subscribe to my Twitch at @zedashaw if you want to watch live, and if you're only on Youtube then you can watch offline and use comments to ask questions.
]]>Whenever you see two factions warring in the technology world the answer is almost always: "Both." The endless debates about SPA vs. MPA are very much this kind of situation, but because everyone is deeply invested in one or another technology their arguments are mostly unrealistic. This blog post is an attempt to drip a tiny amount of realism into the debate by giving a kind of oral history of web development in the 90s to explain the power structures in place today.
To understand the subtext of the arguments between MPA and SPA fans you have to understand the history of web development leading up to the invention of SPAs. Once you know what came before you can begin to understand why some people hate SPAs, and it's not any of the reasons they typically give.
My course currently teaches Svelte, which is solidly an SPA. I am planning on adding Alpine.js because I also teach classic Model-View-Controller MPAs, and I need a "bridge" from MPAs to SPAs. Alpine.js seems like the perfect in-between technology being a framework for simple reactive UIs, but not trying to be an entire SPA system like Svelte.
I view my job as an educator to be one that prepares people for surviving through all these obnoxious arguments technology people constantly have. The most capable student is one who knows how to make both kinds of applications, so that's what I teach. I'm mentioning all of this just in case someone thinks that because I have Svelte in my course that I'm somehow biased against MPAs.
I'm not biased against MPAs, but through a lot of research for this course I've found Svelte to an easier starting technology for beginners. I'll cover why this is later in the post, but let's just say I only need to make 1 file to get started with Svelte.
I was there. I was writing web apps using CGI taking credit card payments shortly after the internet was deregulated in the 90’s. The irony was before you got paid for web apps you had to get your clients a T1 line and set it up and also set up their network and mail servers! The knowledge was still pretty arcane. I’m about to have a trauma flashback now lol.
At the dawn of the web every computer sucked. Servers sucked. Desktops sucked. There were no phones with web browsers. It was all junk desktops with tiny CRTs over slow AOL 56k internet talking to fairly slow servers with web servers that would tank if more than 100 people connected at once.
In addition to this JavaScript didn't exist, and then when it finally did it suuuuuucked. It would wallow in a hell of terrible design for decades before being taken seriously, but what made working with JavaScript so bad was the complete lack of developer tools. I still remember in horror having to use alert("line 1")
just to figure out where my code was bombing. It wouldn't really be until 2006 when Firebug came out before we had decent tools for dealing with JavaScript.
You would make your design in PS, cut it all up and align it as background images in tables. Lots of WYSIWYG HTML code all over the place, rendering differently in different browsers. I remember making invisible text of really small sizes to get smaller row/cell heights.
In the late 90s the idea of getting a Compaq Presario to render a JavaScript front end was absolutely hilarious. Computers then could barely render HTML correctly, and when CSS arrived it wasn't cross platform at all. You couldn't style an <h1>
tag in Netscape with CSS, and since Netscape was king we all had to use <div class="h1">
instead. IE got padding "wrong", but weirdly, right, and still lost. It was a mess.
Can you imagine having to get JavaScript to work across all the competing browsers, and perform well on all the junk computers? Even more laughable is the idea of a programmer actually wanting to write all the JavaScript necessary to do this before Firebug.
I remember doing a website using java or flash or something to take an image, and make it look like it was a video with waves. That was the penultima of my web career. One thing I remember is how user un-friendly everything was, especially documentation. Setting up a working web cart in those days was magic.
The situation was so bad inside browsers that there was a market for software that ran inside the browser just to get a decent GUI. Flash and Java Applets were the two biggest examples, and Microsoft had Active-X. NPAPI was Netscape's answer to…Java Applets? Active-X? I actually have no idea what NPAPI did. Then you had all the editors that tried to help like DreamWeaver and Front Page.
At a big software company at the time, making a very complex B2B site for the power generation market. We used one-of-a-kind stack, with Delphi ISAPI/NSAPI, Netscape web and certificate servers, NPAPI plugins, JavaScript, smartcards and Oracle with SPs. Also C++ and VB.
In this environment we needed the server to render all of the HTML for the little wimpy computers. This way we can optimize it for their tiny AOL connections, and all they have to do is render one time (after a 5 minute loading time). We didn't use JavaScript because it sucked, computers sucked, internet sucked, everything sucked.
We would receive the requests from the user--every single click went to the server--to process their UI events, talk to the databases for them, take the data and push it through raw nasty string templates, and then pass that back to the client. If you think about that really hard, this is ridiculous. Doing the rendering for the browser?! But, as you saw from people's quotes, it really was necessary to avoid all the difficulties inherent in the browsers. Every other week some new company had some new thing that was going to "solve" the web, and some Pointy Haired Boss would walk in and slap their box of software--yes, a real physical box with CD ROMs in it--on your desk telling you to use it because they paid for it.
It didn't feel awful back then at all though. And that's actually the interesting thing about this looking back. For me it just was the way it was and I tried to get along with what I had and knew.
The ridiculousness of the situation is that every user interaction involved usually pointless network traffic to the server. Every tiny little change to the UI required this, but it wasn't a bad idea back then. If someone is using a Compaq Presario over an AOL connection then this is what you want because asking their computer to change a button's color might set their house on fire.
Even more ridiculous is how tightly coupled all of this was. If you wanted to change the color of a button you almost always had to completely redeploy the entire server. That's because the server generated all of the HTML, so if you change the HTML then you change the server, and that might break everything.
Dreamweaver and its Version Control System. Yeah the locking files type of version control, yeah integrated directly with production. Yeah and the manager could unlock files and screw your work up, and PHP of course, and Apache... and PHPMyAdmin (MySQL and friends)
Running everything in the servers gave the programmers who worked on those servers a lot of control, and many people resented this control. Every change you made required begging a group of developers with total control over the code. You want to change a button? Gotta talk to the Database Admin, the System Admin, the Server Admin, the EJB Manager, the Java Developer, the Template Developer, and then finally the Designer got their button change.
There was a lot of job security and power in being the person in charge of the servers, because the entire business went through the servers. Not many designers could grasp all this code to make their own changes, and even if they could they wouldn't be able to deploy the change without going through these gatekeepers.
Keep in mind, this was also entirely necessary at the time. If browsers sucked, every computer sucked, JavaScript sucked, and the internet sucked, then you needed dedicated people who kept it all working in the backend.
Even if Quirks Mode and doctype switching were a thing, I feel as if learning web development was easier in the late 90s. There were less parts to learn, they were simpler and learning them was done incrementally. At least, compared to the modern webpack/react/js stack.
Fast forward 27 years--yes, twenty seven years--and the situation is wildly different. Computers fit in your pocket and are more powerful then 1000s of Compaq Presarios. Many people have internet that's 100,000 times faster than AOL. Browsers are all very compatible and conforming, with great developer tools compared to the 90s. JavaScript is also vastly improved with the arrival of ES6 using real computer science rather than weekend hacks. We even have WASM so we can completely avoid JavaScript and use Rust! Maybe.
Users also expect their UIs to react in incidental asynchronous ways, not because of direct interaction. If they scroll to the bottom of a bunch of videos on TikTok they aren't going to click "next page", they expect the next videos to just load. If they make a mistake in a form they don't expect to wait for their browser to talk to the server, they expect immediate feedback. If they chat with someone, they don't expect the page to reload every time they send a message.
Today the user base on the internet is also far larger than it was in the 1990s or even early 2000s. Because of this far larger internet population we also have far higher costs on the servers. More traffic means more CPU spent making HTML for the browser that could be spent calculating the best ad to show them. Meanwhile, the browser is far better at rending HTML than a server ever will be. It's literally the browsers' primary job.
In this current environment the 1990s style of making servers spend CPU time rendering HTML for browsers seems even more ridiculous, but is the MPA entirely unnecessary now? It may be ridiculous, but there are very valid and specific reason to render the HTML on the server even today, which I'll get into later. Remember, if you're practical about this, you'll realize that it's ridiculous to do work for the browser, but there are definitely situations where that's exactly what you want.
I should also mention that many of the people advocating for MPAs also work at companies that love to track your every click. With an SPA the network traffic that you can track is reduced, which most likely cuts into the bottom line of the click tracking industry.
The reason I'm talking about how terrible the web was in the past, and how it's not like that anymore, is to bring up the real motivations for the arguments on both sides. The vast majority of programmers working in web development today are either from the 90s limitations, or directly descend from that era without realizing it. Every framework until maybe 2016 assumed that it was still the 90s, and they were designed to do everything in the server.
The emergence of SPAs threatens to undermine this control. With an SPA you can actually change the entire UI without doing anything to the backend. I frequently rework whole UIs without ever redeploying my little JSON server. If people are able to work on the whole UI and deploy it without needing the people working on the backend then the power dynamic has changed dramatically.
This is why you see people who work on Ruby on Rails, Django, Pheonix, and other classic MVC web frameworks criticize SPAs with FUD. That's also why they produce their own answers to SPAs that still require redeploying a server on every change to the UI. They're desperate to maintain the control they have over the entire stack.
The people pushing SPAs however seem to be motivated by capturing control of the UI since that's the thing which is viewed as most valuable by the people in charge. To the people who run businesses, the UI is what makes them money, so the people who control the UI are the most important people in your company.
Keep in mind I'm not making any ethical judgement on this behavior, just pointing out how I see the real motivations playing out in the drama of MPA vs SPA. If you want to actually use these technologies it helps to understand these motivations so you can take what these people say with a large grain of NaCL.
The real issues that matter in the decision of MPA vs SPA comes down to Control vs. Interactivity. It's not performance. It's not FOMO. It's not bloat. It's not accessibility. Those issues are important but they exist in all software.
It's whether the UI on your site's interaction needs to be controlled vs how interactive it is. These two needs compete with each other because to control the page you have to move the rendering into the server where you can actually lock it down, but doing that removes your ability to make it highly interactive because you're rending it remotely. If the page needs to be highly interactive you have to do the rending in the browser, but that removes your server as a point of control.
The "second computer logout" issue is a good example of this control vs. interactivity dynamic. The SPA transfers the control of the UI to the browser, so when someone logs out on a different computer most SPA apps have no idea it happened. They continue to operate but the user is logged out at the server, so it takes a few attempted requests before the SPA realizes the situation. With an MPA this isn't a problem, because the very next action will entirely go through the server, and the server will simply return the login page instead of the next information.
A similar issue from the other side is form validation. You always need to validate form input at the server, but if you want to provide instant feedback to the user you can't do that. Validating the form with an HTTP request on every keypress is stupid when you have a perfectly functioning advanced virtual machine on the user's computer. It's better to provide feedback validation to the user in an SPA, but then final validation in the server where the real control needs to be.
After all that, here are my recommendations when an MPA is going to be a better choice than an SPA:
Login, payment, and logout pages are all good examples of things that should be static HTML with no JavaScript (even though I don't even do that). Other examples is if you can't afford to have someone access information that's already on the browser after you've timed out the user. Another example is if you must immediately log the user out no matter where they are, or eject them right away.
In all of these situations the key component is you have to transmit every user interaction to the server so that the server can decide if the user is allowed to do that. This means if they left their session open and the server logged them out, then the next interaction will catch that and boot them to the login.
If all someone does is visit one page and read it for about an hour, then an SPA is pointless. Same with videos. If they just land on the page and watch a video then an SPA is stupid. This blog is static files for this very reason, because you'll come here and spend some time reading, then probably get back to your other interests. Making your computer download my entire website just to give you one article is stupid in this situation.
If the content has no access restrictions, and it doesn't change much, and the user doesn't interact with it, then you really should make it static. There's no reason to have that generated by a slow Ruby on Rails stack.
However, if the content does change, and you have to restrict access, but the user doesn't really interact with it much while they're using it, then an MPA will be better.
If the target audience has very weak computing power or internet, then you'll be better off with an MPA and finely tuned low bandwidth HTML. Effectively this situation is much like the 90s when everyone's computers were terrible and we had to do their work for them. An SPA in this situation is incredibly dumb unless you can somehow preload it onto the phone and make it persistent.
For an SPA the criteria is mostly reversed:
If the interactions the user makes are do not need to be controlled--just the data they need for each one--then having an MPA hand craft HTML for the browser is stupid. The browser is a far superior HTML renderer than any server side programming language will ever be, so stop making servers do it. If there's no reason to control every link and button click then simply have the browser do the rendering and the server send the data.
Notice I say "time spent interacting" and not "time spent on the site." This is a key distinction I think people don't make. If I'm reading a blog and I want to explore everything this person wrote, I might make 1 click of a link every hour. This is not a long time spent interacting with that blog. A site like Facebook, TikTok, Twitter, or an application used by data entry workers are examples of things where the person has to interact with the UI frequently for long periods of time.
Another example is if someone is going to be on the site and watch many long form videos. In the previous section I said an MPA is better if someone watches one video, but if they're going to spend 4 hours watching 10 videos then an SPA wins. Why is that? Because modern video requires absolute mountains of JavaScript to work smoothly, and making someone load all of that code on every browser click is stupid. There's also the issue that a lot of video components load very slowly and look sloppy unless you have the JavaScript already loaded and ready to go. In this situation, an SPA will be the better solution.
Just like with the interaction time spent on the site, if the user also has to interact very frequently with the site then an SPA is superior. If they will sit there watching a video for 30 minutes and then maybe click one link, then either an MPA or SPA is fine. But if they're clicking buttons for a solid 30 minutes, then the SPA is better. A good example of this is a dating site where people get on, swipe for about 30 minutes very fast, then get off.
After all of this I do want to say one thing about SPAs vs. MPAs: SPAs are far easier to use when developing the UI, but only because MPAs don't bother making this easy. When I develop a new UI with Svelte I can create the entire UI, all interactions, the whole design, and mock out the server with one file. I get started by generating that file in bandolier, add it to a routing file, and then done. No need for models, view, controllers, forms, views that are actually controllers, migrations, nothing but one .svelte
file.
As I work I can then slowly add the controller, then when the controller is working I can add the models I need, and then I can bring in some queues if I need. The point is I can grow my stack in consistent logical steps from one file to a working UI, instead of dealing with the entire stack right away.
By contrast, whenever I start working on a new UI in most MVC frameworks I have to lay down anywhere from 4-8 files that all need touching and configuring before I can even get a "Hello World" to show up in the browser. I'm forced to interact with every single piece of the framework even if I don't need to right now, and that's a huge burden when I just want to start with the UI.
In addition to this, when I make changes to the UI with Svelte I only need to run a build that regenerates static files and push it to the server. I don't need to go through huge redeployment scenarios that reboot 20 services just to change the color of a button. I just push the new UI, and there's zero down time because I'm not taking anything down. If I change the JSON server then I only redeploy the JSON server. If I change the queue server then I only redeploy the queue server.
There's no reason a classic MVC framework can't also do this. There's no reason Phoenix needs me to layout 8 files just to get "Hello World". I think if MPA frameworks want to compete with SPAs they need to start making it possible to start with just one file and then progressively grow the rest of the required features. They also need to make changes to the UI not require complete redeployments of the server.
That's primarily why I had to create The Bandolier the way I did. It has MPA elements, static file rendering, but primarily I can use Svelte as the primary teaching method because it is easier to get started and learn each piece of the stack in order, rather than all of the stack all at once.
]]>npm init
and Get Off npmjs.com
One of the largest blind spots for programmers today is their dependence on singular platforms run by giant companies. They have all of their code on github.com, put all of their projects on npmjs.com, and brag about these sites being their "resume" of their accomplishments. They obsess over the stars and likes and downloads per week. Then they're shocked when one day it all goes away, or when Microsoft exploits their kindness to sell their code without attribution.
Don't believe me? Here's a post by someone who was blocked for "star farming" but they weren't the culprit, they were the victim. What happened is they signed up for a 3rd party site named NopeCha, and that site abused the victim's account to add a fake star to NopeCha's projects. Github then banned this person rather than banning NopeCha's accounts.
Imagine if this person had their entire professional career on github? It was their "resume" and one simple mistake, and done. They have nothing. Oddly, programmers claim this never happens despite numerous instances of companies doing this and multiple employees being caught and arrested for exactly this kind of fraud. It's entirely possible for some rogue employee to flag your account and your packages on npmjs.com to get banned. One accidental mistaken identity and all of your hard work is gone. One poorly written moderation bot and you're homeless.
A lot of what I want to do going forward is to be as independent as I possibly can. I don't want to feed any more of my code into Microsoft's Copyright avoidance technology. Thankfully, it's not too difficult to host all of your own code and npm
supports almost enough features to make this seamless. You just have to work around npm init
, and I'll tell you how.
If you want to host your own git repository it's very easy. I use gitea on a simple VPS and it works very well. It used to have a fixed list of licenses, and it used to make you pick only the licenses gitea approved, but now you can easily add your own. I simply added a CopyrightAllRightsReserved "license" and done. It's viewable by anyone just like any of my blog posts, but nobody owns it...just like my blog posts (more on that in another post). A few things you can do with gitea:
LANDING_PAGE = explore
and as long as your public organization is the only one then it'll show all of that organization's projects.Once you have your own git
service running, and you're able to assert your rights as a copyright holder, then npm
will work for most of it's operations with a few modifications.
The next problem is getting npm
to install your code from your git repository. If you want to let people install your software then npm install
supports installing from git URLs but you need a special syntax:
npm install git+HTTPS_URL.git
For example, if you want to install the code I'm discussing later in this post you do this:
npm install git+https://git.learnjsthehardway.com/learn-javascript-the-hard-way/ljsthw-bandolier.git
When someone does that it'll then show up in their npm list
in what I feel is a more informative format:
zedshaw@ /Users/zedshaw
├── commander@9.4.1
├── csv@6.2.4
├── http-server@14.1.1
├── ljsthw-bandolier@0.3.1 (git+https://git.learnjsthehardway.com/learn-javascript-the-hard-way/ljsthw-bandolier.git#HEXCODE)
├── prompts@2.4.2
└── readline-sync@1.4.10
This shows you exactly where that module comes from--which incidentally would reduces the problem of typo-squatting on npmjs.com. Updating even works with npm upgrade
as it will install a new version of the module from the original git URL, and if the version number changes then it will update like normal.
Everything is looking good except...
npm init
ProblemIt's great to be able to install modules but what really helps is if people can quickly install example projects using your modules. In my course I want people to run a few simple commands to get demo code, course exercise code, and be able to start projects quickly. The npm init
command seemed like the winner, but it turns out be very problematic in how it's implemented.
First, the documentation for npm init is really bad. It clearly describes everything without actually showing you how to make an npm init
project for other people. Instead of clear instructions you get this mapping from npm
to npx
:
npm init foo
->npm exec create-foo
npm init @usr/foo
->npm exec @usr/create-foo
npm init @usr
->npm exec @usr/create
npm init @usr@2.0.0
->npm exec @usr/create@2.0.0
npm init @usr/foo@2.0.0
->npm exec @usr/create-foo@2.0.0
Sooooo, npm init
just maps to npm exec
in 5 different ways? Wait, so isn't npm exec
just the npx
command? So we're now at 2 levels of indirection? No, it gets worse because you then have to create a create-foo
package in the npmjs.com repository. Now 3 levels of indirection but we're not done yet, oh no, architecture astronauts are never done with indirection.
This create-foo
package that's required to be on npmjs.com (that's run by npm init
(that's just running npm exec
(that's really just the long form of npx
))) has an additional level of indirection:
initializer in this case is an npm package named create-<initializer>, which will be installed by npm-exec and then have its main bin executed
It's.."main bin"? Is that the main:
or the bin:
key of the package.json
? There's two, so "main bin" means nothing. Turns out you need to create one bin:
entry and also set main:
to that entry in order for npm init
(I mean npm exec
(I mean npx
)) to run the right command.
The convolution in this is astounding for something that's basically just running some code out of a module. In theory this should be nothing more than a reference to a module npm install
can use, and a command to run in that module. None of this triple quadruple obfuscated nested routing to 5 different commands is necessary.
npm init
Always Queries the RegistryWhile trying to figure out how to configure package.json
I ran into the final blocker: Every time I tried to use npm init
the npm
command tries to find the package in npmjs.com, no matter how I installed it. I can't test the command if I always need npmjs.com, and that also defeats the entire purpose of this whole exercise. If there is a way to use npm init
then it's far too difficult to document for my users given I can't figure it out.
The documentation for npm init
is very vague on when it's supposed to look in the npmjs.com registry:
Note: if a user already has the create-<initializer> package globally installed, that will be what npm init uses.
Nope, this is a lie. I tried packages in all kinds of configurations, globally, locally, in package.json
files, everything I could think of and npm init
always tried to find the package in the registry. That means you can't simply publish straight out of your personal git repository even though npm install
can install directly out of a git.
If this documentation is not technically false--and I missed exactly how you make this line of documentation work--then the documentation fails to explain exactly what the condition is for running npm init
without talking to npmjs.com. That's what makes this documentation so bad. It's all written as if it's just tiny notes reminding someone who wrote npm init
how their own code works, not an explanation for other people who want to use it.
npm init
?The purpose of a command like npm init
is to get people started quickly with a new project. Many projects require a lot of boilerplate setup that can be automated, so these "template builder" commands save everyone time. They also help people avoid mistakes in configuration because you're not accidentally manually copying errors into your project.
To create an alternative to npm init
we don't really need much:
npm install
using our git repository.npx
. It'll run any command in the package.json
file's bin:
section.That's it, and that's effectively what npm init
does, just in an insanely convoluted way. Once I realized this it was easy to create a project that did installs for the course. People use it like this:
npm install git+https://git.learnjsthehardway.com/learn-javascript-the-hard-way/ljsthw-bandolier.git
npx bando-up create my-first-project
The first command installs my little ljsthw-bandolier tool. This tool has the following in the package.json
:
"main": "bando.js",
"bin": {
"bando-up": "bando.js"
},
The bando-up
command simply runs the bando.js
script which is a command runner implementing different commands I'll provide users. The first command is a create.js
command that knows how to check out the code for my "educational web framework", configure it, and help the user get started. It does the following:
--depth 1
to keep it small.I'm using the bando.js
command runner since I expect to add more commands to support the course. If you only want to let people install your software then just replace this bando.js
with your own installer script.
No matter what you do Microsoft gets its pound of flesh out of your hard work. Yes, you can run your own registry. Yes, you can simplify that by using a simple git repository. Yes, you can make your own alternative to npm init
to work around their registry demands. Seems all good right?
When you run npx
it tries to find a module that implements the command you want to run. If you run npx bando-up
it should search through your installed modules for that command, and then run the correct one. What happens when two modules list the same command? How does npx
figure out which one you mean?
We can use npx bando-up --using ljsthw-bandolier attack-test
to actually test this. I run that command, then I modify the package.json
and give it a name that starts with a
:
npx bando-up --using ljsthw-bandolier attack-test
vim package.json # change the name to attack-test
cd ..
npm install ./attack-test
npx bando-up --help
So which one is run? I have two projects with bando-up
as commands, and I have no idea which one npx
just ran. Let's modify the attack-test
package so we can tell:
cd attack-test
vim package.json # change the version to 9.9.9
npm remove attack-test
cd ..
npm install ./attack-test
npx bando-up version
0.4.2
npm remove ljsthw-bandolier
npx bando-up version
9.9.9
As you can see, even though I have two packages implementing bando-up
npx seems to choose "randomly"? It's not alphabetic, otherwise we'd get bando-up version
with 9.9.9. After I remove the ljsthw-bandolier
project I get the expected outcome.
There isn't any documentation I could find on how npx
decides which command to run. It's entirely possible for someone to exploit this with the following attack:
bando-up
.npx
runs their version of bando-up
instead of mine.That's a fairly small attack surface, but still totally possible. The only mitigation is to tell people to always include the ljsthw-bandolier
in the npx command:
npx --package=ljsthw-bandolier bando-up version
0.4.2
npx --package=attack-test bando-up version
0.4.2
npm remove ljsthw-bandolier
npx --package attack-test bando-up version
9.9.9
What the hell? The --package
option doesn't even work?! I tried --package=attack-test
, -p attack-test
nearly everything mentioned in the npx documentation and not a single option worked.
I guess we're just screwed, and this is the best we can do with what we're given.
If I could give out a wish list for the trillionaire Microsoft that's failing to run npm
correctly then it would be these:
npm init
to die in a pool of its own vomit.npm create
into a new command that is universal and has zero dependence on any registry.npm create
would accept only direct URLs for installer projects to install. These URLs are anything compatible with npm install
.npm create
will use a totally separate create.json
file with exactly what is necessary to make this feature work, rather than infecting package.json
with convoluted options.create.json
would specify the .js
file to use as the create script, and additional options for things like, removing the git, template patterns, and post install operations.npm create
wouldn't install the project, only use it to make everything work, that way it won't infect the system with additional commands.npx
command should prevent duplicate commands, and if two projects have the same command require the -p
option and refuse to run.No, I won't submit a pull request. I don't give trillionaires free labor.
]]>I just finished integrating Stripe into this site and it was a massive pain. I honestly do not understand the belief that Stripe is better than Paypal. Everyone who begged me for an "alternative to Paypal" would recommend Stripe but my experience so far, and that of other people, is that Stripe is not better than Paypal. They're both moderately terrible. This review is an attempt to make sense of why Stripe's documentation doesn't work, why it's the same for Paypal, and how Stripe's only real advantage now is mostly smoke and mirrors. Hopefully this helps people who are running straight to Stripe avoid a painful lesson.
I used Stripe about 10 years ago and stopped because Stripe kept charging me for Chargebacks even though I offered to refund. I not only lost the original money--which is perfectly fine and what I wanted anyway--but was also forced to pay an additional $15 with no option but to contest the chargeback. I quickly ditched Stripe for Paypal after a string of false charges and paying back hundreds of dollars in fees for chargebacks I didn't want to contest.
My reason for ditching Stripe was both to avoid these attempts to force $15 out of me, but also because Stripe clearly didn't care about fraud prevention. I did a quick analysis of the fraudulent charges and realized that if Stripe only compared the IP address GeoIP to Zip codes they could block 90% of them. Meanwhile, Paypal's fraud prevention is so strong I handled maybe 2 or 3 fraudulent charges a year and Paypal would let me offer a refund rather then contest the charge. Stripe's answer at the time was to tell me to use GeoIP to identify fraud...while they controlled the customer's Zip code...so I could...not do that?
I switched to Paypal and it had many advantages and disadvantages, but overall I didn't have to deal with fraud, and they offered really great loan terms that helped smooth out the volatility in my kind of business. Paypal had a lot of advantages and disadvantages over Stripe at the time:
Paypal these days simply just works, but it does have an equal number of problems when compared to Stripe:
It's this last disadvantage that prompted me to revisit my relationship with Stripe and make sure I can use it as a payment method.
I started getting weird emails from potential customers saying they can't submit a credit card for purchase. I couldn't figure out what was going on until a customer sent me a video showing an attempted purchase, and Paypal's UI just sitting there doing nothing. They looked in the Console for me and there was an error about the address being wrong, but it looked correct. After watching their video probably 20 times I suddenly saw what was going on:
It was so bizarre I couldn't imagine it was a simple mistake. I could see maybe they didn't apply the error condition to the address field on accident, but what made me feel this was secretly done on purpose is the following:
ppxo_inline_guest_unhandled_error
error messages in the console indicating the street address was needed, so they clearly received the error.I wanted to cause this error so I can find a way to test for it, and I finally managed to cause it by using one of Paypal's test card numbers in the Live Environment. When I enter one of these fake cards the Paypal form reports the same error and then shows the address field, which is weird because these card numbers are entirely fake. There should be a solid error message stating invalid card, not, "Hey buddy, what's your address for that fake card that's clearly fake?"
All of this combined tells me that Paypal is using some kind of passive aggressive fraud prevention scheme that involves hellbanning bad cards and getting them to continually enter in a street address. I have no idea why on earth they thought that was a good strategy but it's their money. At least...until it became my money.
This poor UI error design, silently adding form fields, and using that for some weird hellban fraud mechanism, was my motivation for adding Stripe back to my site. After 10 years I thought maybe Stripe had improved and I wouldn't deal with their idiotic fraud system...or lack of a fraud system. I logged back into my account and started going through the docs, and immediately thought, "This is an awful lot like Paypal now."
The more I used Stripe's API the more it reminded me of Paypal's API around 2005-2010. It has the same complexity, multiple redirects, convoluted data responses, and almost everything that Stripe touted as terrible about Paypal in 2010. It isn't terrible, and isn't great, it's just...Paypal.
As I implemented all of the required Stripe integrations I found they did some things really well:
Sounds great right? Despite these nice positives there were quite a few serious negatives:
PaymentIntents
system reminds me of Paypal before Stripe started to compete with Paypal. Stripe's big selling point was simple integration with just a little JavaScript, and now I'm writing multiple backend handlers, front end integrations, handling huge numbers of possible events, juggling multiple rotating IDs for every event, and required handling of webhooks.payment_intent.failed
for example. The description is just "Occurs when a PaymentIntent has failed the attempt to create a payment method or a payment." Holy Tautology Batman! Alright so what do I do? Does Stripe not show users this common failure and expect me to show them an error? Now I have to coordinate an asynchronous webhook to my backend and show an error to the browser? That's nuts.One response I got to these complaints is the classic canard of, "It's for SECURITAY!" I don't buy it. Plenty of other payment processors--including Paypal--charge credit cards without all this insane complexity and they don't have rampant security issues. In fact, making something more complex is how you reduce security, not improve it. They also claim all of this work prevents double spends but I still don't buy it. Given the randomly rotating IDs for nearly every event and the disorganization of the JSON data I suspect this is more to do with improving Stripe's backend performance than anything else they claim.
Then again, the documentation is so fragmented that there may be some clear explanation of everything hidden somewhere. If that's the case then why isn't this just a first course so I know this clear explanation. If it's hidden in the middle of a random paragraph then, no, it is not "documented." It's "mentioned."
I think most people use Stripe's client only checkout but Stripe has this huge warning at the top, and most of what I've read online says they're eventually phasing this out:
Client-only Checkout does not support many of the features available with a client & server integration and Payment Links, which may better fit your use case.
This "client only" option also still does redirects to complete transactions, so...I'm not sure why this is called "client only" at all. You still have to deal with webhooks, and that means you're dealing with multiple IDs for every event. It really isn't buying you much, and smacks of Stripe phasing out the option by coating it with "syntactic vinegar."
What I suspect is Stripe wants people to actually gravitate toward Payment Links so they can start capturing user accounts just like Paypal, get in on that "no code" craze, or increase lock-in with merchants.
After I started the integration a couple students and several people on Twitter messaged me to warn me about Stripe freezing their accounts. I remember one reason for Stripe's success is their marketing that they didn't freeze accounts like Paypal. For those of you too young to know what's going on, Paypal used to pull this definitely illegal scam:
It's the 4th part of the scam that got Paypal sued for fraud and racketeering, for violating Unfair Business Practices Law, and investigated by the FTC. I'm really not sure how they got away with it for so long. In my mind this money is either the payee's or the payer's, and not Paypal's at all. There is no contract or click-wrap EULA that invalidates the law regarding theft, and keeping money that's not yours is clearly theft. In the event of fraud Paypal should have either given the payee the money and closed the account, or refunded all of the payments to the payers then closed the account. Keeping the money indefinitely with no recourse is the literal definition of fraud and theft.
Eventually this caught up to Paypal because companies like Stripe used this theft scam as a main marketing pitch to new customers. WePay even pulled a prank dropping 600lbs of ice in front of Paypal's conference in 2010. The promise from companies like Stripe and WePay was they wouldn't freeze your accounts, and they were easier to use.
Now Stripe is doing everything Paypal was doing in 2010. Not only are the Stripe APIs just as complex as Paypal 2010, but Stripe also arbitrarily freezes accounts and steals money just like Paypal. One of my Students told me this story (used with permission):
Honestly f$ck stripe, I sell digital games and physical copies and 4 days into opening my account and making $600 in sales, they asked for all my ID which I provided, then froze my account until I took photos of my inventory, which I did and linked an ebay account with them all listed and submitted it. 2 Days later they told me my account was deemed high risk and they shut it down, and will be holding my $600 in sales until the end of Feb 2023.
This isn't the only report of Stripe doing this, and just to be clear:
Based on these reports I'm convinced that Stripe is now as evil as Paypal was circa 2010, and that fact makes me very cautious about adding Stripe to my site.
If Stripe is so risky compared to Paypal why bother using it? Stripe has one massive but entirely arbitrary advantage over Paypal:
Stripe only tries to charge cards. Paypal tries to coopt my users to boost their user count.
Originally Stripe's big selling point was their superior API usability and documentation. Their original startup name was even "/dev/payments" with a direction of focusing on developer usability to attract people to their platform. Paypal has done a lot to catch up on that front, and Stripe has done a lot to get worse for developers, which means they're both pleasantly mediocre. Both Stripe and Paypal are about the same in terms of API complexity, thorough but disorganized documentation, and a lack of clear guided curriculum.
Where Stripe beats Paypal is in the UI presented to the users, and I'm not talking about arbitrary design elements like font and line style. I'm talking about Paypal shows you two buttons to choose "Paypal" or "Credit Card", followed by this massive form:
Meanwhile, Stripe only shows the bare minimum needed to charge a card:
It is stupid, but Paypal's insistence on trying to capture new users means they add two additional points of friction and tout to all of my users that Paypal is involved. This drives users away who either have issues with Paypal or are simply frustrated by the process and obvious demands for their data.
Stripe however just wants to charge that card and collect that fee. In the screenshot you wouldn't even know Stripe was involved. I decided to add text stating I used Stripe to give people confidence I'm not taking their information. Without it, a customer would have to inspect the console to see it was Stripe, and most won't bother doing that. Stripe also does the bare minimum necessary to charge the card while Paypal is trying to collect everything possible.
This one small difference in strategy drives a significant advantage for Stripe. Paypal's obsession with User Acquisition is ironically giving their competitors a completely arbitrary advantage, and based on how much money their competitors make I can't see this User Acquisition focus actually being more lucrative.
If Paypal had a new UI that looked exactly like Stripe's I believe they could turn the tables on Stripe's marketing. I'd call the product "Paypal Thin" and simply position it as:
Paypal Thin is simplified credit card only processing with no mention of Paypal or any attempts to get your users to sign up. With Paypal Thin, nobody would even know you're using Paypal.
Paypal could even start a marketing campaign that makes fun of Stripe's claims of being more secure, being easier to use, and their freezing of accounts. Paypal could double up this marketing by showing how they don't need to freeze accounts anymore because their fraud prevention is so good they don't even need to charge you extra for it (unlike Stripe).
Will they ever do this? Hell no. Paypal comes from the old school startup world where you write everything in CamelCaseJava that's impossible to change and you base all of your value on users not money. The fact that they can't even put a red box around an address field that's wrong tells me they couldn't make this UI if their life depended on it. Not to mention this would be a huge shift in philosophy for the company that would probably require firing 20% of the C-suite and most of the VPs.
There's simply no way they'd stop trying to collect signups, but if you think about it logically it is literally the only advantage Stripe has over them. Stripe freezes accounts too. Stipe's API is more complicated than Paypal's now. Stripe and Paypal's documentation is equally thorough but disorganized. Paypal's fees are even lower once you factor in the extra fees to get mediocre fraud protection from Stripe Radar.
Stripe's simply showing users a better UI because they have a slightly different idea about what constitutes success.
I believe one way that all of these companies fail is in protecting users from malicious internal employees. Everyone operates under this delusion that employees working at companies like Stripe and Paypal don't access your account and information. The reality is most companies give employees completely unrestricted access to any user's information, and many low level employees are easily bribed.
Take the Onlyfans and Facebook lawsuit as a recent example (of many). Allegedly OnlyFans partnered with Facebook/Instagram to get competing porn actors flagged as "terrorists" in the Global Internet Forum to Counter Terrorism (GIFCT). The lawsuit claims that 35,000 Instagram accounts were erroneously listed, which effectively blocks that content across multiple companies, not just Facebook and Instagram. Either Facebook and OnlyFans did conspire together, or OnlyFans just bribed some employees at Instagram or the GIFCT to flag the content.
You also have the recent reports from Twitter Whistleblower Peiter "Mudge" Zatko that claimed Twitter had zero security preventing employees from accessing any user's accounts:
2011 FTC Complaint: In 2011, the FTC had filed a complaint against Twitter for its failure to properly protect nonpublic consumer information, which included users” email addresses, Internet Protocol (“IP”) addresses, telephone numbers, and nonpublic information exchanged on the platform.” The complaint alleged that, from 2006 to 2009, far too many Twitter employees exercised administrative (“God mode”) control within Twitter's intemal systems and user data, thereby allowing anyattacker with access to an employee account to easily compromise Twitter systems. And Twitter's systems were, and are, full of highly sensitive personal user data that enable a hostile government to find precise geo-location(s) for a specific user or group, and target them for arrest or violence.
This was exploited by two former Twitter employees who used their access to hack people's accounts for the Kingdom of Saudi Arabia and its Royal Family. Then there was the hack by 17 year old kids that immediately gave them total control of every account by simply phishing a few low level employees.
These are easily not the only incidents, and technology companies have been sued repeatedly for user privacy invasions by employees. In fact, I'd wager that 99% of startups have zero protection against rogue employees until they're sued by the FTC.
Why do I bring this up? Because it's entirely possible that some employee at Stripe or Paypal will read this and angrily click two buttons in some screen to disable my account. There's no way for me to know this--since we don't have laws requiring disclosure of employee access--but if my accounts suddenly have problems I'm positive this is the case. Criticizing these giant powerful companies comes with the risk that a random rogue employee will exercise their powers to block my access.
This is why I advocate to people to be ready to switch payment processing at a moment's notice. Right now I could flip two options and I'd be back on Paypal. If I was pressed I could probably implement any other payment processor in about a week or a day. The ability to quickly switch payments is your only defense against frozen accounts, fraud, and rogue employees.
]]>I constantly forget all of the random steps necessary to make equal height columns so I'm writing this down and sharing it. Most of the information out there on how exactly to create the "holy grail" layout is fairly lacking in explaining why each setting works, or is too old and doesn't use flexbox. A lot of the existing documentation also glosses over everything that's actually needed, and just says "flexbox! yay!" This blog post attempts to explain everything used to make this style of layout, but if you find there could be improvements then please let me know.
To create the "holy grail" layout you need to solve three things:
With flexbox this becomes easier, but it is still more complex than usually explained. The actual requirements are:
display: flex
.<header>
and <footer>
set to a var(--height-footer)
and var(--height-header)
variables to fix the height.<main>
(grandparent) tag set to flex-direction: column
and a height calculation of calc(100vh - var(--height-header) - var(--height-footer))
.<main><columns>
(parent) tag set to flex-direction: row
and flex: 1 1 auto
.<main><columns><left>
and <right>
(child) set to flex: 1 1 auto
with display: block
optional if you don't want its contents to flex.:root{}
variables --height-header
and --height-footer
to whatever you want for them.<body>
to have margin: 0px
and padding: 0px
so everything is full screen.This will give you a starter "holy grail" layout that you can then adjust and alter depending on what you need.
The "Holy Grail" comes from the period right after everyone on the internet decided that <table>
tags shall be henceforth banned by Edict of Zeldman. The layout is this:
Seems simple right? You have a header, usually with a navigation element, content with a left and right side (usually of different widths), and then a footer with a ton of links in it. The problem comes when you aren't allowed to use a <table>
and have to use only <div>
tags. You're then stuck using crazy CSS tricks that barely work, and ultimately fail when you switch to mobile.
Modern CSS lets you use flexbox
and grid
to create the holy grail layout more easily, and in this article I'll show you how to use flexbox
to do this. I'll use grid
in another blog post to compare the advantages/disadvantages. This description will also explain why this works so you (and I) can hopefully remember this in the future.
Keep in mind this is a starter and not meant to be your entire layout done for you. It simply solves the key problem of getting a header, footer, main content, and two columns on the screen and filling it in a reliable way. You'll still have to do work if you want it to look different, but starting with this will get you farther than trying to figure it all out on your own.
The simplest HTML headers to get started are:
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset='utf-8'>
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta content='text/html; charset=UTF-8' http-equiv='Content-Type' />
<title>Holy Grail Starter</title>
<style>
/* CSS goes here. When you see me describe CSS you put it here. */
</style>
</head>
<!-- body here -->
</html>
Your next step is to create the first body, header, and footer tags:
<body>
<header>
<h1>Header</h1>
</header>
<!-- replace this with the next step -->
<footer>
<h1>Footer</h1>
</footer>
</body>
This needs to be modified with the following CSS:
:root {
--height-header: 80px;
--height-footer: 200px;
}
body {
margin: 0px;
padding: 0px;
}
footer {
display: flex;
background-color: hsl(0, 0%, 80%);
min-height: var(--height-footer);
}
header {
display: flex;
background-color: hsl(0, 0%, 80%);
height: var(--height-header);
}
In my CSS here I'm setting the <header>
and <footer>
to have background-color: hsl(0, 0%, 80%);
which makes them gray. Change this later when you figure out your design. You should also note the use of :root{}
variables --height-header:
and --height-footer
as they help with the height calculations later.
Once you have your HTML up and running you can create the <main>
and <columns>
tags that are sandwiched between the <header>
and <footer>
:
<main>
<columns>
<!-- next step left and right go here -->
</columns>
</main>
Now comes the "trick" which is to set <main>
to use column
orientation, and then set <columns>
to row
orientation. Confused? You should be because column
and row
are almost entirely arbitrary, but I'll try to explain why this is going to work after you see the CSS:
main {
display: flex;
flex-direction: column;
min-height: calc(100vh - var(--height-header) - var(--height-footer));
}
main > columns {
display: flex;
flex-direction: row;
flex: 1 1 auto;
}
To understand what's going on you need to know how flexbox
orients things (which also explains why "row" and "column" is so confusing):
flex-direction
determines something called the "main axis" and "cross axis" (or dimension), which is the direction that elements are "stacked" or arranged.flex-direction: column
means to set the main axis to go vertically, or "along a column" which is up and down. This stacks elements like a stack of books.flex-direction: row
set the main axis along the row, or horizontally. This stacks elements like beads on an abacus.flex-direction
to go horizontally, then say "hoROWzontally" and you'll know to use row
. If you want it to go vertically then say "vertiCOLUMNly" and you'll know to use column
.column
stacks vertically (vertiCOLUMNly) like a normal HTML page, and row
stacks horizontally (hoROWzontally) like text in this sentence.flex-direction
, so if you choose column
(vertiCOLUMNly) then the cross axis is row
(hoROWzontally).flex: 1 1 auto
property applies a "grow shrink basis" modifier to this child using the parent's flex-direction. I'm constantly tripped up and think that flex:
says how I want children to be sized, but it goes on the child and determines how it will grow. You'll find whether a property is for a child or a parent is rarely mentioned in the flexbox specification.flex: 1 1 auto
to main > columns {
we're saying, "Expand <columns>
to fill the <main>
parent's main axis (vertiCOLUMNly)."main {
rule is set to flex-direction: column
our <columns>
tag will now stretch to the bottom of main, and because it automatically stretches along the cross axis (horizontally) this makes it stretch both ways.That's a complex breakdown, but the simplest explanation is that you need make the <columns>
element to stretch across both the vertical and horizontal dimensions (axis) of <main>
. Since flexbox will do this across one dimension by default, you only need to set <columns>
to flex: 1 1 auto
to make it also stretch across the other dimension.
flex
property is part of flexbox it doesn't need to be in a block set to display:flex
. I'm not sure if this is standard, but it did work in display:block
definitions so it must only require a parent set to display:flex
. This makes sense since there's certain points when you don't want to use flexbox, but you still want to size that element.
min-height
MathIn the previous section I had this seemingly complicated and potentially dubious math:
main {
min-height: calc(100vh - var(--height-header) - var(--height-footer));
}
This uses a fixed height <header>
and <footer>
to create a center <main>
that fills the remaining space. It does this by:
min-height
requires that main not shrink below this height.calc
is a function for calculating math.var
references the variables --height-header
and --height-footer
we set in :root{
above.100vh
means "100% of the viewport height" so if your browser window is 500px tall then 100vh
becomes 500px.min-height
of main
to 100% of the browser window viewport minus the <header>
and <footer>
height.Does this mean that if the contents are greater than this height you'll get the classic CSS "explode out of box" problem? Not that I've seen, but then again CSS will surprise you and one day, for some random reason, your content will explode out or weirdly overlap. This setting seems to work as follows:
<columns>
to fill that vertical space.If you see this acting differently please let me know and hopefully someone can explain why exactly this works or doesn't work.
When a block has a fixed height you'll see strange "exploding out" of the contents. Normally this is obvious since it happens inside blocks where you've set a min/max height to a specific value, but sometimes it's deceptively hidden. Add this to the main {
CSS block:
max-height: 300px;
After you refresh this should look normal, depending on your screen height. It's subtle, but shrink the height of your window and you should see the text of <footer>
float above the left column. If you scroll down you'll see that actually the entire <footer>
block is sliding under the main content as you shrink it down. Weirdly, this also happens long before you reach the 300px
setting, and actually happens when the contents of <left>
are reached.
Even though everyone says that "exploding out" of the box only happens when you set the size of that box, it seems that the browser will punish you for setting any size on anything. We set a min-height:
on main, so now the browser takes liberty to completely screw up your layout in an unexpected way by not exploding the contents of <left>
out. No, it...scrolls <footer>
under <main>
?
Another way to put this is, according to the "rules", setting a min-height:
on <main>
should cause the contents of <left>
to explode out. It's entirely evil and unexpected to have <footer>
slide under <main>
instead, but I'm sure some CSS aficionado will have a detailed explanation that still doesn't forgive this behavior.
Inside <columns>
you'll have <left>
and <right>
children that contain your actual content. Here's an abbreviated sample as an example:
<left>
<!-- put any amount of content here -->
</left>
<right>
<!-- try less content here to test different heights -->
</right>
If you stop here and refresh the file you'll see that these two columns fill the vertical space, but they don't stretch across the horizontal space. We can break down why this happens like this:
<columns>
is set to flex-direction: row
so its main axis is hoROWzontal leaving its cross axis as vertical.<left>
and <right>
will stretch, and hoROWzontally (main) it will compress because of flex-direction: row
.To fix this we have to apply the same trick from <columns>
and add flex: 1 1 auto
to the <left>
and <right>
elements:
The CSS for this part is:
main > columns left,
main > columns right {
display: block;
flex: 1 1 auto;
}
Notice here that I'm adding display: block
to show that you don't need these elements to be display: flex
for the flex: 1 1 auto
setting to work. This lets you stop using flexbox but still get the flexbox layout technology. If you were to put another block inside <left>
or <right>
then you would want to set <left>
or <right>
to display: flex
so you can apply these same techniques again. You'll see this in the later section Two Tone Columns.
flex: 1 1 auto
is a combination of flex-grow
, flex-shrink
, and flex-basis
. These are too complicated for one blog post, so just remember the magic incantation of flex: 1 1 auto
when you want something to "fill the space." If you want one element to be smaller than the other, then either change "auto" to be a fixed size, or try changing the "1 1" ratios until it's what you want.
I also add some color to the left and right so I can see it for debugging to make sure they are equal height.
main > columns left {
background-color: hsl(0, 0%, 100%);
}
main > columns right {
background-color: hsl(0, 0%, 60%);
}
When you're done you should have something like this:
Try messing with the width, height, and contents of each block. You should try to break this in different ways and then find solutions. I give a few in the next sections.
The current layout is doing an equal sized left and right, which isn't usually what people want. I started with this because it's easiest to get working first. If you want the <left>
side to be smaller than the <right>
side remove this rule:
main > columns left,
main > columns right {
display: block;
flex: 1 1 auto;
}
And change these two:
main > columns left {
display: block;
flex: 0 1 300px;
background-color: hsl(0, 0%, 100%);
}
main > columns right {
display: block;
flex: 3 1 100%;
background-color: hsl(0, 0%, 60%);
}
This adds the flex:
property to both so you can specify exactly how they should size hoROWzontally (see <columns>
). The flex: 0 1 300px
property says to not grow <left>
and give it a base size of 300px (width). The flex: 3 1 100%
on <right>
says to make the right side grow 3x and default basis of 100%.
This will mostly work, but it'll depend on what you put in the left side, and you might need to use the classic width:
property instead. If everything works it should look like this:
Another common style is to have the <columns>
split in half with two "tones" or colors so the screen is divided exactly in half, but to have the contents inside a different width. You can pull this off by first adding your inner content:
<left>
<info>
<h1>Left Info</h1>
</info>
</left>
<right>
<info>
<h1>Right Info</h1>
</info>
</right>
Next we change the main > columns left
and main > columns right
rules to implement row-reverse
on the left, and row
on the right:
main > columns left {
display: flex;
flex-direction: row-reverse;
}
main > columns right {
display: flex;
flex-direction: row;
background-color: hsl(0, 0%, 60%);
}
Here's how this CSS works:
flex-direction: row
stacks hoROWzontally then row-reverse
simply reverses the order of blocks.row-reverse
on <left>
will cause that side's <info>
block to push to the right. That then makes the two <info>
elements sit next to each other in the center.background-color: hsl(0, 0%, 60%)
on the <right>
tag so it has a background that covers the entire right side of the screen.Once that's working we can give the two inside <info>
elements a width using flex: 0 1 400px
:
main > columns left info {
flex: 0 1 400px;
border: 1px solid black;
}
main > columns right info {
flex: 0 1 400px;
border: 1px solid black;
}
This sets the <info>
elements to have a base size of 400px
, but they will shrink as needed (up to the content size...hopefully). The flex: 0 1 400px
says, 0 growth, 1 shrink, 400px base width.
I also have a border: 1px solid black
on these <info>
blocks so you can see them, but remove these if you use this. The end result of these changes is this:
Supporting mobile with the first layout requires only changing the direction of the <columns>
:
@media only screen and (max-width: 700px) {
main > columns {
flex-direction: column;
}
}
This says when the screen width dips below 700px
the browser should apply the rules inside { }
. In this code I'm simply changing the flex-direction:
from row
to column
which makes the <left>
and <right>
elements stack vertiCOLUMNly and that fits on a mobile screen.
If you wanted the <right>
element to be on top then you would use column-reverse
to reverse the new stack. Either way this is how is should end up:
You can view all of the code in the site blog git where you can view the following files:
basic.html
html raw -- The first layout created, no mobile fix.twotone.html
html raw -- The "Two Tone" left and right side demo.mobile.html
html raw -- The simple first step for a mobile responsive layout.sidebar.html
html raw -- The demo with a small left sidebar.all.html
html raw -- All of the features in one file: left sidebar, two tones, and simple mobile.