Week 8, 9 and Te..: WebTech for The Spatial Scientist.

From Week 8, work and other pressing issues broke my week-on-week streak! Necessary learnings where gotten nonetheless. With a planned more intense study schedule, I will make use of “work-family-study-time balance” lessons learnt when the studies assume.

My Website
A Modal Pop-up - JS/ CSS

Content in Week 8 was not too foreign. Being the network administrator of my fibre to the home network, lessons on TCP/IP was familiar ground. I knew enough of how traffic traverses the web - my home network is customised with a PiHole and has network wide DNS filtering.

Having setup my blogging site in 2015 from scratch and through several iterations, writing HTML code felt at home. I have in my portfolio, some web-mapping projects which leverage HTML. The twist came when it was time to delve deeper in Java Script. Earlier learning through the course drove home the fact that the course is really about learning to learn new languages more quickly and easily.

In the years past I have fiddled with the LAMP Stack. Experiencing Flask in Week 9 was a breath of fresh air. Quickly seeing the results of one’s development efforts. Early on, it was not easy untangling the interlinked parts (documents) of a Flask app. But once understood, it makes sense as the power of templating and frameworks is exposed. It becomes rather convenient.

My Flask App
A Flask App. Running

Meeting The Objective

One of the objectives of this blog series was to:

  • Have a fresh and solid base for when starting the PGDip studies

As the deadline for application for the PG Diploma drew close. I pushed to get across the completion line of CS50. There just wasn’t enough time to finish Week 10, The Project. I successfully completed Week 0 - 9 with the required assignment passmarks.

I used the tasks transcript to ‘motivate’ for my ‘Computer Science Course equivalent to a University First year course. It turned out I didn’t need to take this course! 20 years ago I did Computer Science for Engineers for my BSc Studies and that was sufficient. Nothing though replaces the 6 months of refreshing my knowledge, un-learning, re-learning and cementing current knowings though, before 31 October 2025.

With time, I must still do the Final Project to get the Harvard Certificate. ..before then, let me drop the morphed on-wall study schedule.

CS50 Personal Schedule
Personal CS50 Schedule

Week 7: Reflecting on CS50’s SQL as a Spatial Scientist.

As opportunity would have it, this week I was on a work sponsored esri training course, ArcPro Advanced. One of the topics covered was Simple SQL Queries. CS50’s SQL Topic became an opportunity to re-learn SQL, dive deeper and master the proper crafting of queries.

The first part turned out to be a great easy-going refresher. As a spatial scientist I ask spatial data questions all the time. As geohipsters, we actually have a cute file format, geopackage which is essentially a sqlite database. In the course of the material. I couldn’t help reminisce PostGIS and SpatialSQL. I have a decent exposure to SQL but, further on with Week 7 content, queries became complex and my learning began to compound.

I noticed a personal preferance when it came to constructing queries, favouring

JOIN ... ON ... ---Inner Joins  

versus

SELECT ... FROM ( SELECT ... ---Subqueries/ nested SELECT statements  

Which was an interesting introversion of how I understood things. At the end of it, it was great mastering the alternative.

The brief discussion of SQL Injection attacks during lectue was very intriguing. A peek into cybersecurity.

Learnings for the geo-person

  1. Geospatial software leverage relational databases ubiquitously. As such, the topic of databases is one that cannot be side stepped in the profession. It is one that can actually be divergent, as some projects demand tools which emphasise data storage versus a spatial view. Few years back I was intrigued by a command line spatial data viewer which connected to PostGRES with a PostGIS extension. Not long DBeaver, a universal databae tool, debuted with a spatial viewer.
    Something that was domain specific to GIS.

  2. A great geospatial scientist should know above average SQL. Increasingly, it is apparent that spatial is just another column in the database. But the gis tech, has the advantage of spatial thinking.

  3. After completing this section, I can now do nested and more complex queries. Behind every icon in the spatial manipulation software GUI. The ready to use Intersect icon, is actually a

         SELECT spatial_features FROM spatial_table_A  
             JOIN spatial_table_B  
             ON spatially_congruent_features
    
  4. My greatest gain from this section was learning to work with database informaton programmatically. This came through the Fiftyville task of Problem Set 7.

  5. In all honesty, I underestimated the depth of topic for this section. It was a great break from general purpose programming and I learnt much much more on SQL. I now appreciate better, the responsibilities of a DBA. The prospects of the average geospatial scientist expanding into webmapping and scripting become greater.

That’s Week 7, now looking forward to HTML, CSS and JavaScript. Again topics not too foreign but another opportunity to re-learn stuff. As a parting note…Little bobby tables :laughing: .

`Robert'); DROP TABLE Students;--`?

Week 6: Learning a Programming Language, Again! From C to Python, Quickly.

As a spatial scientist, starting on Python was particularly interesting. The language is largely preferred for spatial data manipulation operations. At this point, I hadn’t delved Python to any great depth except encountering it in scripts at my 9 - 5 job, in esri software.

In this week’s Lecture, Shot and Section, Python is really dealt like rock skipping, touching the surface but, never sinking to depth. It was really about “How to Teach Yourself a New Programming Language”. Even with the Problem Set, the same was reiterated.

If I should gripe a little, every time the codespace had to be updated, I lost my theme - Cobalt 2. It however was not a pain getting things back to the way they were.

Codespace Rebuild
Codespace My Way - A Dark Mode

Learning a New Programming Language

It was not a struggle following highlights of differences between C and Python. What I had to contend with was ‘throwing away’ all the attention to detail and ‘pseudo-control’ over what the code was doing. Python is high level and I had to actively ignore the under the hood stuff I knew coming from C and ‘trust’ the computer to do the right thing. The compactness of Python code was a breath of fresh air!

Because with Python indentation matters, it sort of forced one to create clean code from the start. The CS50 coding environment screamed Problem, when a (tab) was missing.

After this brief run with Python, I found myself also asking the popular question Python or R. Because of the time I have invested in R for Data Science, I have developed some fondness for R. Curious about what the future holds.

Revisiting Code

I grappled briefly converting my for loops from C to Python (sentimental-mario-less). But because I understood my code, with a little experimentation, I eventually got the code to work.

In one of the problems, I found myself using Debug50 to trace variable changes. The problem brief from C had been twisted from dealing with integers to now dealing with floating point numbers in Python. It was exciting experiencing first hand float point precision! The computer was working with 0.009999999999999967 whereas my expectation was 0.01. As a consequence, my function returned 0.0. With some research I employed round(change,2) which solved the problem. A successful debug, always scratches the inquirer’s itch.

I also found a bug in my older code which I had inadvertently gotten away with. It was with a word counting function. Occurrence of ‘.’ or space indicated end of a word but hadn’t factored where they occurred together viz. (dot)(space). I therefore ended up with an inflated word count.

Coding fatigue is a real thing. Even after six weeks of coding, tiny errors are still possible and fundamentals of debugging still have to be observed. This was after my program just hung on execution because I had missed code to increment a counter.

Reflections

Again I learn that every problem at first glance looks insurmountable. The problem set on DNA, felt overwhelmed on first encounter. But by applying modular programming, sub-tasking and pseudo-coding, I was soon on my way disintegrating it piece by piece.

I am looking forward to SQL and having to take a break from general purpose programming languages. The week-to-week tempo I adopted in the face of limited time makes everything feel like a rat race, albeit exciting, in the face of a weekend deadline.

Let me put an end of post bar hereunder in the form of a progress bar, which surprisingly is blue from red at the moment.

Past the Halfway Mark
The Gradebook - Blue Zone Progress

Week 5: Hash, Hash. Thinking About Code-Collaboration and Nodes

Pogress Bar
The Gradebook - Looking at Progress

So after submission of Week 5 Problem Set solutions, the above is my current progress. Now just a little over the halfway mark of the course. I foresee of course that the Final Project (FP) will take longer than a week. The week’s lecture material was quite comprehensible, relaxing almost. Dr Malan has a way of explaining complicated concepts from basic building blocks, from the ground up. The video clip aids, such as the t-shirts from a box and those from hangers, really drove home the concepts of Stacks, FIFO and LIFO.

The previous week on memory, was a bit intense as the code became more complex. Had the motivation for studying not been strong, Week 4 was a good time to quit. But, as I have learned from Learning Hard Topics, some topics are challenging like that. With the content of Week 5, It’s clear now that was necessary foundation. The week is a culmination of C and will veer off to Python in Week 6 (looking forward to that). It feels truly like a peak of the course and I am expecting to now plateau.

From so much use of loops and defaulting to ‘i’ as a loop variable I concluded that the i stands for any of - integer, iterate, initialise, index.

For Loop
The ‘i’

Being Practical with C

Going over the Inheritance (Problem Set 5), one really appreciates how everything learnt up to this point - assignment (now of pointers), arrays, memory, loops, conditionals, truth checking and more, ties together. The utility of the C Programming language is demonstrated through implementation of a practical biology ‘problem’ - determining permutations of Blood Type.

I couldn’t help pondering though, the experience by someone encountering this computer science material for the ‘first time’, in my limited time circumstance would fare. A week-by-week tempo, is manageable for someone who can spare a few hours a day delving the content and not necessarily a full-time worker without a burn of midnight oil.

What I learnt About Complex Problems

Problem Set 5, Part 2, Speller was the most challenging, yet, of the course. It has so many ‘moving parts’ i.e. functions and also incorporated a lot, if not all, of the concepts learnt so far. I perceive the objectives I set out at the beginning of the course are gradually being met. Some take-aways:

  • I learnt (again) and got reinforcement that breaking a problem into sub-problems should be a default approach when tackling a complex task. For each of the 5 functions to be completed, I could independent apply the most appropriate logic to each.

  • Benchmarking is a fascinating way of improving one’s code and algorithms. I found the exercise of improving the efficiency of my already functional code exciting. The Ideas of good program design, I was experiencing in practice.

  • The many functions and and code files other people had written which I had to work with, made me understand better the concept of code collaboration.

  • Walking away from a challenge and coming back later seem to be working for me versus banging my head on the keyboard until I get it. There are instances though where once you get a ‘scent’ to cracking the problem, you can’t detach until you get it done, then only walking away with the delight of success.

Benchmarking Image
Benchmarking - How good is your implementation?

Week 4: Pointers, Memory and 16 Digits to Work With

Pointer Dog
German Shorthaired Pointer Dog

Well, this is not a post about a dog or dogs, but yes, it is about pointers. The nervousness with which I started Week 4 Memory, was justified. One of the instructors explicitly said that the topic of pointers was probably going to be the most challenging topic to be tackled in CS50. malloc() was lingering in my memory as I started this Week 4’s topic.

Prof. Malan brought literal training wheels to class which he threw to the ground to demonstrate things were being escalated i.e. opening the fire hose.

The highlight this week was learning about Hexadecimals (base-16). The shots video on Hexadecimal really brought me to understand the system versus the coverage in the main lecture. It brought the light-bulb moment when I saw the notation and a recall of some computer science literature I have come across , 0xADC et. al.

Learning Hard Topics

I ‘sat’ through the pointers lecture (shots) in order to get it! I was in no rush to get this topic over with, nor rush to the Problem Set to move along the course. I wanted to develop my ability/ resilience to learn difficult stuff. As Doug Lloyd pointed out

pointers take a lot of practice to master. But the benefit we get from their correct use, far outweighs the effort it takes to learn and master them.

Doug’s quick take really exposed the fundamentals of pointers and the topic sunk in. The main consolation was learning about their utility in File I/O. An area which should excite any enthusiast as it avails the programmer power to manipulate files!

Getting through Problem Set 4 was through small progressively victories, getting through one function at a time. Through it all, I got to understand nested loops and 2 dimensional arrays even better. There were a lot of those with this pset. Victory run, solving an upper limit case for the image manipulation routine, by investigating the output from the program. The correct program control and condition selection was the sticking point. More learning.

Sepia Trouble
Sepia Problems - Pixel Values > 255

Taking from the lecture, I had opportunity to investigate memory issues when my code was found faulty by the check50, using Valgrind

Memory Issues with Valgrind
Memory Issues

What I learnt About Pointers

  • Entirely within the content of CS50, (i’m certain anywhere else,)

    Pointers are just addresses . Addresses to memory where variables live.

  • Pointers allow us to pass data back and forth between functions. Overcoming the limitation of local variable scope.

  • A lot of doodling with arrows helps expose the foundational concept of what is happening when dealing with pointers.

  • A single pass of the topic might not be sufficient to get what pointers are all about. Several and alternative takes on the subject will most likely bring the point home. In the course there are three!