----------------------------------------
       PicoGopher Part 4: the more we are, 
                          the funnier it is.
       November 14, 2022
      ----------------------------------------
       Written on my laptop, offline, during
       a flight and a train trip
      ----------------------------------------
  
  
  Welcome to Part 4 of the PicoGopher journey! If you want to
  see how everything started, you can find the previous parts
  in my phlog at gopher://gopher.club/1/users/mala/ (also
  mirrored at gopher://gopher.3564020356.org). You can find the
  project's code at https://github.com/aittalam/PicoGopher.
  
  In the previous posts I described the steps I followed to 
  create PicoGopher from scratch. Thanks to the large amount of
  available code and documentation, it was not too hard to
  crank up some simple implementation of the Gopher protocol
  and allow anyone to serve their own contents from a RasPi
  Pico W. While this is an interesting hack, though, I think
  that requiring people to (1) have a Gopher client installed
  and (2) know your server's IP address is, perhaps, a bit of
  a stretch. And for PicoGopher to be actually useful and used,
  I think it should be accessible to a wider audience.
  
  For this reason, today's post is dedicated to extending
  PicoGopher so it can provide contents via (a very simplified
  version of) an HTTP server and automatically redirect users' 
  requests to the AP's IP address. All of these options can be
  turned on or off at will, allowing for different level of
  access - from the most esoteric one (protected WiFi, Gopher
  only, non-default IP address) to the most open (open WiFi,
  HTTP access, automatic redirection to server IP).


  =========  Project status =========
  
  Below you can see the current status of the project. The
  steps marked as "x" have been completed, while those marked
  as "+" are described in this post. You will first read about
  going async (required as soon as we start running more than
  one server listening on different ports), then a few more
  details about the new HTTP and DNS servers.
  
  
    - [x] connect to the WiFi
    - [x] run a simple HTTP server
    - [x] run a mockup Gopher server
    - [x] load/save files
    - [x] make the Gopher server not a mockup anymore:
      - [x] translate gophermaps following gopher protocol
      - [x] load any file from disk
    
    - [x] set up the pico as an access point for geolocalised
          access
  
    - [+] make the server a bit more accessible
      - [+] enable async
      - [+] enable HTTP
      - [+] captive portal with PicoDNS
  
    - [ ] powering PicoGopher
      - [ ] better understand power saving
      - [ ] playing with batteries
  
  
  =========== Going Async ===========
  
  The translation of our server code to async becomes necessary
  when we want to simultaneously run more than one server on the 
  Pico (e.g. both Gopher and HTTP). This allows our servers to
  listen to requests on different ports and answer them without
  blocking each other.
  
  The datasheet "Connecting to the Internet with Raspberry Pi
  Pico W" [1] shows an example of an (HTTP) async server. The
  main difference wrt the non-async version is that socket
  binding, listening, and accepting are all dealt with by the
  `uasyncio` micropython lib [2], and our main() code will boil
  down to starting a new server, defined by a callback method,
  and host ip + port to listen to.
  
  The callback function is passed reader and writer streams by
  default, and we can use them to communicate with clients
  connecting to our servers. This does not change the way our
  Gopher server works in a dramatic way, but for the sake of
  readability I moved all its code to a PicoGopher class and
  created a similar one for HTTP (PicoHTTP). For DNS instead, 
  I just copied/pasted the code from P Doyle's Micropython 
  DNSServer Captive Portal project [4]. Yeah that's quite ugly,
  but I was too eager to see that working! I hope you can
  understand what I mean... And if you don't, let me cite Larry
  Wall: "Laziness, Impatience, Hybris: the three virtues of the
  programmer" :-)
  
  
  =========== HTTP Server =========== 
  
  Our HTTP server is a bit of a Frankenstein's creature, but we
  love it anyway :-) It is because it's a patchwork of code
  coming from the datasheets (for the async part), from the
  Gopher listener (to parse gopherfiles) and, erm, stack
  overflow (if I remember well!) to deal with URL decoding.
  We love it because it is another great example of the 80:20
  rule, and it shows us that you can get a reasonably working
  website running on a Pico, without the need to write one line
  of HTML (well, just because I did it for you... but look at
  the source code and you will convene with me that's a really
  small amount anyway).
  
  How can we get an HTML-free website? Given our gopherhole is
  (at least for now) made only of gopherfiles and plain text
  files, all we have to do is convert a gopherfile into an HTML
  page which links to the other files. The picohttp.py file
  shows how I did it - it is still very simple and limited, but
  I think it can be expanded rather easily to support more 
  advanced features. If you look for `gophermap` in the code,
  you will find the logic behind this: at the moment I do 
  nothing more than copying plain text rows as they are, and 
  translating 2-column links into HTML <a href...> links.
  Text formatting is managed by some improvised CSS that works 
  decently on Chrome&co but screws everything on Lynx, which
  means I have already added it to the list of things that
  need to be fixed (well, yeah, the more the 80% grows in 
  features, the more the 20% of things to fix grows too...)
  
  The other main difference when compared to Gopher is that the
  HTTP server gets multiline requests (with a set of headers
  for each request, that we currently just ignore), prepends
  resource descriptors with a GET (because GET requests are the
  only ones we accept), and encodes URLs by converting many
  characters into their hex equivalent (if you ever saw a URL
  containing a `%20` whenever you had originally typed a space,
  that's URL encoding for you). This last part is taken care of 
  by the `urldecode()` function (yeah that's the one I copied
  from stack overflow). 
  
  
  === DNS server + Captive Portal ===
  
  The concept of captive portal is one I had no clear idea
  about until just a few weeks ago, when I stopped after work
  to have a drink with some colleagues and friends (back when
  it was possible to invite guests at Twitter, but also back
  when I still had a job at Twitter). After the second pint 
  I came out with the question "how can I pop up a gopherhole 
  on someone's screen the same way those login pages appear 
  when you connect to some WiFis?". 
  
  A colleague (thanks Matt!) introduced me to the fabulous
  world of captive portals and once I knew the right terms to
  search (as in +fravia's arrows [3]) I was able to find their
  Micropython implementation. The one in [4] is the simplest
  one I found - and decided to use for a quick and dirty
  experiment with PicoGopher. 
  
  The captive portal in PicoGopher is implemented as (yet)
  another server, a DNS, that basically replies to every domain
  name resolution request with the same IP address, that is the
  one of your Pico (which is 192.168.4.1 by default). When a
  Mac or an iPhone detects this, it will also try to open a
  `hotspot-detect.html` page on the captive portal, so I
  updated the HTTP server to redirect the request to the root
  of our gopherhole.
  
  There are still a few caveats to this captive portal. First
  of all, this is just one of different ways of implementing it
  and I am not sure whether this is the best one yet. Then, I
  realised that when *all* DNS requests from a laptop or a
  phone are redirected to a Pico then the device might have a
  hard time replying to all of them (and even in the best case 
  when nothing breaks, it might still take way more power). 
  Finally, I found it hard to understand how the response 
  payload was built using the code and comments alone. If you 
  are feeling the same, I found [4] a bit more detailed and 
  with better references to DNS documentation and RFC [5,6]. 
  
  
  =========== Conclusions =========== 
  
  The code in the latest commit (4d085f4) implements all the
  changes described in this post. In `main.py` there is now a
  set of variables that you can customize not just to set up
  your AP but also to enable / disable both the HTTP and the 
  DNS server.
  
  Once again I think there's still some work to do here, but I
  hope this is enough to get started with some interesting new
  functionalities.
  
  
  ============ References =========== 

[1] https://datasheets.raspberrypi.com/picow/connecting-to-the-internet-with-pico-w.pdf
[2] https://docs.micropython.org/en/latest/library/uasyncio.html
[3] https://fravia.2113.ch/targets.htm
[4] https://github.com/p-doyle/Micropython-DNSServer-Captive-Portal
[5] https://github.com/jczic/MicroDNSSrv/blob/master/microDNSSrv.py
[6] https://www.iana.org/assignments/dns-parameters/dns-parameters.xhtml
[7] https://www.rfc-editor.org/rfc/rfc1035.html