Posted on

Cloudless in the Cloud!

A few weeks ago everyone on earth learned about Map Box’s cloudless atlas, a cloud free mosaic dataset, generated from freely available Lansdat data.  It was all over the twitterverse and Wired magazine even wrote about it.

But, did you know Google came out with a cloudless imagery layer last week too?  They did!  Read about it here! Sometimes it’s good being the second one to the party, but I think only a couple people noticed that Google showed up at the cloudless atlas party.  Whoever does the marketing and PR at MapBox should get a bonus and a raise.  It’s not everyday you get more media traction than Google.

 

I think the clouds look nice
I think the clouds look nice

It is pretty well known that cloudless imagery is nothing new.  I remember working on projects in grad school several years ago on this same problem and researchers have been developing tools and processes to analyze cloudless imagery for years, as a quick search on Google Scholar will demonstrate.  Regardless, these are great datasets for visualization, and the fact that they are easily accessible to those making “beautiful maps”, not just GIS nerds, is excellent.

We need to give these organizations a lot of credit.  They took a phenomenal, freely available data source, added some intelligent and creative processing, developed some incredible datasets, and got the spatial and non-spatial world to notice. Kudos to them.

 

 

Posted on

Geography is important, no matter what the commenter says

I usually don’t respond to comments on my blog post, but I feel the need to respond to a couple items from a recent comment posted on everyone’s favorite article on this blog. Before I start my response let me say that I agree with some parts of the comment, which I will discuss later.  However, I really disagree with other passages and I’ll talk about those first.

But first, the comment…

From Chris, posted on 6/18/2013

“ESRI days are numbered. The place is a sinking ship. They have totally lost control of both the gis data and software monopoly they once had back in the 80s and 90s.

Open Street Maps and QGIS are hammering ESRI now. You can’t give away ArcGIS since EVERYTHING is free with other gis packages and data.
There are so many map apps and programs that BEAT ESRI at their own game. Newer and easier ones are popping up faster everyday now.

A person off the street can make a cool mash-up using QGIS and geoJSON.
If you are really map challenged there is:
http://www.theatlantic.com/technology/archive/2013/06/stamen-design-reveals-an-instagram-for-maps/276713/
http://mapstack.stamen.com/

And there is NO need for a degree in Geography to make maps anymore! This field of study is basically dead and has been bypassed in the last few years. Colleges need to stop teaching it since the public uses digital maps everyday now and are getting daily geography lessons for free!

ESRI better be shopping themselves around before Google adds massive gis tools to Google Earth and finishes them off. This multibillion dollar company will be worth nothing in a few years at the rate of free map tool and data advancement.
Sounds like not too many smart people are left at ESRI since most have left to go to start ups. ArcMap 10 basically validates this.

And the way Open Source is going, their won’t be to many companies left that will be able to demand thousands for their software. Especially bad software. ESRIs biggest customer, which is the U.S. Government, is slowly waking up to this fact. The Gov needs to save our tax dollars and go more to the free open source software ASAP.”

Now, my response:

The “open vs. commercial GIS” mentality is getting old.  I am getting sick of it, and you should be too.

We should all be vested in the development of GIS as a science, as a tool set, and as a way of thinking, whether it is commercial or free, open or closed.  GIS is aided by the growth of both free and open source and commercial software. It is pretty well known that the commercial sector now has real competition and they need to respond. This is a good thing.  Saying that open source GIS is going to kill commercial GIS software is like saying that Linux has killed Windows or PostgreSQL killed Oracle. These commercial GIS shops are embedded deep in many organizations and they aren’t going to be dropping them any time soon. That is a choice that they made, and when the business case dictates a change they will make it.

Now, will organizations who are new to GIS or at a point of transition choose to go with open source platforms that are light-weight, reliable, and free and easy to use when they design their next implementation? Probably. I would, especially if cost was a factor.

The more scientists, engineers, planners, civic leaders, decision makers, concerned citizens, business leaders and educators who use GIS the better it is for GIS community as whole. The GIS community should be working together to move technology forward – not digging trenches and setting in for battle.  It is a horribly counter productive strategy.  Any users of GIS, online mapping, or spatial analysis would logically want the field to grow and evolve. Competition helps drive that growth and evolution and I am all for it.

And, why do people keep saying/thinking that Google should buy Esri, or that Google Earth is going to overtake ArcMap? I’ve never understood this argument. Never. Why would Google, who has really failed at commercializing their current geo-stack, go after such a small market when compared to their other endeavors?  If you were a smart company that makes a lot of money what market would you focus on as revenue driver? Millions and millions of mobile users, or a few thousand specialized (and picky) GIS software users?  How many of you are paying for Google Maps or have bought Google Earth Pro? Not many? That’s what I thought.  Let’s just drop this train of thought.

Now for the second statement in the comment that drives me absolutely crazy:

“And there is NO need for a degree in Geography to make maps anymore! This field of study is basically dead and has been bypassed in the last few years. Colleges need to stop teaching it since the public uses digital maps everyday now and are getting daily geography lessons for free!“

This statement is so horribly misguided I don’t even know how to respond.

Let’s try. There is a calculator on every computer and smartphone made, and we use them all the time.  Does this mean we need to eliminate math as a discipline at the university level?  Anyone can download a content management system and build a website.  Time to get rid of computer science departments!  Turbo Tax! Get rid of accounting majors! WebMD. Who needs pre-med?

See where I am going with this? Just because a tool exists does not mean that a particular discipline should be eradicated.  You still need some background to understand what you are looking at.

I have three degrees in geography.  A bachelors, masters, and Ph.D. .  During those years of schooling and research I did far more than make maps and use online mapping tools.  Geographers study far more than just cartography and learn about place names, check out my dissertation for proof.

Geographers have been crucial in the development of the theory, logic, and science behind the G in GIS.  The field of geography has also provided countless contributions to spatial analysis, policy and planning, environmental science, economics, anthropology, sociology, biology, civil engineering, and many more and will continue to so.  In fact, we need more geography being taught at all education levels.

Geography matters. It always has and it always will.

To say that geography doesn’t matter displays a lack of understanding that is all to common in the GIS community.  Sometimes I am floored with the lack of understanding of the basic principles and fundamentals of geography among those who use GIS, online mapping tools, or any other type of spatial decision making system.  Without a basic foundation in geography how do you expect to make the correct decisions using a GEOGRAPHIC information system?  I’m not saying that everyone who uses GIS needs a Ph.D. in geography, but taking a couple geography courses during your undergraduate years isn’t going to hurt.

Now, what do I agree with from the comment?

  • Governments need to invest more resources in free and open source software of any type.  No excuses.
  • You don’t need a degree in geography to make a map, but it sure does help
  • OpenStreetMap is great, but let’s not forget where a majority of the US data came from in the early uploads (TIGER).
  • I love QGIS.

That’s enough for this post.  I didn’t do a god job summarizing my thoughts at the end, but it is getting late and I want to go to bed. More rambling and ranting will come in later posts

As always, thanks for reading.

Posted on

Spatial SQL – Multi-Point to Line Example

I have been using Spatial SQL for a while now.  I like it.  A few lines of code can do a lot of analysis or data processing.  I’ve covered a number of basic topics but there are always more to do.  Here is a script to take a series of points and convert them into a single line.   This script will use a few SQL commands, including using a cursor, developing a linestring, and STLineFromText.

The sample data comes from NOAA and the National Hurricane Center. The points represent some sample tropical cyclone forecast points for Hurricane Sandy.

The basic idea of the script is to link a set of points together using a common attribute using a cursor, combine the coordinate pairs into a line string, and use the  STLineFromText method to convert the coordinate pairs into a single line.  The script works pretty well, but since I am using a cursor it can slow down with larger (a few hundred thousand rows) datasets. A good SQL programmer probably wouldn’t use a cursor here since they can be slow and cumbersome to use.  In fact, I’m sure a good SQL programmer wouldn’t use a cursor.  I am investigating ways to not use a cursor, so if you have a suggestion let me know!

The sample script includes a sample dataset that is available here.  Take a look at the script.  If you have any suggestions to make this script faster or more efficient let me know.

/*#################################################

Script Name: multi_points_to_Line.sql
Purpose: This script will take pairs of coordinates
 of the same line and convert them into a LineString 
that can be used to generate lines of the geometry 
data type. User will need to update the database, 
tables, and column names relevant to their own analysis.

Sample Sandy data tracks represent five day
models from the National Hurricane Center.

Prepping the data - The user will need to download 
the following file and load into their SQL database:

http://www.gisdoctor.com/downloads/Line_Parts.txt

Here is a quick script to take the text file and load 
it into a table generated for this exercise:

create table Spatial_Database.dbo.Distinct_Points
([ADVISNUM] varchar(3), [lat] float, [lon] float, 
[MaxWind] int)

BULK INSERT Spatial_Database.dbo.Distinct_Points
FROM 'Path to Line_Parts.txt file'
WITH (FIELDTERMINATOR =',',FirstRow = 2);

###################################################*/
 
--Set the database to process in
use Spatial_Database
--Drop temporary table
drop table #Sandy_hur_tracks
--Create new temporary table
create table #Sandy_hur_tracks
([EventID] int, [line] geometry)

--Declare cursor variable
DECLARE @eventID varchar(10)
--Declare text string that will store coordinate pairs to
--populate the LineString
DECLARE @coordString VARCHAR(MAX)

--Initialize the cursor using the ADVISNUM column
DECLARE db_cursor CURSOR FOR
select distinct ADVISNUM
from Spatial_Database.dbo.Distinct_Points
order by ADVISNUM asc

OPEN db_cursor
FETCH NEXT FROM db_cursor INTO @eventID

WHILE @@FETCH_STATUS = 0
BEGIN

-- Clear the coordinate string with each iteration of the 
-- cursor - otherwise the coordinate string will 
-- append itself each time
set @coordString = ''
--collect all coordinate pairs and add them to a single row. 
-- Coordinate pairs are separated by a comma.
select @coordString = (COALESCE(@coordString + ', ', ' ') + 
(cast(Lon as varchar) +' ' + CAST(lat as varchar)))
FROM Spatial_Database.dbo.Distinct_Points
WHERE ADVISNUM = @EventID

--Insert the eventId and coordinate pairs into the table.         
--Coordinate pairs string is used to build the LineString to      
--create the line geometry
insert into #Sandy_hur_tracks
select @eventID as EventID,
Geometry::STLineFromText('Linestring 
(' + right(@coordString,LEN(@coordString)-1) + ')' 
, 4326) as line

FETCH NEXT FROM db_cursor INTO @eventID
END
--Close and delete the cursor
CLOSE db_cursor
DEALLOCATE db_cursor

--Select results from the temp table.
use Spatial_Database
select * from #Sandy_hur_tracks
Posted on

Ignite LocationTech Boston 2013 Wrap-Up

One of the great things about living in Boston is that there is a very active geo-community.  Every few weeks there is something interesting happening, whether it is an AvidGeo meet-up, a geo-colloquium at one of the many schools in Boston, or an industry sponsored event.

One of those events happened this past Monday at Space with a Soul in Boston’s Innovation District, organized by Avid Geo and the Eclipse Foundation’s LocationTech group.  The event focused on open source geo-based projects. The room was full of geo-thinkers from a variety of backgrounds, and thanks to a number of sponsors (AppGeo, Azavea, IBM, Actuate) there was plenty of food and beer!  Added bonus, participants from the PostGIS code sprint were in town!

The lighting talk format – five minutes, 20 slides, auto advancing – works really well for these types of events.  The speakers are energized and the crowd stays captivated.  I tried in vain to keep up with Twitter during the event.  Let’s take a look at my 140 character rundown of the evening.

Quick note – I missed a couple speaker’s names.  If you know them please post a comment so I can update accordingly. UPDATE – I only need one more name!

Second quick note- Ignore the grammar mistakes in my tweets.  I am a horrible with my thumbs.

Andrew Ross from LocationTech opened up the evening.  He talked about the mission of LocationTech and explained how the Eclipse Foundation helps open source projects get off the ground and stay relevant.

Max Uhlenhuth from SilviaTerra then gave a great overview how he started contributing to open source projects.  Max had a number of great points but a couple really stuck with me.  In his professional career he has only used open source software, and he was using FOSS in high school!  When I was in high school my parents had just gotten this new thing called the “internet.”  His best quote came about halfway through his talk:

Michael Evans and his colleague whose name escapes me (if you know, please let me know so I can update this post) talked about ongoing efforts in Boston’s City Hall to improve data sharing, analysis and visualization.  They talked about the extremely popular blizzard reporting site that famously crashed and how it was both a success and a failure.   It was a success because it was so popular, and it was a failure because it crashed, and crashed hard.  They also talked about efforts to make data resources work with more efficiency within City Hall.

I’ve seen Jeffrey Warren from the Public Laboratory give talks a few times over the past couple years and he and his colleagues are always doing something innovative.  He didn’t disappoint during his talk on Monday.  He talked about the Public Laboratory’s open source spectrometer.  It was pretty amazing.  I wish he had another 25 minutes to go into more detail.

Leaflet made an appearance at the event as well.  Calvin Metcalf, leaflet guru and cat enthusiast, gave a talk titled –Leaflet for some cats.  Using Max Ogden’s Javascript for Cats for inspiration, Calvin went through a few examples of how easy Leaflet is to use and customize.  If you haven’t tried Leaflet yet you should.  Calvin’s slides can be found here.

Christian Sparning from the Metropolitan Area Planning Council talked about the Hubway’s visualization hackathon.  If you are in the Boston area you have probably seen some of the results of this event over the past several months.  The Hubway folks released a whole bunch of data  – ride numbers, origin/destination data, temporal data – and then held an hackathon.  Christian talked about the variety of people who participated and the variety of creative ways they analyzed and visualized the data.

UPDATE:  Thanks to Andrew Ross I got an update on the evening’s last speaker (I didn’t catch his name at first).  The last talk was from Ken Walker from Eclipse, talking about the Orion Editor.  I’m not too familiar with Orion, which is a browser-based tool for developing on and for the web, but it looked like something I should learn about, soon.  I encourage you to check out the link to learn more.  Ken was kind enough to post his slides as well.

Overall, this was a great event.  Avid Geo and LocationTech did a great job putting this together and the speakers inspired those in the audience.  I really think you are seeing the future of geo at events like this.  Geo is no longer just for technicians working in municipal offices.  Geo is moving beyond GIS, web mapping, and cartography.  Geo is now everywhere and anywhere and that is a great thing.  In the near future geo will be even more ubiquitous throughout the business and technology worlds and there will be a growing demand for people trained in the geospatial sciences.   Events like this keep the field moving forward!

Posted on

Google Maps API v2 Deprecation and GISDoctor.com Mash-Ups

The old news on the street is that the Google Maps API v2 will be deprecated in just a few weeks in May 19th 2013.  I have a few old Google mash-ups that I never upgraded to v3 of the API that are floating around this site and I will retire those pages when the time comes.

However, one of my v2 mash-ups is a nifty transparency slider that I modified to work with WMS data.  I will be updating that site to v3 of the API before the last days of v2.  I’ve been saying that I was going to upgrade this app to v3 for a while now and I should probably do that soon.  Nothing like the deprecation of a API to get you motivated to write some code!

For more info on the v2 deprecation and a whole bunch of Google Maps news visit the Google Geo Developers Blog.

 

Posted on

Today’s Old School Geo-Reading

I was thinking about map projections earlier today and I wanted to reference a couple ideas against something other than what I could find through Google. I dug out my copy of the 5th edition of Elements of Cartography by Robinson, Sale, Morrison, and Muehrcke (1989) and I found my answer (when was the last time you used a book’s index?).

elementsofcartography

This book came into my possession during my undergrad years when I found it in a pile of books that a retiring geography professor was sending to the recycling bin.  I also have a 1st edition copy (acquired at the same time as the 5th edition), but that’s on my bookcase at work, next to my copy of David Harvey’s Explanation in Geography and Mark Monmonier’s How to Lie with Maps.

In today’s world of point and click cartography it’s nice to know that geo-visualization has always been deeply rooted in theory and science.  If you can find a copy of this book, at any edition, check it out.  I believe the way we think of, and use maps, has really changed since the rise of GMaps.  Reading through a classic book like lets you appreciate what truly goes into building an effective and beautiful map.

Posted on

The Basics

What are our geo-analysis fundamentals?

Fundamentals should be conceptually simple, so that one can learn them and understand them quickly and easily.  Lately, three basic fundamentals of geo-analysis have been ringing in my head and I think anyone who works in “spatial” should really understand them.  Here they are, in no particular order:

If geo-analysis is your area of expertise, you should be able to discuss all three topics with some intelligence.  I know there are many more fundamentals that build the foundation of geo-analysis, but these are the three that have been thinking off lately.  Do you have any other geo-analysis fundamentals that you think are crucial to know?  If so, leave a comment.

Posted on

Ignite Locationtech Boston – March 25

The folks from AvidGeo, Boston’s #1 social spatial special interest group, are at it again.  Another awesome Meet-Up is scheduled for the end of this month.

From the AvidGeo Meet-Up Page:

 “In collaboration with the Eclipse Foundation’s LocationTech, this month we are doing another Ignite event around geospatial and open source technology.

 Space is limited, to register for this event please visit the eventbrite site here: http://ignitelocationtechboston1.eventbrite.com/

!!IMPORTANT!!
This event is not being managed through meetup, registering on this meetup page does not mean you have a ticket to attend, please visit the eventbrite site in the link above.”

The event is at Space with a Soul, 281 Summer Street in Boston (a short walk from South Station) and it starts at 7pm.

There are a few talks already in place and they look great – Hubway mapping, open-source multi-spectral imaging, Leaflet mapping, contributing to open source geo projects – what more could you ask for!

If you are in the Boston area and are into open geo you should need to come to this event.  I’m pumped that I can finally make an AvidGeo Meet-Up for the first time in months!

For more info check out Avid Geo’s website or hit them up on Twitter (@avidgeo).

See you there!

Posted on

Happy Presidents’ Day – Leaflet Edition!

I’ve been seeing a lot of Leaflet lately, whether it’s in my twitter stream, on Boston.com, or hearing others in the geo-community talk about enthusiastically.  So, on a snowy Sunday in Somerville, I decided to give a Leaflet a try.  To honor the 43 presidents (remember, Grover Cleveland was both the 22nd and 24th president) of the United States of America I put together a simple leaflet app of each president’s birthplace (according to Wikipedia).

Leaflet in Action
Leaflet in Action

I’ve built plenty of web mapping apps with Google and Esri APIs, but by no means am I an expert.  If you have ever built a web-map using either of those APIs you will be able to build and launch a web map with Leaflet, no problem.  More than likely, you will be able to create a web ready map more quickly with Leaflet as well (as was my experience).

The app I built adds a few markers with custom icons and modified popups.  I read in and customized a state boundary geojson file from a Leaflet tutorial to give the user some context of “where” at larger scales.  The background tiles are from CloudMade and are nice and fast.

I don’t think I configured the autopan for the popups correctly as it doesn’t work as I think it should.  More than likely I just don’t have the right settings configured.  If you see something in my jumbled (and undocumented) code leave a comment and I’ll make the necessary updates.  I would eventually like to add a drop shadow to the presidential seal icons and perhaps read the data directly from a PostGIS database, as opposed to creating static markers.

The Leaflet documentation was easy to understand and the samples provided enough guidance to get a map online that did what I wanted it to do.  I wish there were a few more samples available through the Leaflet tutorials section, but those will come as the user community grows.

Overall, my first experience with Leaflet was generally pleasant.  I know JavaScripting  but I am not an expert and I was able to get a map online pretty quickly.  I spent more time collecting and formatting the data than I did getting the map online.  That is a good thing.  I recommend to those who have programmed with the Google or Esri API to check Leaflet out.  Added bonus – free and open source.

The page I created is for demo purposes only.  I only tested this in Chrome and Firefox (sorry IE users).  If there is incorrect information in the map please let me know and I will update it accordingly.

Finally, here is the direct link to the app:

http://www.gisdoctor.com/presidents.html

Happy presidential mapping!

 

Posted on

Awkward Geo-Coding

Back in the “old” days (2004-2007) I used to do a lot of geo-data cleaning for municipalities and county governments across the northeast for the consulting firm I was working for.  I was mostly cleaning address ranges on centerline datasets, checking road directionality and working on parcel addressing.  It wasn’t glamorous work, but it served an important purpose for the clients.  If the work I was doing was wrong, the services within the municipality could suffer, and I didn’t want to be the guy who messed-up.

I used to take a lot of pride in getting addressing correct or generating datasets of important locations within local municipality. Whenever I see a bad geo-code or a misplaced placemark in an online mapping service I simultaneously cringe and chuckle.  It isn’t easy work to build these massive databases and for the most part they are  remarkably correct. However, every now and then, an error creeps through.  For example, my parents home address registers three miles from where it actually is located when using the Google Maps, however Bing, OSM, and the municipality’s local online mapping tool all place it in the correct location.

There is also the issue of bad placemarkers across the worldwide web of mapping.  I particularly enjoyed what I believe to be a bad placemarker that I found the other day. I work in downtown Boston and I’m often using Google Maps to search for directions to restaurants, bars, friends places, etc.  For some reason that escapes me, I was map-browsing around Fenway Park when I came across a strange placemarker a few dozen rows up from home plate on the third base side:

image credit - Google 2013
image credit – Google 2013

Huh?!?!

I’ve walked around Fenway many times in my life.  I was just over there a couple weeks ago and I didn’t notice this establishment (neither has Bing or OSM).  Maybe this is a new venture by the Fenway Sports Group to keep disgruntled Red Sox fans at the park during a prolonged losing streak? Perhaps this is a new type of viral marketing campaign? I don’t know if this placemark is correct or not, but I found it hilarious.  As of 1/27/2013 the map still had the unique placemark, but if it is wrong I believe it won’t be online much longer, especially if Larry Lucchino sees it.

Here is the full screen grab of the park taken on 1/27/2013 with the exciting new business on Yakwey Way:

image credit: Google 2013
image credit: Google 2013

When clicking on placemarker you get this interesting balloon:

image credit: Google 2013
image credit: Google 2013

As I have mentioned before, building and maintaining these massive address and placemarker datasets is tough work and if you have ever done it you know what I am talking about.  But this brings up a larger question.  Have you ever encountered an awkward geo-code?  Have you ever seen a placemarker that’s obviously in the wrong location, or tried searching for an address and taken somewhere you didn’t expect?  If so, post it to twitter and add the hashtag #awkwardgeocode or leave a comment. Let’s start capturing these map anomalies before they are corrected.  If you have a #awkwardgeocode feel free to share!