Node Dangles

WordPress Permalinks Snafu

Mon, 21 Jan 2019 05:19:03 -0600

Using a free StatCounter.com add-in, I get weekly emails showing the number of visitors to this blog. During the summer, the numbers plummeted. At the time, I thought it had to do with Google/Chrome requiring https.

Luckily, HostGator (disclosure: I’m working on becoming a HostGator affiliate), who I use to host this blog and a few other websites, provided free https certificates for all of my sites. So I made sure that https://MapRantala.com worked and thought the numbers would rebound.

Looking at the chart, you can see they did only slightly. I assumed that it would just be a matter of time for Google to see that https was enabled and since the blog is more a place for me to make notes to myself, I wasn’t concerned.

This weekend, I decided to take a look at Google’s Search Console and see if I could determine what was going on. Looking in the coverage, I saw many 404 results for pages. As I investigated some of them, I realized that it was due to my Permalinks setting.

The URLs, as shown below, were not working.

The website was using ‘https://maprantala.com/index.php/' as the base URL. So, for example, ‘https://maprantala.com/2017/08/07/arcgis-pro-2-0-migration-overview' did not work but ‘https://maprantala.com/index.php/2017/08/07/arcgis-pro-2-0-migration-overview’ did.

To change the WordPress Permalinks setting, I went to Settings-Permalinks and removed he ‘index.php/’ from the Custom Structure. Waiting a day, the 404s have disappeared in Search Console and, in theory, I just need to wait for Google to re-index the pages which are currently shown as ‘Crawled – currently not indexed’

WordPress Permalinks Setting

That may not have been the whole problem. I republished this blog under a different name for a year. But moved it back to this domain when that domain registration expired. So that may have been another factor.

Quick & Dirty: REST calls via python

Fri, 24 Aug 2018 18:11:30 -0500

Was sharing a python script with a php developer to illustrate how a couple of requests can be made to an ArcGIS REST endpoint.

Nothing fancy–one gets the data for one record, one gets data for all records, and the final gets the data for the record that intersects a point.

The only thing to note is that the original version that I shared gets a token because the service it uses is secured. I commented out the routine to get a token and just substituted in a blank string and pointed the code at one of esri’s sample services at https://sampleserver6.arcgisonline.com/arcgis/.

The file can be downloaded here.

import requests, urllib, httplib, urllib2, json

serverURL = 'https://sampleserver6.arcgisonline.com/arcgis/'
tokenURL =  serverURL+"tokens/"
phaseQueryURL = serverURL+"rest/services/Census/MapServer/3/query"

#Fields to be aware of:
# PhaseName = Name of the phase, a unique value.
# Status_Date = Date a phase was marked as "Live"
# PhaseStatus = Live, Provisioning, Pending

def makeRequest(inURL,inPayload):
    theRequest = urllib2.Request(url=inURL,data = urllib.urlencode(inPayload))
    theResponse = urllib2.urlopen(theRequest)
    responseString = theResponse.read()
    theJSON = json.loads(responseString)
    return theJSON

def getToken(inUser,inPass):
    inputData = {'username':inUser,'password':inPass, 'f':'json'}
    theJSON = makeRequest(tokenURL,inputData)
    print(theJSON)
    token = theJSON['token']
    return token

def queryAllStates(inToken):
    inputData = { 'where': "1=1", 'f': 'geojson', 'outFields': '*' , 'outSR': 4326, 'token':inToken}
    theJSON = makeRequest(phaseQueryURL,inputData)
    return theJSON

def queryState(inPhase,inToken):
    inputData = { 'where': "STATE_NAME = '{}'".format(inPhase), 'f': 'geojson', 'outFields': '*' , 'outSR': 4326, 'token':inToken}
    theJSON = makeRequest(phaseQueryURL,inputData)
    return theJSON

def findStateForPoint(inX,inY,inToken):
    inputData = {'geometry': { 'x': inX, 'y': inY}, 'geometryType': 'esriGeometryPoint', 'inSR':4326, 'spatialRel':'esriSpatialRelIntersects', 'outFields':'*', 'f': 'json','token':inToken}

    theJSON = makeRequest(phaseQueryURL,inputData)
    return theJSON

username = ""
password = ""

##theToken = getToken(username,password)
theToken = ""
print(theToken)

#Attribute Query to get information about one State
stateInfo = queryState('Wisconsin',theToken)
print(stateInfo)

#Attribute Query to get information about all States
statesInfo = queryAllStates(theToken)
print(statesInfo)


#Find phase for a lat-lon
theState = findStateForPoint(-93.280842,44.95531,theToken)
print(theState)

Forcing ArcGIS Online to Update Feature Layer Information

Fri, 22 Jun 2018 20:26:28 -0500

Because we use ArcGIS Online Web Maps in our daily work-flows, I created an AGO group that contains Feature Layers that are links to our self-hosted ArcGIS Server services. This way, users can quickly add data from our ArcGIS Server Site and not have to look up URLs/Usernames/Passwords (I’ve save the username/password as part of the ArcGIS Online service when appropriate).

But I’ve had a problem where if I change the service, the changes are not reflected in the AGO Feature Layer for that service and I have to monkey about to get the changes to migrate to the maps.

Finally annoyed with doing that for the Xth time, I decided to see if I could get the feature service information to refresh automatically. Documentation search found nothing; call in to tech support–still waiting. While experimenting, however, I found that newer Feature Layer I added this way do actually update when I made changes. WTH?!

Using ArcGIS Online Assistant, something caught my eye. My older Feature Layers had a Description and Data but the newer (automatically updating ones) do not.

Was this the answer–the 912 line JSON file for this service had all the information about the Feature Service. So it seems like the information about my Feature Service probably got saved when I create the Feature Layer and this cache was used later instead of gathering new information. Now that I had a suspected reason why this was occurring, I needed to figure out how to either delete or force a refresh of the cache. There’s no delete option next to the Data section so I figured I would empty the content and see if that would work.

WARNING: Unsupported Technique Ahead!! Proceed at Own Risk–Test First with a Copy of Data!!!

Using AGO Assistant, I copied the Feature Layer that did not recognize when I made changes to the Feature Service and then made a map with that new Feature Layer so that I could test against a copy of the data. Once I had those, I went to the Copy of my Feature Layer in AGO Assistant and deleted almost everything in the Data section and saved that. (Note that it despite saying it saved my edits when I deleted everything, the contents were still there so you have to leave something in there).

Once I went and looked at the Feature Layer and it showed the layer I had previously added that was not showing up. And when added to a New map, it reflected all my changes. The existing map, however, still had previous settings–a new field, for example, did not appear.

So I found a partial fix–the Feature Layer hopefully will not cache any information about the Feature Service so when I add it to a map, the most recent settings will be used. Will still need to update maps but at least I have a Feature Layer now that reflects the most recent changes to the Feature Service.

Using ArcPy.Mapping to update Definition Queries

Tue, 08 May 2018 17:56:09 -0500

Working with a routine process today that I normally only do once in awhile but today needed to do it several times. It requires changing the definition query on several features classes. Being the ‘lazy’ GIS guy that I am (owner of a company I used to work at called me that once as a sort of compliment for my tendency to script a lot of what I did), I decided to finally script it instead of changing definition queries about 42 times.

The quick & dirty ~~script~~ code that I wrote & can be run in the ArcMap python window consists of two parts. The first three lines just need to be run once per session to get a variable set with the list of layers. The loop I ran multiple times, updating the ‘.replace’ settings each time. I ran the loop, did my other processes, and re-ran the loop as needed and was a happy GISer.

#These first 3 lines only need to be run once.
mxd = arcpy.mapping.MapDocument("CURRENT")
l = arcpy.mapping.ListDataFrames(mxd)[0] #Note the hard-code [0], meaning the first dataframe. YMMV
ll = arcpy.mapping.ListLayers(l)

#Everytime I wanted to update the definition queries just need to update the replace parameters.

for l in ll:
     try:
        print(l.name)
        print(l.definitionQuery)
        z = l.definitionQuery.replace("_011","_012")
        l.definitionQuery = z
     except:
        print("")

ArcToolbox Tool: Add [ST_CON_ABR] to Hennepin County, MN Centerlines

Wed, 04 Oct 2017 22:17:11 -0500

One of the great advancements over the last decade plus in GIS is that government agencies have started to move away from a ‘recover-our-cost’ mentality to more of an ‘Open Data’. Minnesota, for example, has launched their Geospatial Commons as a platform for sharing data.

And while getting free, authoritative data is awesome, it can leave you in a bind if the structure of the data changes. Sometime between April and September, Hennepin County, Minnesota, changed the schema of their publicly available street centerlines data.

The data used to have both a full, concatenated street name field ([ST_CONCAT]) and an abbreviated version ([ST_CON_ABR]). A record might have ‘James Lofton Avenue North’ and ‘James Lofton Ave N’, respectively, in these two fields. The abbreviated version was nice for labeling but it disappeared from the most recent updates.

So, as you might guess from the fact that I’m posting about it, I wrote a script to add that field in and launch it from an ArcToolbox Tool. Nothing fancy going on in the code, just a series of replaces, depending on the field. Using dictionaries instead of arrays of paired values might have been better but the script takes just a few seconds to run so I can live with it as-is.

The list of street type abbreviations came from a combination of ESRI’s standards and those found in an older version of the Hennepin County data. There were no conflicting abbreviations between the two. The code warns if a street name occurs in the data that is not in the list.

While I’m including the code here for reference, it’s probably best to download the code from GitHub.

#-------------------------------------------------------------------------------
# Name:        usi_dataprep_Add_STCONABR
#
# Purpose:     This can be used to add [CT_CON_ABR] to Hennepin County, MN
#              centerlines. This is a concatenated, abbreviated full name of
#              the street. This used to be included in the data but
#              disappeared from the downloads in the summer of 2017.
#
#              Data available at: http://www.hennepin.us/gisopendata
#
# Author:      mrantala
#
# Created:     2017.10.04
#
#------------------------------------------------------------------------------

import arcpy

############################################
## Custom Variables
#These are the fields that are concatenated. Hennepin has others but were always blank.
requiredFieldList = ["ST_PRE_DIR","ST_PRE_TYP","ST_NAME","ST_POS_TYP","ST_POS_DIR"]
#This is the name of the field to add
newFieldName = "ST_CON_ABR"

#These are the abbreviations for [ST_POS_TYPE]. The list was created using a sample of
#Hennepin's centerline data & Esri Tech article: http://support.esri.com/en/technical-article/000008454
# Note that I intentionally left cases in where there is no abbreviation (Fall, for example) as a means of
#documenting the fact that it should NOT change.

abbList = []
abbList.append(["Alcove","Alcove"]) #Hennepin Specific
abbList.append(["Alley","Aly"])
abbList.append(["Annex","Anx"])
abbList.append(["Arcade","Arc"])
abbList.append(["Avenue","Ave"])
abbList.append(["Bay","Bay"]) #Hennepin Specific
abbList.append(["Bayoo","Byu"])
abbList.append(["Beach","Bch"])
abbList.append(["Bend","Bnd"])
abbList.append(["Bluff","Blf"])
abbList.append(["Bluffs","Blfs"])
abbList.append(["Bottom","Btm"])
abbList.append(["Boulevard","Blvd"])
abbList.append(["Branch","Br"])
abbList.append(["Bridge","Brg"])
abbList.append(["Brook","Brk"])
abbList.append(["Brooks","Brks"])
abbList.append(["Burg","Bg"])
abbList.append(["Burgs","Bgs"])
abbList.append(["Bypass","Byp"])
abbList.append(["Camp","Cp"])
abbList.append(["Canyon","Cyn"])
abbList.append(["Cape","Cpe"])
abbList.append(["Causeway","Cswy"])
abbList.append(["Center","Ctr"])
abbList.append(["Centers","Ctrs"])
abbList.append(["Crossings","Crossings"]) #Hennepin Specific
abbList.append(["Crossroad","Xrd"])
abbList.append(["Chase","Chase"]) #Hennepin Specific
abbList.append(["Circle","Cir"])
abbList.append(["Circles","Cirs"])
abbList.append(["Cliff","Clf"])
abbList.append(["Cliffs","Clfs"])
abbList.append(["Club","Clb"])
abbList.append(["Close","Close"]) #Hennepin Specific
abbList.append(["Common","Cmn"])
abbList.append(["Commons","Cmns"]) #Hennepin Specific
abbList.append(["Corner","Cor"])
abbList.append(["Corners","Cors"])
abbList.append(["Corridor","Corridor"]) #Hennepin Specific
abbList.append(["Course","Crse"])
abbList.append(["Court","Ct"])
abbList.append(["Courts","Cts"])
abbList.append(["Cove","Cv"])
abbList.append(["Coves","Cvs"])
abbList.append(["Creek","Crk"])
abbList.append(["Crescent","Cres"])
abbList.append(["Crest","Crst"])
abbList.append(["Cross","Cross"]) #Hennepin Specific
abbList.append(["Crossing","Xing"])
abbList.append(["Curve","Curve"])
abbList.append(["Dale","Dl"])
abbList.append(["Dam","Dm"])
abbList.append(["Divide","Dv"])
abbList.append(["Down","Down"]) #Hennepin Specific
abbList.append(["Downs","Downs"]) #Hennepin Specific
abbList.append(["Drive","Dr"])
abbList.append(["Drives","Drs"])
abbList.append(["Edge","Edge"]) #Hennepin Specific
abbList.append(["Entry","Entry"]) #Hennepin Specific
abbList.append(["Estate","Est"])
abbList.append(["Estates","Ests"])
abbList.append(["Expressway","Expy"])
abbList.append(["Extension","Ext"])
abbList.append(["Extensions","Exts"])
abbList.append(["Fall","Fall"])
abbList.append(["Falls","Fls"])
abbList.append(["Ferry","Fry"])
abbList.append(["Field","Fld"])
abbList.append(["Fields","Flds"])
abbList.append(["Flat","Flt"])
abbList.append(["Flats","Flts"])
abbList.append(["Ford","Frd"])
abbList.append(["Fords","Frds"])
abbList.append(["Forest","Frst"])
abbList.append(["Forge","Frg"])
abbList.append(["Forges","Frgs"])
abbList.append(["Fork","Frk"])
abbList.append(["Forks","Frks"])
abbList.append(["Fort","Ft"])
abbList.append(["Freeway","Fwy"])
abbList.append(["Gables","Gables"]) #Hennepin Specific
abbList.append(["Garden","Gdn"])
abbList.append(["Gardens","Gdns"])
abbList.append(["Gate","Gate"]) #Hennepin Specific
abbList.append(["Gateway","Gtwy"])
abbList.append(["Glade","Glade"]) #Hennepin Specific
abbList.append(["Glen","Gln"])
abbList.append(["Glens","Glns"])
abbList.append(["Green","Grn"])
abbList.append(["Greens","Grns"])
abbList.append(["Greenway","Greenway"]) #Hennepin Specific
abbList.append(["Grove","Grv"])
abbList.append(["Groves","Grvs"])
abbList.append(["Harbor","Hbr"])
abbList.append(["Harbors","Hbrs"])
abbList.append(["Haven","Hvn"])
abbList.append(["Heights","Hts"])
abbList.append(["Highway","Hwy"])
abbList.append(["Hill","Hl"])
abbList.append(["Hills","Hls"])
abbList.append(["Hollow","Holw"])
abbList.append(["Horn","Horn"]) #Hennepin Specific
abbList.append(["Inlet","Inlt"])
abbList.append(["Island","Is"])
abbList.append(["Islands","Iss"])
abbList.append(["Isle","Isle"])
abbList.append(["Junction","Jct"])
abbList.append(["Junctions","Jcts"])
abbList.append(["Key","Ky"])
abbList.append(["Keys","Kys"])
abbList.append(["Knoll","Knl"])
abbList.append(["Knolls","Knls"])
abbList.append(["Lake","Lk"])
abbList.append(["Lakes","Lks"])
abbList.append(["Land","Land"])
abbList.append(["Landing","Lndg"])
abbList.append(["Lane","Ln"])
abbList.append(["Light","Lgt"])
abbList.append(["Lights","Lgts"])
abbList.append(["Loaf","Lf"])
abbList.append(["Lock","Lck"])
abbList.append(["Locks","Lcks"])
abbList.append(["Lodge","Ldg"])
abbList.append(["Loop","Loop"])
abbList.append(["Mall","Mall"])
abbList.append(["Manor","Mnr"])
abbList.append(["Manors","Mnrs"])
abbList.append(["Meadow","Mdw"])
abbList.append(["Meadows","Mdws"])
abbList.append(["Mews","Mews"])
abbList.append(["Mill","Ml"])
abbList.append(["Mills","Mls"])
abbList.append(["Mission","Msn"])
abbList.append(["Motorway","Mtwy"])
abbList.append(["Mount","Mt"])
abbList.append(["Mountain","Mtn"])
abbList.append(["Mountains","Mtns"])
abbList.append(["Neck","Nck"])
abbList.append(["Orchard","Orch"])
abbList.append(["Oval","Oval"])
abbList.append(["Overpass","Opas"])
abbList.append(["Park","Park"])
abbList.append(["Parks","Park"])
abbList.append(["Parkway","Pkwy"])
abbList.append(["Parkways","Pkwy"])
abbList.append(["Pass","Pass"])
abbList.append(["Passage","Psge"])
abbList.append(["Path","Path"])
abbList.append(["Pike","Pike"])
abbList.append(["Pine","Pne"])
abbList.append(["Pines","Pnes"])
abbList.append(["Place","Pl"])
abbList.append(["Plain","Pln"])
abbList.append(["Plains","Plns"])
abbList.append(["Plaza","Plz"])
abbList.append(["Point","Pt"])
abbList.append(["Points","Pts"])
abbList.append(["Port","Prt"])
abbList.append(["Ports","Prts"])
abbList.append(["Prairie","Pr"])
abbList.append(["Radial","Radl"])
abbList.append(["Railroad","Railroad"]) #Hennepin Specific
abbList.append(["Ramp","Ramp"])
abbList.append(["Ranch","Rnch"])
abbList.append(["Rapid","Rpd"])
abbList.append(["Rapids","Rpds"])
abbList.append(["Rest","Rst"])
abbList.append(["Ridge","Rdg"])
abbList.append(["Ridges","Rdgs"])
abbList.append(["Rise","Rise"]) #Hennepin Specific
abbList.append(["River","Riv"])
abbList.append(["Road","Rd"])
abbList.append(["Roads","Rds"])
abbList.append(["Route","Rte"])
abbList.append(["Row","Row"])
abbList.append(["Rue","Rue"])
abbList.append(["Run","Run"])
abbList.append(["Shoal","Shl"])
abbList.append(["Shoals","Shls"])
abbList.append(["Shore","Shr"])
abbList.append(["Shores","Shrs"])
abbList.append(["Skies","Skies"]) #Hennepin Specific
abbList.append(["Skyway","Skwy"])
abbList.append(["Spring","Spg"])
abbList.append(["Springs","Spgs"])
abbList.append(["Spur","Spur"])
abbList.append(["Spurs","Spur"])
abbList.append(["Square","Sq"])
abbList.append(["Squares","Sqrs"])
abbList.append(["Station","Sta"])
abbList.append(["Stravenue","Stra"])
abbList.append(["Stream","Strm"])
abbList.append(["Street","St"])
abbList.append(["Streets","Sts"])
abbList.append(["Summit","Smt"])
abbList.append(["Terrace","Ter"])
abbList.append(["Throughway","Trwy"])
abbList.append(["Trace","Trce"])
abbList.append(["Track","Trak"])
abbList.append(["Trafficway","Trfy"])
abbList.append(["Trail","Trl"])
abbList.append(["Tunnel","Tunl"])
abbList.append(["Turn","Turn"]) #Hennepin Specific
abbList.append(["Turnpike","Tpke"])
abbList.append(["Underpass","Upas"])
abbList.append(["Union","Un"])
abbList.append(["Unions","Uns"])
abbList.append(["Valley","Vly"])
abbList.append(["Valleys","Vlys"])
abbList.append(["Viaduct","Via"])
abbList.append(["View","Vw"])
abbList.append(["Views","Vws"])
abbList.append(["Village","Vlg"])
abbList.append(["Villages","Vlgs"])
abbList.append(["Ville","Vl"])
abbList.append(["Vista","Vis"])
abbList.append(["Walk","Walk"])
abbList.append(["Walks","Walk"])
abbList.append(["Wall","Wall"])
abbList.append(["Way","Way"])
abbList.append(["Ways","Ways"])
abbList.append(["Well","Wl"])
abbList.append(["Wells","Wls"])

#List of changes for [St_POS_Dir]
posDirList = [["North","N"],["East","E"],["South","S"],["West","W"],["Northeast","NE"],["Northwest","NW"],["Southeast","SE"],["Southwest","SW"]]
preDirList = [["North","N"],["East","E"],["South","S"],["West","W"]]
############################################
## Read Arguments

if (len(sys.argv) > 1):
    inFC = sys.argv[1]

############################################
# General Purpose Functions
def printit(inputString):
    try:
        print(inputString)
        arcpy.AddMessage(str(inputString))
    except:
        pass

def printerror(inputString):
    print (inputString)
    arcpy.AddError(inputString)

def getField(inFeatureClass, inFieldName):
  fieldList = arcpy.ListFields(inFeatureClass)
  for iField in fieldList:
    if iField.name.lower() == inFieldName.lower():
      return iField
  return None

def fieldExists(inFeatureClass, inFieldName):
  return getField(inFeatureClass,inFieldName) <> None

############################################
# Initial QC

def initialQC():
    if (arcpy.Exists(inFC)):
        printit("PASS: Feature Class {} Exists".format(inFC))
    else:
        printerror("ERROR: Feature Class {} Does Not Exist, Cancelling...".format(inFC))
        return False

    for iFld in requiredFieldList:
        if (fieldExists(inFC,iFld)):
            printit("PASS: Feature Class {} Has Field [{}]".format(inFC,iFld))
        else:
            printerror("ERROR: Feature Class {} Does Not Have Field [{}], Cancelling...".format(inFC,iFld))
            return False

    if not (fieldExists(inFC,newFieldName)):
        printit("GOOD: Feature Class {} Does Not Already Have Field [{}]".format(inFC,newFieldName))
        printit(" ADDING Field [{}]".format(newFieldName))
        try:
            arcpy.AddField_management(in_table=inFC, field_name=newFieldName, field_type="TEXT", field_precision="", field_scale="", field_length="100", field_alias="", field_is_nullable="NULLABLE", field_is_required="NON_REQUIRED", field_domain="")
        except:
            printerror("ERROR: Error While Adding Field [{}], Cancelling...".format(newFieldName))
            return False
        if not (fieldExists(inFC,newFieldName)):
            printerror("ERROR: Unable to Add Field [{}], Cancelling...".format(newFieldName))
            return False
    else:
        printerror("ERROR: Feature Class {} Already Has Field [{}], Cancelling...".format(inFC,newFieldName))
        return False

    return True

############################################
# Main

def makeSubstitution(inList,inValue,inFieldName):
    for iAbbreviationPr in inList:
        if (inValue == iAbbreviationPr[0]): #Found a Match
            return iAbbreviationPr[1]
    printit("WARNING: [{}] of {} does not have a value in the abbreviation list! Potential Error...".format(inFieldName,inValue))
    return inValue

def main():
    cursorFieldList = requiredFieldList
    cursorFieldList.append(newFieldName)

    try:
        iUCursor = arcpy.da.UpdateCursor(inFC,cursorFieldList)
        iRowCount = 0
        iRowMax = 1
        for uRow in iUCursor:

            #Just to give user an indicator that progress is being made
            if (iRowCount > iRowMax):
                printit(" {}".format(iRowCount))
                iRowMax *= 10
                iRowCount+=1

            abbreviateConcatenatedName = ""
            iFldIndex = 0
            for iFld in requiredFieldList:


                if (iFld == newFieldName):
                    uRow[iFldIndex] = abbreviateConcatenatedName
                    iUCursor.updateRow(uRow)
                else:
                    iValue = uRow[iFldIndex].strip() #Strip is just a safe-guard


                    if ((iValue != "") and (iValue != None)):
                        if (iFld == "ST_PRE_DIR"):
                            iValue= makeSubstitution(preDirList,iValue,"ST_PRE_DIR")
                        if (iFld == "ST_POS_TYP"):
                            iValue= makeSubstitution(abbList,iValue,"ST_POS_TYPE")
                        if (iFld == "ST_POS_DIR"):
                            iValue = makeSubstitution(posDirList,iValue,"ST_POS_DIR")

                        if (abbreviateConcatenatedName == ""):
                            abbreviateConcatenatedName = iValue
                        else:
                            abbreviateConcatenatedName+=" "+iValue
                iFldIndex += 1

        del iUCursor
    except RuntimeError as e:
        printerror("ERROR: Error {} Occurred, Cancelling...".format(e))
        try:
            del iUCursor
            del uRow
        except:
            return False
    return True


if __name__ == '__main__':
    if (initialQC() == True):
        if (main() == True):
            printit("Done!")

ArcGIS Pro 2.0 Migration Overview

Mon, 07 Aug 2017 19:48:08 -0500

In April I started a new position at a company that had no existing GIS.

Nothing.

There was a definite need for GIS and some GIS-type functions were occuring but basically when I started, I had an ArcGIS Enterprise license and a mess of KML files.

An exciting opportunity. And since I was starting from scratch, I had zero legacy concerns. No existing data, workflows, custom code, or maps to tie me to a specific software package. Knowing that ArcGIS Pro is taking over the world, I decided to transition to ArcPro and use that as our (GIS staff count of 1) company standard for desktop mapping.

That plan got side-tracked somewhat when we purchased a data management tool (CrescentLink Network Manager) that is only available for ArcMap. But I have still been using Pro on a regular basis.

I started a brilliant post about the transition–the pros, the cons, and my overall experience. But before I could finish it, out came 2.0 and my post was suddenly out-dated. So I’ve decided to split that post into a bunch of smaller, more focused posts.

Overview

Without getting into to deep on specific features or functionality, I do have some broad comments to make.

Learning Curve

There is definitely a learning curve in moving from ArcGIS Desktop to ArcGIS Pro–even though there is a lot of the same concepts between Pro & Desktop (the Toolboxes have barely changed), just finding the tools was a huge hurdle at first. I’ve never been a fan of the ribbon interface, I like my tools to be where they are and accessible.

But with regular use, I have gotten more adept at finding what I want to use. Still end up hunting for tools at times but it has gotten better. I’m probably at about 75% efficiency as compared to Desktop although I switch between the two on a regular basis because there are times where I just need to get something done.

Stability

Even though I just had to force-stop a session, 2.0 has made significant strides in stability. I had near daily crashes with 1.5 but now it is maybe a once-a-week. Maybe I’ve learned what not to do in Pro but I don’t remember any pattern to the crashes I had with 1.5.

Performance

Performance is still painful to me. There are too many times when the wait cursor shows up when you do simple things like clicking on a button. A lot of the processing has been routed through the geoprocessing system and it just seems much slower.

Version 2.0 Highlights

While I’ll probably do some whining about ArcGIS Pro in this planned series of posts, there were a couple of significant highlights to the 2.0 release that took care of two of my major usability concerns.

Simultaneously running multiple instances of ArcGIS on the same machine. I did some initial scripting for a data preparation process. The script took a good 15-20 minutes to run. Using 1.5, I had to launch it and then work in something other than ArcGIS Pro. Now I can launch that process and continue to work in a second session.
Highlighting. Maybe I am just weird but I really missed the ability to highlight records in table. As part of my QC process, I will often select records using a spatial or attribute query and then go through that set and either unselect or reselect them in batches by first highlighting them. This was a huge issue I had with 1.5 especially since it does not seem like it should be difficult functionality to add in. I was going to wait on 2.0 until I saw this functionality.

Summary

Overall, I’ve grown accustom to ArcPro. Like any new software, it takes time to get a feel for it. The transition is not that much different from going from ArcView 3.x to ArcGIS Desktop (or whatever 8.x was called). I am not yet as productive in it as Desktop but there are some things I really like about it to go along with my complaints.

Esri Web AppBuilder 2.4, Empty Basemap Gallery Follow-Up

Sun, 06 Aug 2017 10:12:07 -0500

A while ago I posted a work-around for a problem I was having with a Web AppBuilder application. Working with Esri tech support, we determined what I was doing to cause the problem.

In my config.json, I was using a relative path to the proxy. Note that I had the proxy as a separate application at the root level of our domain because I intended to have a shared proxy for all of our applications instead of individual ones. So the WAB application would be using http://ourarcgisdomain.com/proxy/proxy.ashx.

"httpProxy": {
	"useProxy": true,
	"alwaysUseProxy": true,
    "url": "/proxy/proxy.ashx",
    "rules": []
  }

Well, that was causing the hiccup in the Basemap Gallery Widget. I simply modified the url parameter to have the full path of our domain, undid my custom code, and the Basemap Gallery Widget was happy.

"httpProxy": {
     "useProxy": true,
     "alwaysUseProxy": true,
     "url": "http://ourarcgisdomain.com/proxy/proxy.ashx",
     "rules": []
   }

Score one for Esri Tech Support.

Although, in my defense the documentation on the configuring the proxy is silent on the pathing.

Esri Web AppBuilder 2.4 Empty Basemap Gallery

Mon, 10 Apr 2017 20:08:37 -0500

I recently downloaded Web AppBuilder 2.4, Developer’s Edition and built a quick & dirty app. For some reason, it worked fine in Web AppBuilder but once I deployed it, no basemaps would show in the Basemap Gallery widget, it would just spin, spin, spin.

Digging around, I found some old posts where users were experiencing similar problems but none of the solution resulted in a great solution. I was able to get it to work by listing the basemaps individually in the config_BasemapGallery.json file but I that was more of a work-around. Notice in the thumbnailUrl, I have to put the full path to the thumbnails. Note that for brevity’s sake, I’m only showing one of the basemaps in this sample.

{
  "basemapGallery": {
    "showArcGISBasemaps": true,
    "basemaps": [{
      "layers": [{"url": "http://services.arcgisonline.com/ArcGIS/rest/services/World_Imagery/MapServer"}],
      "title": "Imagery",
      "thumbnailUrl": "http://<org>.maps.arcgis.com/sharing/rest/content/items/<id>/info/thumbnail/tempimagery.jpg",
      "spatialReference": {
         "wkid": "102100"
      }
     }]
   }
}

Eventually, I started comparing the requests generated by the version in the development version and the one in production and tracked the problem down to this POST request in production, which was returning a 400 Bad Request error:

http://<server>/proxy/proxy.ashx?https://<org>.maps.arcgis.com/sharing/rest/search&wab_dv=2.4

To this one in dev:

https://<org>.maps.arcgis.com/sharing/rest/search

I stripped off the ‘&wab_dv=2.4’ parameter from the production request and Boom! it worked. So I now knew what was causing the problem. Just had to figure out why it was occurring. I had actually already seen part of the answer while digging around in env.js while looking for a way to test my changes without having to clear my cache or remember to use an Incognito session. On line 81 of env.js, there is this variable, deployVersion which is set to 2.4.

I had recently worked on a solution that involved appending a time stamp onto scripts we were loading so I was familiar with what is going on–the code was appending a version onto the call so that if you change the deployVersion, the browser would load a new version of the code. The appendDeployVersion function on line 307 of env.js was doing the append. Notice the if statement on line 312, it checks for ‘?’ in the url. If it finds one, it appends ‘**&wab_dv=’ and the version, otherwise it appends ‘?**wab_dv=’ and the version–the difference being the ‘&’ vs ‘?’. A question mark is used before the first parameter in an URL, each additional parameter is separated by a ampersand (&) so that logic makes sense.

However, the monkey wrench is that I’m using a proxy, so my URL has a question mark related to the proxy:

http://<server>/proxy/proxy.ashx<strong>?</strong>https://<org>.maps.arcgis.com/sharing/rest/search&wab_dv=2.4

So the code was dutifully appending an ampersand and the version parameter ‘&wab_dv=2.4’ to my URL so the proxy would be requesting:

https://<org>.maps.arcgis.com/sharing/rest/search&wab_dv=2.4

when it should have been requesting:

https://<org>.maps.arcgis.com/sharing/rest/search?wab_dv=2.4

Once I had determined the problem (99% of the battle), I was able to quickly put in a patch that works for my instance. This patch isn’t a great one that Esri should use but it solves it in my case. I just added logic so that it would ignore proxy-related question marks:

function appendDeployVersion(url){
    if(/^http(s)?:\/\//.test(url) || /^\/proxy\.js/.test(url) || /^\/\//.test(url)){
      return url;
    }

	var proxyURL = url.toLowerCase().replace(".ashx?","");
    if(proxyURL.indexOf('?') > -1){
      url = url + '&wab_dv=' + deployVersion;
    }else{
      url = url + '?wab_dv=' + deployVersion;
    }
    return url;
  }

And voila! my Web AppBuilder now had all the basemaps options a mapineer could hope for:

2016 Grandma's Marathon Map

Mon, 16 May 2016 23:26:12 -0500

A Little Map I made for fun for Grandma’s Marathon ( @grandmasmara ) or click Here for the stand-alone map.

Converting MXD to Layer file in Arcpy

Fri, 14 Aug 2015 10:01:54 -0500

Working on doing some advanced ArcGIS server printing and had the need to batch convert many existing .mxd files to .lyr files. So instead of opening up X number of map documents, thought I would do it via code. All of my .mxds in this case had just one data frame so the process was pretty simple–I add an empty group layer (Thanks Petr Krebs for the idea), copy all the existing layers into it, and save it out as a layer file.

I created an ArcGIS toolbox with two options–one to convert a single .mxd and one to batch convert an entire folder. To use it, make sure to have the EmptyGroup.lyr in the same directory as the .py file.

Here is the raw code or git it:

import os
import arcpy
import inspect
import glob
import uuid
import inspect

codeDir = os.path.dirname(inspect.getfile(inspect.currentframe()))
EmptyGroupLayerFile = codeDir+"/EmptyGroup.lyr"
inArg1 = sys.argv[1]
inArg2 = sys.argv[2]

def printit(inMessage):
    arcpy.AddMessage(inMessage)

def makeLyrFromMXD(inMXD, outLyr):
    if not (os.path.exists(inMXD)):
        printit( "ERROR: {} does not exist".format(inMXD))
        return False
    if not (os.path.exists(EmptyGroupLayerFile)):
        printit( "ERROR: {} does not exist".format(EmptyGroupLayerFile))
        return False
    if  (os.path.exists(outLyr)):
        printit( "Skipping: {} already exists".format(outLyr))
        return True

    printit( "Making Layer file: {0}".format(outLyr))

    mxd = arcpy.mapping.MapDocument(inMXD)
    ###Right now, just doing the first Dataframe, this could be modified
    df = arcpy.mapping.ListDataFrames(mxd)[0]

    theUUID = str(uuid.uuid1())

    iGroupLayerRaw = arcpy.mapping.Layer(EmptyGroupLayerFile)
    iGroupLayerRaw.name = theUUID
    arcpy.mapping.AddLayer(df,iGroupLayerRaw,"TOP")
    groupBaseName = os.path.basename(outLyr).split(".")[0]

    for lyr in arcpy.mapping.ListLayers(df):
        if not (lyr.name == theUUID):
            if (lyr.longName == lyr.name):
                arcpy.mapping.AddLayerToGroup (df, iGroupLayer, lyr, "Bottom")
        else:
            iGroupLayer = lyr

    iGroupLayer.name = groupBaseName
    arcpy.SaveToLayerFile_management(iGroupLayer, outLyr)
    return os.path.exists(outLyr)

def doMultiple(inDir,outDir):
    for iMxd in glob.glob(inDir+"/*.mxd"):
        lyrFile = outDir+"/"+os.path.basename(iMxd).lower().replace(".mxd",".lyr")
        makeLyrFromMXD(iMxd, lyrFile)

if(not os.path.exists(EmptyGroupLayerFile)):
    printit("Error: {} is missing, can not run.".format(EmptyGroupLayerFile))
else:
    if (os.path.isdir(inArg1) and (os.path.isdir(inArg2))):
        doMultiple(inArg1,inArg2)
    elif (os.path.isfile(inArg1)):
        if (os.path.exists(inArg2)):
            printit("Error: {} already exists".format(inArg2))
        else:
            makeLyrFromMXD(inArg1,inArg2)
    else:
        printit("Unable to understand input parameters")

Calling os.startfile and webbrowser.open from ArcGIS

Wed, 31 Dec 2014 14:38:16 -0600

Recently I’ve created Python add-ins for data entry for our staff. Most of these have a toolbar with a ‘Help’ button that opens a help file in .pdf format.

Sample python add-in toolbar.

The first add-in was for ArcCatalog and this worked splendidly. I was using os.startfile(path to help.pdf).

However, when I started doing ArcMap add-ins, clicking the Help button would open the help.pdf but ArcMap would crash. Oops!

Luckily the Python development team at Esri already had a blog post about this at their ArcPy Café blog.

They report that the root of the problem is ‘conflicts in the way the Windows libraries expect to be called, they can fail or crash when called within ArcGIS for Desktop in an add-in script or geoprocessing script tool’. But this can be overcome by using a decorator function that calls os.startfile from a new thread. Another function effected by these conflicts is webbrowser.open.

Example code is shown below:

import functools
import os
import threading
import webbrowser
 
# A decorator that will run its wrapped function in a new thread
def run_in_other_thread(function):
    # functool.wraps will copy over the docstring and some other metadata
    # from the original function
    @functools.wraps(function)
    def fn_(*args, **kwargs):
        thread = threading.Thread(target=function, args=args, kwargs=kwargs)
        thread.start()
        thread.join()
    return fn_
 
# Our new wrapped versions of os.startfile and webbrowser.open
startfile = run_in_other_thread(os.startfile)
openbrowser = run_in_other_thread(webbrowser.open)

Then whenever you call startfile or openbrowser, it will be routed through your decorator function and, as far as I’ve been able to tell, works fine without crashing your ArcMap session.

Cheers!

Quick & Dirty Arcpy: Verify a Coded Value Domain Code

Thu, 18 Dec 2014 22:32:11 -0600

I’ve been working on a few different data import routines and one of the things I recently built was the ability to verify that a potential Code to be entered into a field with a Coded Value Domain is valid.

The logic of the code is pretty straight-forward. Get a field’s domain and check that a potential value is one of the code values. The biggest ‘trick’ in this code is that arcpy.da.ListDomains, which locates a field’s domain, takes a geodatabase (or Enterprise geodatabase connection file) as its only parameter. The documentation says it takes a workspace, but it does not like a feature dataset, which a feature class might be in.

A couple caveats about the code. It only returns True if a field exists, has a coded value domain, and the value tested is one of the (case-sensitive) valid codes. While I have an ArcToolbox tool to call it for illustration purposes, I’m only calling it from code so I wanted tight requirements.

Anyhow, here is the code or download it from GitHub.

import arcpy

inFeatureClass = sys.argv[1]
inField = sys.argv[2]
inValue = sys.argv[3]

# getFeatureClassParentWorkspace: This script gets the geodatabase for a
# feature class. The trick here is that feature classes can be within a
# feature dataset so you need to account for two possible levels in the
# directory structure.
def getFeatureClassParentWorkspace(inFeatureClass):
    describeFC = arcpy.Describe(inFeatureClass)
    if (describeFC.dataType == 'FeatureClass') or (describeFC.dataType == 'Table'):
        workspace1 = describeFC.path
        describeWorkspace1 = arcpy.Describe(workspace1)
        if (describeWorkspace1.dataType == 'FeatureDataset'):
            return describeWorkspace1.path
        return workspace1

    return None

# Find a field within a feature class
def getField(inFeatureClass, inFieldName):
  fieldList = arcpy.ListFields(inFeatureClass)
  for iField in fieldList:
    if iField.name.lower() == inFieldName.lower():
      return iField
  return None

#Get a field's domain
def getDomain(inFeatureClass, inField):
    theField = getField(inFeatureClass,inField)
    if (theField <> None):
        searchDomainName = theField.domain
        if (searchDomainName <> ""):
            for iDomain in arcpy.da.ListDomains(getFeatureClassParentWorkspace(inFeatureClass)):
                if iDomain.name == searchDomainName:
                    return iDomain
    return None

#Get the domain.
def validDomainValue(inFeatureClass,inField,inValue):
    theDomain = getDomain(inFeatureClass,inField)

    if not (theDomain is None):
        if (theDomain.domainType == "CodedValue"):
            if theDomain.codedValues.has_key(inValue):
                return True
    return False

if (validDomainValue(inFeatureClass,inField,inValue)):
    arcpy.AddMessage("Value ({0}) is valid for field [{1}].".format(inValue,inField))
else:
    arcpy.AddError("ERROR: Value ({0}) is invalid for field [{1}].".format(inValue,inField))

Quick & Dirty ArcPy: Listing Data Sources

Wed, 03 Dec 2014 08:05:10 -0600

I just had the need to go through a directory containing many (100+) layer files (.lyr) and verify the data sources in each. I could have loaded each into ArcMap and checked the properties, but choose not to. Here’s the bare-bones script I used instead:

import arcpy, glob,os

theDir = r"L:\gdrs\data\org\us_mn_state_dnr\elev_minnesota_lidar\\"
os.chdir(theDir)

for iFile in glob.glob("*.lyr"):
    print iFile
    lyr = arcpy.mapping.Layer(iFile)
    for i in arcpy.mapping.ListLayers(lyr):
        try:
            print "    {0}: {1}".format(i,i.dataSource)
        except:
            print "    {0}: Does not support dataSource".format(i)

print "Done!"

ArcMap Field Calculator: Python Nulls (None)

Fri, 10 Oct 2014 14:22:40 -0500

I’ve been asked this twice this week, so thought I should post it. Calculating Null values in the ArcMap Field Calculator is easy, if you can remember the right work for ‘Null’. The proper word is ‘None’ (without the quotes). Good luck & have a good weekend.

Quick & Dirty arcpy: Compare Feature Class Table Schemas

Mon, 07 Jul 2014 05:06:18 -0500

I’m in the process of rewriting a process, moving most of the processing from arcpy to postgresql-enabled python (love me some psycopg2).

One of the QC checks I’m doing at the end of this re-write is just verifying that the feature class schemas are the same (or that the differences are intended) under the new process as they were in the old process.

And while ArcGIS does have a good tool for this, there were a couple tweaks I wanted to make. Most notably, I wanted a list of fields that are not in both feature classes.

So I made a quick & dirty script to do that, nothing especially clever but I’ve found it useful. Download it from GitHub. I have it currently set up to work on feature layers but you should be able to change the toolbox parameter types to allow feature classes or tables.

import arcpy,sys,os

def printit(inMessage):
    print inMessage
    arcpy.AddMessage(inMessage)

featureclass1 = sys.argv[1]
featureclass2 = sys.argv[2]

tableheaders = 'name, type, width, precision, domain'

def makeFieldDict(inFC):
    d = arcpy.Describe(inFC)
    printit("Dataset: "+d.baseName)
    printit("Type: "+d.dataType)
    printit("Path: "+d.catalogPath)
    printit(" ")

    lFields=arcpy.ListFields(inFC)

    printit (tableheaders)
    fieldDict = dict()
    printit (lFields)
    for lf in lFields:
        fieldDict[lf.name] = [lf.name,lf.type,lf.length,lf.precision,lf.domain]
        printit (lf.name+", "+lf.type +", "+str(lf.length)+", "+str(lf.precision)+", "+lf.domain)
    return fieldDict

fieldDict1 = makeFieldDict(featureclass1)
fieldDict2 = makeFieldDict(featureclass2)
errorList = []
printit(" ")
printit(" ")
printit("Comparing Fields:")
for iField in sorted(list(set(fieldDict1.keys()+fieldDict2.keys()))):
    if not (fieldDict1.has_key(iField)):
        theResult = " {0} not found in {1}".format(iField,featureclass1)
        errorList.append(theResult)
    elif not (fieldDict2.has_key(iField)):
        theResult = " {0} not found in {1}".format(iField,featureclass2)
        errorList.append(theResult)
    else:
        if (fieldDict1[iField] == fieldDict2[iField]):
            theResult = " {0} OK".format(iField)
        else:
            theResult = " {0} Have Different Definitions \n   {1}: {2}\n   {3}: {4}".format(iField,featureclass1,fieldDict1[iField],featureclass2,fieldDict2[iField])
            errorList.append(theResult)

    printit( theResult )

printit(" ")
printit(" ")
if len(errorList) == 0:
    printit("GOOD! No difference Found!")
else:
    printit("These Differences Found:")
    for iError in errorList:
        printit(iError)

printit("Done!")

Garmin GPX to Shapefile (SHP) conversion GPX2Shp.py

Fri, 16 May 2014 21:11:20 -0500

I mentioned using Tapiriik to batch download my entire Garmin Connect history–over 1,000 separate .GPX files. I found several tools to convert .GPX to shapefiles that worked but none seemed to recognize my heart rate data.

The trick is Garmin extends the GPX specification to incorporate the heart rate:

<span style="color: #333333;">xmlns:gpxtpx="http://www.garmin.com/xmlschemas/TrackPointExtension/v1"</span>

Each track point looks like this:

2014-03-16T20:35:47+00:00

296.20001220703125

gpxtpx:TrackPointExtension>

trkpt>

Since the first few exiting GPX converters failed to meet my needs, I decided to make my own, at least partially.

I used Joel Lawhead of GeospatialPython.com‘s pyshp library to handle writing the shapefile. I added some basic loop and I stuck a template.prj in the workspace that gets copied once for each shapefile.

Otherwise, nothing too fancy going on. The code can be downloaded from Github.

import glob, os
import xml.etree.ElementTree as ET
import shapefile
import shutil

theNS = "{http://www.topografix.com/GPX/1/1}".lower()
theNS2 = "{http://www.garmin.com/xmlschemas/TrackPointExtension/v1}".lower()
templatePRJfile = "template.prj"

def elementIs(inElement,inTag):
    item1 = inTag.lower()
    item2 = elementName(inElement)
    return (inTag.lower() == elementName(inElement).lower())

def elementName(inElement):
    item1= inElement.tag.lower().replace(theNS,"").replace(theNS2,"")
    return item1

def convertTimeToSeconds(inTime):
    theSeconds = -1

    if (inTime.count(":")) == 2:
        try:
            inHour = inTime.split(":")[0]
            inMin = inTime.split(":")[1]
            inSec = inTime.split(":")[2]

            totalSec = float(inSec)
            totalSec += (float(inMin) * 60)
            totalSec += (float(inHour) * 3600)
            theSeconds = totalSec
        except:
            pass

    return theSeconds


def writeSHP(inSourceFile,inTrkList):
    w = shapefile.Writer(shapefile.POINT)
    w.field("file")
    w.field("segment","N","8",0)
    w.field("vertex","N","8",0)
    w.field("datetime","C",30)
    w.field("date","C","10",0)
    w.field("time","C","8",0)
    w.field("sec","N","8",0)
    w.field("isec","N","8",0)
    w.field("totsec","N","8",0)
    w.field("elev","N","24",14)
    w.field("hr","N","8",0)
    w.field("last","N","1",0)
    w.field("lat","N","24",16)
    w.field("lon","N","24",16)

    iTrkSegIndex = 0
    startSec =-1
    prevSec = -1
    for iTrkSeg in inTrkList:
        iTrkPtIndex = 0
        for iTrkPtDict in iTrkSeg:
            thisLine = "{0},{1},{2},*time*,*ele*,*hr*,*lat*,*lon*".format(inSourceFile,iTrkSegIndex,iTrkPtIndex)

            theLat = None
            if (iTrkPtDict.has_key('lat')):
                try:
                    theLat = float(iTrkPtDict['lat'])
                except:
                    pass

            theLon = None

            if (iTrkPtDict.has_key('lon')):
                try:
                    theLon = float(iTrkPtDict['lon'])
                except:
                    pass

            theDate = None
            theTime = None
            theSeconds = -1
            segSeconds = -1
            totSeconds = -1

            if (iTrkPtDict.has_key('time')):
                theDateTime = iTrkPtDict['time']
                if ("T" in theDateTime):
                    theDate = theDateTime.split("T")[0]
                    theTimePlue = theDateTime.split("T")[1]
                    if ("+" in theTimePlue):
                        theTime = theTimePlue.split("+")[0]
                        theSeconds = convertTimeToSeconds(theTime)

                        if (prevSec < 0):
                            prevSec = theSeconds
                        if (startSec < 0):
                            startSec = theSeconds

                        segSeconds = theSeconds - prevSec
                        prevSec = theSeconds
                        totSeconds = theSeconds - startSec
            else:
                theDateTime = None

            if (iTrkPtDict.has_key('ele')):
                theElev = iTrkPtDict['ele']
            else:
                theElev = None

            if (iTrkPtDict.has_key('hr')):
                theHR = iTrkPtDict['hr']
            else:
                theHR = None

            if (iTrkPtIndex == len(iTrkSeg) - 1):
                theLast = 1
            else:
                theLast = 0

            w.point(theLon, theLat)
            try:
                                  w.record(inSourceFile.replace(".gpx",""),iTrkSegIndex,iTrkPtIndex,theDateTime,theDate,theTime,theSeconds,segSeconds,totSeconds,theElev,theHR,theLast,theLat,theLon)

            except:
                print "############## ERROR ####################"
            iTrkPtIndex+=1

        iTrkSegIndex+=1


    w.save(inSourceFile.lower().replace(".gpx",""))
    w = None
    if (os.path.exists(templatePRJfile)):
        newPRJFN = inSourceFile.lower().replace(".gpx",".prj")
        shutil.copyfile(templatePRJfile,newPRJFN)

def mainLoop():
    for iFile in glob.glob("*.gpx"):
        print iFile
        tree = ET.parse(iFile)
        root=tree.getroot()

        theTrkList = []

        for iRoot in root:
            if elementIs(iRoot,"trk"): #"http://www.topografix.com/gpx/1/1}trk" in iRoot.tag.lower():
                for iTrkSeg in iRoot:
                    if not elementIs(iTrkSeg,"trkseg"):
                        continue
                    thisTrk = []

                    pntIndex = 0
                    for iTrkPt in iTrkSeg:
                        if not elementIs(iTrkPt,"trkpt"):
                            continue
                        trkPntDict = dict()
                        trkPntDict["pntIndex"] = pntIndex
                        trkPntDict['lat'] = iTrkPt.get('lat')
                        trkPntDict['lon'] = iTrkPt.get('lon')

                        pntIndex+=1
                        for iElem in iTrkPt:
                            if elementIs(iElem,"extensions"):
                                for iSubElem in iElem:
                                    if (elementIs(iSubElem,"TrackPointExtension")):
                                        for iExtensionElem in iSubElem:
                                            if elementIs(iExtensionElem,"hr"):
                                                trkPntDict[elementName(iExtensionElem)] = iExtensionElem.text
                            else:
                                trkPntDict[elementName(iElem)] = iElem.text

                        #print trkPntDict
                        thisTrk.append(trkPntDict)

                    theTrkList.append(thisTrk)
        writeSHP(iFile.lower(), theTrkList)


theLineList = mainLoop()

Zipping a Shapefile from ArcCatalog

Tue, 06 May 2014 05:42:04 -0500

Back in 2010, I posted a python script and an ArcToolbox tool for zipping a shapefile.

Well, I had a request to modify the code so it would not error out if it encounters a .lock file. While .lock files exist for a reason and shouldn’t be totally ignored, in some cases it is safe to do so, so I went ahead any modified the code, which can be downloaded from Github.

The guts of the code is here, though:

import zipfile
import sys
import os
import glob

theShapeFile = sys.argv[1]
outputZipFile = sys.argv[2]
skipLockFile = sys.argv[3]

def zipShapefile(inShapefile, newZipFN, skipLockFile):
    print 'Starting to Zip '+inShapefile+' to '+newZipFN

    if not (os.path.exists(inShapefile)):
        print inShapefile + ' Does Not Exist'
        return False

    if (os.path.exists(newZipFN)):
        print 'Deleting '+newZipFN
        os.remove(newZipFN)

        if (os.path.exists(newZipFN)):
            print 'Unable to Delete'+newZipFN
            return False

    zipobj = zipfile.ZipFile(newZipFN,'w')

    for infile in glob.glob( inShapefile.lower().replace(".shp",".*")):
        print infile
        if not ((os.path.splitext(infile.lower())[1] == ".lock") and (skipLockFile.lower() == "true")):
            zipobj.write(infile,os.path.basename(infile),zipfile.ZIP_DEFLATED)

    zipobj.close()

    return True

zipShapefile(theShapeFile,outputZipFile,skipLockFile)
print "done!"

Friday Fave: Geodatabase Geek

Fri, 02 May 2014 05:02:12 -0500

This Friday Fave is more for utility than pleasure.

Unfortunately, I have been working to determine why my views and query layers perform so much worse than directly accessing my feature class.

My Googling led me to Geodatabase Geek, by Trevor Hart, Eagle Technology Group Ltd. Trevor has some real good information about Geodatabases and also gave a good lightening talk on Usage Reporting on ArcGIS 10.1 for Server at the 2013 ESRI International Developer’s Conference.

One tool he pointed out was Mxdperfstat for benchmarking the performance of your MXD. Trevor used it to compare the performance of a Feature Class vs Query Layer vs Spatial View. While the official version is available for ArcGIS 9.3 through 10.2, I do want to point out Hussein Nasser’s 10.1 version which he put out before the official 10.1 version came out (it’s not really a version, more of a work-around but I like his ingenuity).

My results were significantly different on our 10.0 database server, the spatial view I was testing was much slower. The query for both the spatial view and query layer was simply ‘Select * from featureclass’

So not sure what to make of the performance yet, I’ve got a spatial index made so not sure what else I can try.

ArcSDE 10.0 Performance

Friday Fave: Tapiriik

Fri, 25 Apr 2014 05:55:13 -0500

This Friday Fave is a little bit different.

My interest in geospatial technologies (although we just called it GIS back then) largely because I wanted to measure my running routes more accurately and efficiently than the paper map & scrap of paper method I was using in the early 90s. When I was introduced to GIS, I knew what I was going to use it for.

Now that GPS technology is ubiquitous–I’m currently using four different GPS devices, at the same time, on my bike rides–I seldom have to use a map to measure my routes. I may still use MapMyRun.com to plan a route ahead of time if I’m running in a new area or trying to plan a loop of a certain distance but GPS has really made it so simple to just go out and run.

While there are several GPS options available, I have used Garmin ever since their 405 Forerunner came out. This was their first watch that didn’t get confused for a Timex-Sinclair 1000 strapped to your wrist.

Timex Sinclair 1000

Garmin’s watches upload their data to Gamin Connect, which works fine, but for a project I recently started, I wanted to down load all my data which Garmin Connect does not make easy. I had over 1,000 data logs and downloading them individually was not going to happen.

A little Googling led me to tapiriik.com, which allows you to share data amongst several different online services that endurance athletes might use including Runkeeper, Strava, Garmin Connect, SportTracks, DropBox.com, and Training Peaks.

You can use Tapiriik for free (you just need to visit their website to start synchronization) or for $2 per year they will automatically synchronize data between your accounts. You just provide your account information for whichever sites you want to synchronize and either visit their site or pay $2 and it will automatically share data between your accounts.

I linked to my Garmin Connect and Dropbox accounts and tapiriik and after some chugging, I had .GPX files for all of my data.

It was easy and didn’t take very long. Definitely met my needs–I haven’t shared data between other services but I do use the desktop version of SportTracks and considered using their web version of the software but didn’t know how I would upload all my data. Now I know.

This product definitely saved me a bunch of time. The one thing I wonder, though, is what does ‘Tapiriik’ means?

Quick & Dirty arcpy: Bulk Changing Field Values

Wed, 23 Apr 2014 05:18:24 -0500

In mapping cross sections, our geologists often find themselves renaming their stratigraphic units midway, or at the end, of creating multiple cross sections. This can cause a situation where we need to change multiple values in multiple fields in multiple feature classes–a situation that can get messy very fast.

Perfect situation for a quick & dirty arcpy script and, in this case, an ArcToolbox tool that can be downloaded.

This tool will change all feature classes in the O:\clay_cga\sand-distribution_model\dnrPackages\stratlines directory.

It will look at two fields, [strat] and [unit] and make these changes:

‘go’ becomes ‘gro’
‘goc’ becomes ‘grc’
‘sgb’ becomes ‘grb’

And since I have Case Sensitive checked, ‘Go’ will not get changed to ‘gro’. Also note that only full values that match values in the Old Value List get changed, part matches are left as-is so ‘got’ would be left as-is even though the first two characters match ‘go’.

import arcpy
import sys, string, arcgisscripting
import arcpy

def printit(inString):
    print inString
    arcpy.AddMessage(inString)

def printerr(inString):
    print inString
    arcpy.AddError(inString)

def fieldExists(tablename,indexname):

 if not arcpy.Exists(tablename):
  return False

 tabledescription = arcpy.Describe(tablename)

 for iField in tabledescription.fields:
     if (iField.Name.lower() == indexname.lower()):
         return True

 return False


if len(sys.argv) > 1:
    inDirectory = sys.argv[1]
    inFieldNameRaw = sys.argv[2]
    oldValue = sys.argv[3].replace(","," ")
    newValue = sys.argv[4].replace(","," ")
    caseSensitiveRaw = sys.argv[5]
else:
    inDirectory = r"C:\temp\test\stratest"
    inFieldNameRaw = "strat"
    oldValue = "go, goc, sgb".replace(","," ")
    newValue = "gro grc grb".replace(","," ")
    caseSensitiveRaw = "true"

caseSensitive = (caseSensitiveRaw.lower() == "true")
fieldNameList = inFieldNameRaw.replace(","," ").split()

printit("Starting")
printit(" Workspace: "+str(inDirectory))
printit( " inFieldName: "+str(inFieldNameRaw))
printit( " oldValue: "+str(oldValue))
printit( " newValue: "+str(newValue))
printit( " caseSensitive: "+str(caseSensitive))

valueDict = dict()

def initialQC():
    global valueDict

    if not (arcpy.Exists(inDirectory)):
        printerr("Workspace {0} does not exist".format(inDirectory))
        return False

    if (len(oldValue.split()) <> len(newValue.split())):
        printerr("Number of values in {0} does not equal number of values in {1}".format(oldValue,newValue))
        return False

    iValueIndex = 0
    for iOldValue in oldValue.split():
        if (caseSensitive):
            thisKey = iOldValue
        else:
            thisKey = iOldValue.lower()

        if (valueDict.has_key(thisKey)):
            printerr("ERROR: Value, {0}, is repeated, cancelling...".format(thisKey))
            return False

        valueDict[thisKey] = newValue.split()[iValueIndex]
        iValueIndex+=1
    return True

def makeFieldList(inFC):
    thisFieldList = []

    for iField in fieldNameList:
        if (fieldExists(inFC,iField)):
            thisFieldList.append(iField)

    return thisFieldList


def main():
    arcpy.env.workspace = inDirectory
    printit(valueDict)
    for iFC in arcpy.ListFeatureClasses():
        printit("Working on {0}".format(iFC))

        iFieldList = makeFieldList(iFC)
        if (len(iFieldList) == 0):
            printit(" No fields to change, Skipping...")
            continue

        rows = arcpy.UpdateCursor(iFC)

        changes = 0
        printit(" Changing Rows")
        for row in rows:
            iChange = 0
            for iField in iFieldList:
                iValue = str(row.getValue(iField))
                newValue = iValue

                if valueDict.has_key(iValue):
                    newValue = valueDict[iValue]
                else:
                    if not (caseSensitive):
                        if valueDict.has_key(iValue.lower()):
                            newValue = valueDict[iValue.lower()]

                if (newValue <> iValue):
                    printit("CHANGE {0}".format(newValue))
                    row.setValue(iField,newValue)
                    iChange+=1

            if (iChange > 0):
                changes+=1
                rows.updateRow(row)
        printit(" Made {0} changes".format(changes))
        del row
        del rows

    printit("Main")

if (initialQC()==True):
    main()

printit("Done")

Friday Fave: Custom Maps App for Android

Fri, 18 Apr 2014 06:11:58 -0500

I admit, I love picking up freebie maps. Whether it is from the front desk of a hotel or from the bicycle shop, there is a certain appeal to seeing what people put on maps. I have maps organic orchards, breweries, Minnesota authors, rails to trails, zoos, fictional places, race maps, and a variety of other things that someone felt the need to cartographize.

(http://thefriends.org/wp-content/uploads/2012/12/mn_writers_on_the_map_web_download.pdf)

http://thefriends.org/

So, with all these paper maps lying around, I was thrilled to find Custom Maps, a free app on Google Play.

This app allows you to take a picture (or use an image on your device) and georeference it. You can then view your location on the map. I’ve tried it with a few maps and have been happy with the results.

Similar to georeferencing in a desktop application, you select points on the map and then the same points on a control. You need at lest two or three points. After doing it a couple of times, the process becomes pretty easy and it just takes a minute or two from taking the picture to viewing your current location on the map.

http://www.custommapsapp.com

An example use case scenario for this app is you are lugging your family around at an overwhelming large amusement park and never quite sure where you are. You can take a picture of the map you picked up at the entrance, georeference it, and your phone will then show exactly where you are on that map and where the nearest loo is.

The biggest limitation I’ve seen so far is that there is a limit of 3-5 megapixel size limit on the image. Apparently this is an android limitation on how much memory an app can consume. But if you adjust your camera settings not to exceed this, you should be good.

So far, I’m enjoying this app. The author, Marko Teittinen (a good Finn name), has made the source code open source so I look forward to digging into it in more detail.

Friday Fave: Cartozia Tales

Fri, 11 Apr 2014 06:15:02 -0500

Obviously Cartographers belong in the same category as other superheroes like Superman, Batman, and Spiderman and we finally have a our own comic book to prove it.

Cartozia Tales is a collaborative effort of nine indy artists with two guest artist each issue.

They have an interesting plan, they’ve split the world into nine regions (what’s the name for the ninth of an area, nona-rant?) and the artist will tell a story from a different region each issue. They may build off the region’s story from the previous issue, continue on the story they told the previous issue, and start something fresh.

After being funded by a KickStarter campaign, they’ve shipped three issues already and have at least seven more promised. Think the project definitely deserve’s a look.

Friday Fave: Public Workout Maps

Sat, 05 Apr 2014 05:15:31 -0500

When I first found out about GIS, the first application that came to mind was using it to map my running routes. At that time, I was using paper maps and scraps of paper to measure how far I was running each day. GIS obviously offered a better method.

Almost twenty years later, GPS has become so common place that I think we have four or five devices in our household that have GPS capabilities and measuring my runs has become ridiculously simple.

Given that background, you can understand why I love projects like this project by Nikita Barsukov, who compiled publicly available data from Endomondo and created maps of public running workout tracks for a variety of European Cities, including this sample from Helsinki.

Barsukov.Net

Flowing Data, using a similar concept, generated similar maps for several cities in the United States, including Minneapolis–some areas I know.

http://flowingdata.com/2014/02/05/where-people-run/

I’ve got about six years of running data of my own that I should use to generate a personal running route map but would be a fun project.

arcpy.ListFeatureClasses() Bug

Wed, 02 Apr 2014 18:59:06 -0500

I was recently re-evaluating our back-up procedures and discovered and found a nasty bug with the arcpy’s ListFeatureClasses request. If you have a feature class in a feature dataset with the same name, ListFeatureClasses may not find it or anything else in that feature dataset.

Unfortunately, we recently made our daily backup a python-based system that uses ListFeatureClasses and got bit by this bug.

After discovering missing data in our backups, I reconstructed what happened and found this bug. Below is arcpy code that iterates through the feature datasets in a geodatabase and lists the feature classes:

import arcpy

def copyAll():
    for iFeatureClass in arcpy.ListFeatureClasses():
        print(" Feature Class: {0}".format(iFeatureClass))
    iFeatureClassFull = None

testGDBname = "mgs_sandbox.sde"
arcpy.env.workspace = testGDBname

copyAll()
for iFD in arcpy.ListDatasets("","Feature"):
    print("Feature Dataset {0}:".format(iFD))
    arcpy.env.workspace = testGDBname+"/"+str(iFD)
    copyAll()

And here is a screen shot of the contents of a test enterprise geodatabase, you’ll see it has a feature data set named ‘outcrops’ that has a feature class also named ‘outcrops’ within it:

And the results list only the feature dataset:

But if I rename the feature dataset (e.g. outcrop_fd), the results are what I would hope for:

I found that the feature class does not even need to be within the feature dataset and also the problem does not always occur, I have had the code successfully run in some cases.

Once I confirmed the problem, I did find this thread from almost three years ago that mentions the bug. One poster indicated the same thing occurs in ArcObjects which leads me to think something may not be getting registered right in the sde tables.

I was not able to re-create this using either personal or file geodatabases.

So I adopting the policy of not using the same name for a feature dataset as for a feature class.

ArcMap Field Calculator: Calculating Running Total with arcpy

Thu, 13 Feb 2014 06:06:37 -0600

I had a user with a series of GPS points (that were in chronological order) that they wanted to know the accumulated distance from the start to each point in their shapefile. First, we calculated the distance from each point to the previous point into a field called [DistFt].

Then, we hacked out this quick python function to accumulate the total distance in Arcmap’s Field Calculator:

totalDistance = 0

def accumulateDistance(inDist):
   global totalDistance
   totalDistance += inDist
   return totalDistance

And we called it:

accumulateDistance(!DistFt!)

And we got what we wanted.

ArcMap Maximum Legend Items from a ArcGIS Server service

Tue, 11 Feb 2014 06:02:42 -0600

Recently we took a call from a user who could not see the legend for one of the feature classes in one of our services. (Precambrian Bedrock in http://mgsweb2.mngs.umn.edu/arcgis/services/state/mnbdrkgeology )

After trying some standard things–restarting the service, checking the source .MXD–I turned to The Google Machine and quickly found help from ESRI: http://support.esri.com/zh-cn/knowledgebase/techarticles/detail/33741 .

Turns out the default number of legend items ArcMap will display from an ArcGIS Server map service layer is 100 and we had 102 in the problematic layer. (I plead innocence, blame the geologist for needing that many categories).

The solution was to edit the Windows registry and change the setting for ‘Maximum Legend Count’ from 100 to something higher than 102. After doing that (see path note below) and restarting ArcMap, the legend showed for us.

Path Details
The path turned out to be a little different on Windows 7 than the paths indicated in the help article.
ESRI indicated that the path varied by ArcGIS version:

ArcGIS 9.3.x: HKEY_CURRENT_USER > Software > ESRI > MapServerLayer
ArcGIS 10.0: HKEY_CURRENT_USER > Software > ESRI > Desktop10.0 > ArcMap > Server > MapServerLayer
ArcGIS 10.1: HKEY_CURRENT_USER > Software > ESRI > Desktop10.1 > ArcMap > Server > MapServerLayer

I actually changed this in a number of locations–presumably once for each user profile on my machine. Each followed a pattern something like:

HKEY_USERSS-1-5-23412341234-123412342134-12341234123-1302SoftwareESRIArcMapServerMapServerLayer

Extra Information

Out of cursiousity, I wondered if the 100 item limit was a per service or per layer limitation so I set my limit at 103. Because the service has several other layers, the total number of layers in the legend was about 130. Everything drew OK so it appears the limit applies per layer.

Arcpy: Check if a field exists

Mon, 23 Sep 2013 06:47:36 -0500

I was helping a co-worker who needed to check if a field exists in their arcpy script. Since we were located at their computer, I thought I would just do a quick Google search and pull the code off this blog. Seemed logical since I the original purpose was exactly that—to serve as a handy, public place to store code snippets that I use & that others might find handy.

Anyhow, my Google Search on “Node Dangles field Exists” came up with a 9.3 script to check if field index exists. And I also have a 10.0 version but did not come up with the field exists snippet. So here it is:

def fieldExists(inFeatureClass, inFieldName):
   fieldList = arcpy.ListFields(inFeatureClass)
   for iField in fieldList:
      if iField.name.lower() == inFieldName.lower():
         return True
   return False

ArcMap Field Calculator: Number of parts in multi-part feature

Thu, 05 Sep 2013 14:04:40 -0500

In the last week, I have looked for multi-part features a couple of times. Today, I was looking for multi-part polygons after dealing with the fall-out of a case of Clip Gone Wild as shown below.

I have not found a way to write a query to find these but Field Calculator does allow you to calculate a field’s value to the number of parts.

Using the Python parser, just write the formula (note that case matters): !shape!.partCount

ArcSDE Tip: Finding an ArcSDE instance's port

Tue, 26 Feb 2013 05:25:03 -0600

One of my main tasks right now is to document many of the details of maintaining ArcSDE geodatabases so I anticipate having several blog posts on this topic that are re-writes of documents I am working on. I am presuming that the person will have no ArcSDE experience so I am documenting very detailed information.

Almost all of the ArcSDE commands require that you specify which instance (service/port) the command applies to by using the ‘-i’ parameter.

ArcSDE Instance Parameter

Since we have multiple ArcSDE geodatabases, I like to have a handy-dandy sticky note with all the geodatabases and their respective ports on the side of my monitor.

But when that is not handy and I can never remember the ArcSDE command line syntax to get a list of instances and their ports–I mean remembering ‘sdeservice -o list’ is difficult at my age.

The quickest and most reliable way I’ve found to get the instance number is just to check the properties of your SDE connection file in ArcCatalog, right-click on it and select ‘Connection Properties’.

And the port is right there in the service entry (5164).

I'm Back!

Sun, 24 Feb 2013 11:04:35 -0600

Ok, it has been to long since I last posted and I thank those few people who asked If I had suffered a Segmentation Violation or something.

By means of a brief explanation, I changed jobs last summer, going back to a previous employer in a very different role than what I’ve ever had before. It was a challenging and difficult and one that I had minimal success at but, in the end, didn’t work out. Mostly because I wasn’t a good fit for a variety of reasons including, not surprisingly, geography! I was mostly tele-commuting because the office was two hours away. I really needed to be in the office more than I was and I wasn’t in a position to relocate.

During my time with that employer I was doing less technical stuff and just did not feel the urgency to be blogging.

I’ve left that job (which was very hard to do because I felt like I was abandoning friends) and have a short-term gig with the employer I left in the summer which will be almost entirely technical work, so I hope to crank out some more technical blog posts.

Also, I’m hoping to shift my career focus somewhat. I’ve been somewhat stuck as a desktop guy–developing mostly desktop applications. I’ve wanted to do more web development stuff but never made the switch. I’m hoping to build my web skills in the short-term and have been cramming on topics like Java, HTML5, CSS, Map Server, Geo Server, php, jQuery, and anything else that might enhance my web-abilities.

So expect some changes to the blog and I thank everyone who comes to the blog, while the numbers aren’t huge, I am still surprised at how many people find their way here.

SDEINTERCEPT and SDEINTERCEPTLOC

Wed, 01 Aug 2012 07:17:41 -0500

Awhile ago, I had a ArcSDE problem that required ESRI technical support to help trouble-shoot. The problem was odd but was resolved by rebooting the server.
During the process, though, the support person had me set a couple of environment variables for logging SDE activity on the client machine.

The settings were SDEINTERCEPT and SDEINTERCEPTLOC.

From ESRI’s Help, SDEINTERCEPT specifies what activity to log and SDEINTERCEPTLOC specifies where to save the log files.

I recently deleted the directory I made for the log files but did not remove the variables and I noticed that one of my python scripts reported a weird error (but continued to run, I think). I tracked it back to these variables and realized what I had done.

Googling SDEINTERCEPTLOC did lead me to some helpful information like:

The SDEINTERCEPT blog where Ken posts ArcSDE help.

This ESRI post about troubleshotting geoprocessing problems.

And this ESRI technical article about diagnosing ArcSDE Connections.

Unsupported Arc: 'Rebox'ing or updating the extent of a feature class

Tue, 05 Jun 2012 21:16:28 -0500

I’ve found that sometimes I can not find the answer to a question until I know the answer & then it becomes ridiculously easy to find the answer.

One small annoying thing that I never spent much time was when you delete features from a feature class making it significantly smaller but the envelope does not get re-sized so the zoom extent (still the original extent) is too large. This often happens to use when we convert tables to an XY theme and there are blank records–most of our data shows in Minnesota but there are some in Oklahoma (I think). Once we eliminate or correct the blank records, our data view still pops out to include a large section of the United States even though we only have data in Minnesota.

A long, long time ago, Workstation ArcInfo had a simple command, Rebox, for just this purpose (actually it still does, I just don’t get to use it anymore)–it shrunk the extent to the smallest rectangle required to enclose all the data. Up until today, I thought the request for this feature was completely ignored.

While researching something else, I was digging around in the sde tables and found one, sde.sde_layers, that had the interesting fields, minx, miny, maxx, and maxy. My quick & dangerous test (I performed it on a throw-away feature class in a throw-away geodatabase) gave me the results I wanted–once I loaded the feature class into ArcMap, the extent was a nice, tight rectangle around my features.

Is this a supported way to Rebox the extent? No.

Is it recommend by ESRI or me? No.

Will it screw up your entire geodatabase, making you lose all your data & costing you your job? Probably not but do you want to take that chance?

Will it get the job done? Maybe. But in the process of writing this post, I found two safer ways to go about it. First, the straight-forward, sde command-line way that probably always existed that I never found until today, sdelayer -o alter had an -E option to reset the extent, including the ability to either specify it or have sde calculate it. Ok, that is usable for one person in our organization.

Previously, we had found either a VBA or other tool for doing this but had minimal success with it. Today, I found an ArcGIS 10 Add-In that is suppose to do the same thing. In my experiments (sample size n=1) it worked perfectly. If you need this sort of functionality, I would recommend trying out this Add-In first, if that fails go the sde command line route. Use the direct SQL method at your own risk!

Quick & Dirty python: Converting a text file to audio (.wav)

Thu, 31 May 2012 23:12:28 -0500

This is a bit of a tangent but for some crazy reason, I wanted to convert some text to audio so I could listen to it while I drive. A quick Google search left me without any freeware that could handle the 53 page document–there are some cool websites that do text to mp3 like vozme and YAKiToMe! but they didn’t convert the whole document. I then found pyTTS, a python package that serves as a wrapper to the Microsoft Speech API (SAPI) , which has been in version 5 since 2000. But I didn’t easily find a version of pyTTS for python 2.6. So I decided to see if I could roll my own.

As it turns out, getting python to talk using SAPI is relatively easy. Reading a plain text file can be done in a few lines.

from comtypes.client import CreateObject

infile = "c:/temp/text.txt"

engine = CreateObject("SAPI.SpVoice")

f = open(infile, 'r')
theText = f.read()
f.close()

engine.speak(theText)

And it wasn’t that much more to have it write out a .wav file:

from comtypes.client import CreateObject

engine = CreateObject("SAPI.SpVoice")
stream = CreateObject("SAPI.SpFileStream")

infile = "c:/temp/text.txt"
outfile = "c:/temp/text4.wav"
stream.Open(outfile, SpeechLib.SSFMCreateForWrite)
engine.AudioOutputStream = stream

f = open(infile, 'r')
theText = f.read()
f.close()

engine.speak(theText)

stream.Close()

And with that chunk of code, I was able to convert my 54 page document into a 4 hour long .wav file (over 600 MB) that I used another software package to convert to .mp3 (200 MB). The voice is a bit robotic but not too bad, I just hope the content that I converted (a database specification standard) doesn’t put me to sleep while I drive.

Quick & Dirty arcpy: Autopan ArcMap using arcpy

Wed, 02 May 2012 05:19:40 -0500

Question: How do I get ArcMap to automatically pan through an area.

As I mentioned in a previous post, I recently had the need to have ArcMap automatically pan through a project area. My first attempt was to print a series of data-driven pages (using a fishnet polygon layer as the index) this but that did not accomplish what I needed so I switched to arcpy, which made the task simple enough. Nothing special or tricky about this code, but just did not find it anywhere else.

The one thing to note is that I have a 1 second pause between pans–this was to allow image tiles to download. You will need to adjust the delay to meet your needs. The toolbox and code can also be downloaded.

import sys,arcpy,datetime
inLayer = sys.argv[1]

def printit(inMessage):
    print inMessage
    arcpy.AddMessage(inMessage)

mxd = arcpy.mapping.MapDocument("CURRENT")

arcpy.MakeFeatureLayer_management(inLayer, "indexLayer")
cur=arcpy.SearchCursor("indexLayer")

df = arcpy.mapping.ListDataFrames(mxd)[0]
newExtent = df.extent

iCount = 0
iTotal = (arcpy.GetCount_management("indexLayer").getOutput(0))

for row in cur:
    thisPoly = row.getValue("Shape")
    newExtent.XMin, newExtent.YMin = thisPoly.extent.XMin, thisPoly.extent.YMin
    newExtent.XMax, newExtent.YMax = thisPoly.extent.XMax, thisPoly.extent.YMax
    df.extent = newExtent
    time.sleep(1)
    iCount+=1
    printit("Panned to feature {0} of {1}".format(iCount,iTotal))

del row
del cur

Building a local, permanent cache of Bing Imagery.

Mon, 30 Apr 2012 05:28:15 -0500

I recently had an internal request to capture and store the Bing imagery for an area for future use. The user was interested in some specific images that were taken after a fire, making the ground surface-and certain geological features-much more visible. His concern was that in the future this imagery might get replaced with updated imagery taken when the vegetation has grown back.

Since it is unknown when/how this data might be used by us, we mostly wanted to capture it now & find a way to use it.

While we initiated the process of finding out what agency the data was available through, we also came up with a quick & dirty way to download the data.

Since ArcGIS 10 has made the process of loading cached basemap data a trivial process through ArcGIS Online, I have not used it much since taking a quick first look at it in 2010.

After removing my old, forgotten version and installing the latest, shiniest version of ArcBruTile, I verified it was able to display the imagery we wanted. ArcBruTile can be used to ‘display maps from OpenStreetMap, Bing, SpatialCloud, MapQuest, Europa Technologies, VR-TheWorld Online, Mapbox, Stamen Design and others in ArcGIS Desktop’. The cool thing for me was it builds a local cache in an open format–a bunch of jpeg files in a directory structure. All I had to do was clear the cache, and pan through the area of interest at the desired scale.

I could either spend many long boring hours manually panning, go through the process of renting a chimp to do it for me, or write some code to do it for me. I ended up making a fishnet of the area of interest and wrote a python script to pan through the area (to be posted).

After I had the images, I ended up build a Mosaic Dataset and added the images to it. The last trick I that I had to figure out–and really I just found in it ArcForums–was how to create a mosaic dataset using relative paths. Can not be done, at least in 10, but by using the ‘Repair…’ option to reset paths, you can make the mosaic dataset portable enough that if the reason you wanted to use relative paths was so you could move the data around or to other machines, you can. Just need to repair the paths.

So now, until we can actually track down the original data, we at least have a usable, archive of the imagery we wanted to preserve and have a way to access it in the field in a non-connected environment.

ArcMap Field Calculator: Identifying Unique Cases, Single Field

Sat, 28 Apr 2012 05:55:34 -0500

Seems like a lot of people are finding the ArcMap Field Calculator examples that I have posted useful so I will make an effort to post more of them. Most posts are generated after I do something and think that others might want to know how to do it. (Or so I can go back and remember how I did something without re-inventing it).

Something I did today was create a field (!Case!) and then populated this with a unique identifier for each different value (case) that occurred in a different field (!Feature!).

Note: python’s index statement is a 0-based search so the first case will have the value 0, the second will have 1, and so on. If you want to start the results at 1, you can make the last line: ‘return caseList.index(inValue) + 1’.

The basic structure for this is shown:

caseList = [ ]

def returnCase(inValue):
   global caseList

   if not inValue in caseList:
      caseList.append(inValue)

   return caseList.index(inValue)

ArcMap Field Calculator: Identifying Unique Cases, Multiple Fields

Mon, 23 Apr 2012 06:06:25 -0500

You may have noticed that this post–ArcMap Field Calculator: Identifying Unique Cases, Single Field–specifies ‘Single Field’. Yes, that was my version of a cliff-hanger post.

The basic structure I listed in that post can be expanded on to satisfy your needs. The example in my earlier post was case sensitive for example, you could modify it so it treats ‘a’ the same as ‘A’.

Today’s example groups records into different cases based off the values of two fields, !county_c! and !feature! and required only minor modifications.

The calling line was modified from:

From:

   returnCase(!feature!)

To:

   returnCase(!county_c!,!feature!)

to accommodate passing both values.

The function definition likewise was modified to accept two values, this:

   def returnCase(inValue1):

to:

  def returnCase(inValue1, inValue2)

And this line was added, creating a list from the two values passed in:

   inValue = [inValue1, inValue2]

**(Note: The same results could have be achieved by using the original function by creating the list in the calling statement: returnCase([!county_c!,!feature!] )
**

caseList = [ ]

def returnCase(inValue1, inValue2):
   inValue = [inValue1, inValue2]
   global caseList

   if not inValue in caseList:
      caseList.append(inValue)

   return caseList.index(inValue)

ArcMap Field Calculator: Beware of Integer Division!

Mon, 02 Apr 2012 05:34:54 -0500

Apparently, if you post one time about ArcMap field calculator, you’re bound to get additional questions. After my recent post about using field calculator to convert text values to numeric, someone asked about a problem they were having with another calculation they were having.

The underlying problem was that python 2.6, which is installed with ArcGIS 10, uses integer division when both the numerator and denominator are integers. The result of integer division is an integer rounded towards negative infinity.

If you’re not aware of this (or forget) then you open yourself up to some unexpected results–they can be especially hard to catch if you are using it within a larger block of code.

In this example, I’m calculating a percentage but the result is 0 for all the records because of the rounding.

The easiest thing to do in this simple example is just convert one of the two value to a non-integer value. This can be done by multiplying by 1.0 (not just ‘1’, you need to include the ‘.0’) which is a float datatype. Multiplying one of the values by a float makes that value a float and integer division no longer applies & we end up with a happy GISer.

Another option if you are using a Codeblock is to include ‘from __future__ import division’ in your code block. Python is slowly moving away from using integer division and the _\_future__ module overrides the default behavior.

ArcMap Field Calculator: Text to Double

Wed, 28 Mar 2012 05:23:32 -0500

Received a request yesterday asking how to use the ArcMap Calculator to copy values from a Text field to a Double field using python syntax. As any good blogger would do, I immediately thought, ‘Awesome! Someone’s question is the perfect topic for a new blog post’.

The python parser is actually pretty good at casting values on the fly so if the values in your text field (!Day! in my example) are valid values that can be converted to a Double value, it is as simple as just setting the formula to be the text field. In my example case, I wanted to copy the value from !Day! to !DecDay! so I set the formula to be DecDay = !Day!.

That should work fine if you have clean values in your text field. In the example above, you might notice I had a selected set of 3 records that all had numeric values in the !Day! field. When I included the fourth row, which does not have a numeric value in the text field, I get this error message (‘There was a failure during processing, check the Geoprocessing Results window for details.’ when I use the same formula. Time to add in an error exception.

For more advanced logic, the Field Calculator dialog allows you to use a python function if you check on the ‘Show Codeblock’ option. In the ‘Pre-Logic Script Code’ area (Seriously, who at ESRI came up with that name?) I entered the following function. If the value in my text field (!Day!) can be cast to a number of type float, that value is returned. If the cast is unsuccessful (IE the value in !Day! is not a number), then I return -99.

def toNum(inValue):
   try:
      outValue = float(inValue)
      return outValue
   except:
      return -99

Then in the formula portion of the dialog, I call the function, passing the value in the !Day! field: DecDay = toNum(!Day!).

Now, if you would prefer not to set all the records with non-numeric values to be -99 or other error value, not return anything. To do this, I replaced the ‘return -99’ in the original function with a filler line (‘doNothing = 4’) since the try block needs an non-empty except clause.

def toNum(inValue):
   try:
      outValue = float(inValue)
      return outValue
   except:
      doNothing = 4

And that should leave the values in the double field unscathed in your records with non-numeric values in the text field.

Shameless Plug: Check out my other blog posts on using ArcMap’s Field Calculator to calculate geometry and converting a date value to an 8 digit numeric value.

Domain Sorter Add-In Version 1.1

Wed, 21 Mar 2012 18:19:03 -0500

Almost a year ago, I updated ERSI’s Domain Sort code for VB 6 to work with ArcGIS 10. Recently, I had a comment that this Add-In caused ArcCatalog to explode if you had an open OLE connection. When I tested it, it turned out the reports were accurate.

I got around to adding in a Try-Catch around the offending chunk of code & it is now better than ever. You can download just the Add-In or the Add-In with source code or get it from ESRI’s ArcGIS Resource Center.

Quicker and Cleaner python recursive folder search

Wed, 14 Mar 2012 17:28:59 -0500

As contributor of the day, Jason Scheirer, pointed out, python has a simple, direct way to browse through the subdirectories of a directory–os.walk

Here is a bare-bones example of using it to print out the subdirectories in a path. The files variable of the 3-tuple is a list of files similar to the dirs variable that I loop through.

Thanks Jason for pointing out something I missed.

import os

theDir = 'c:/temp/'

for root, dirs, files in os.walk(theDir,True,None):
    for idir in dirs:
        print "     directory:   {0}/{1}".format(root,idir)

Quick & Dirty python recursive folder search

Tue, 13 Mar 2012 15:55:55 -0500

Someone asked how to have python recursively search a folder structure. There may be a better way but this is how I typically do it–it basically starts with one directory and loops through the contents compiling a list of sub-directories as it goes through the contents.

import glob, os

theDir = 'c:/temp/'
theDirList = []
theDirList.append(theDir)

while len(theDirList)> 0:
   newDirList = []
   for iDir in theDirList:
      print iDir
      for iFile in glob.glob(iDir+"/*"):
         if (os.path.isdir(iFile)):
            newDirList.append(iFile)

   theDirList = newDirList

ArcMap Field Calculator: ArcPy Date to Decimal Function

Thu, 06 Oct 2011 21:08:35 -0500

One of the standards in our databases is to store dates as 8-digit integer values in the format of yyyymmdd. This requires us to occasionally convert values from date fields into this format.

We can do this in the ArcMap Field Calculator using this arcpy function:

def datetodouble(inNum):
   splitList = str(inNum).split("/")
   return  splitList [2]  +("0"+ splitList [0])[-2:]  +("0"+ splitList [1])[-2:]

Data Extration Failed Reverse Proxy Time-out

Thu, 18 Aug 2011 19:55:47 -0500

Testing one of our geodata services, we discovered that it allowed us to extract a portion of our feature class but when we tried to extract the entire data set, we received this Data Extraction error: Data extraction failed. Proxy or Gateway Server did not allow the URL. Check with your LAN administrator that Proxy or Gateway server is configured to allow the URL.

The fact that I was able to extract a portion of the data and I could see the entire geodatabase get made and zipped led me to believe it was more of time-out issue.

Reading through this thread at ArcForum led to some good information. But Thomas' comment that he was using ‘IIS7 for my reverse proxy server’ and had to change one more setting led me to the solution. In Server Manager, the default Proxy Time-Out is set at 30 seconds by default. I bumped that up, 60 seconds shown below but I ended up going to 300 seconds and the problem was resolved.

44.852994-93.55073

ArcMap Field Calculator: Create a Unique ID

Thu, 28 Jul 2011 20:08:13 -0500

One of the common functions I have to do is assign each record in a feature class with a unique identifier–normally just a sequential number from 1 to N. In ArcView 3.x, the formula was simply ‘rec + 1’ if I wanted to start with the number 1.

In ArcGIS, the process got a little more complex–you had to write a little VBA in Field Calculator as described by ESRI.

While this option still exists in ArcGIS 10, I believe it will disappear when 10.1 comes out and VBA support is completely eliminated. But it is doable using Python which will continue to be supported.

Googling around, I did not find an exact answer but Dave Verbyla, Professor of GIS/Remote Sensing at the University of Alaska has a posted some samples that served as a good starting point.

In the Pre-Logic Script Code box, I declare a variable (counter) and a function. Then in the formula, I call the function.

counter = 0
def uniqueID():
  global counter
  counter += 1
  return counter

While composing this post, I actually wanted a concatenated value; ‘OC’ plus an 8 character numeric sequential number starting at OC00000001 so the actual code is shown below:

44.852994-93.55073

ArcGIS Add-In Custom Mouse Cursor

Tue, 19 Jul 2011 20:47:03 -0500

I was working on a project and wanted my own custom mouse cursor and did not easily find a way to make your own in ESRI’s instructions. But, once you know how to do it, it is pretty easy. In Visual Studio, Add a New Item:

Add a Cursor File:

You can edit your cursor with the editor program in Visual Studio. Once you satisfied with how it looks, make sure that the Build Action on the cursor is ‘Embedded Resource’.

Then you can set your cursor with two lines of code. Note that my cursor is in my QDI.QdiAddIn Namespace:

Dim pCursorStream As System.IO.Stream = Me.GetType.Assembly.GetManifestResourceStream("QDI.QdiAddIn.NewCursor.cur")
MyBase.Cursor = New System.Windows.Forms.Cursor(pCursorStream)

44.852994-93.55073

Debugging a Python Scheduled Task

Mon, 11 Jul 2011 18:11:13 -0500

I have been working on a python script that I want (NEED) to run as a scheduled task on a remote machine. I got to the point that the script did exactly what I needed when I was interactively running it in a Windows session but had problems when running it as a scheduled task. The debugging process was cumbersome–make a change, schedule a task to run it, log out of the machine, and wait. The log back in and repeat the process.

That got old.

So I wrote a script (tester.py) that calls any other python scripts in the same directory that (1) start with ‘test_’ and (2) there is not a corresponding file with the same base name and ‘.start’ extension. It would launch ‘test_BaBing.py’ as long as there is not a ‘test_BaBing.start’ in the same directory. Tester.py continued to run, looping every 60 seconds, until tester.stop exists.

This made the process easier because I could work on my local machine, editing the problematic script, saving changes and within 60 seconds it would be launched on the remote machine. I could view the results, make additional edits, delete the .start file and it would launch again within 60 seconds.

Within a couple minutes I was able to determine the problem (path related) and fix it.

Happy programmer.

I would recommend using this only while debugging a script–routinely running it could be a security risk since someone could copy a destructive python script into the directory and this would run it.

Download: tester.py

import sys, string, os
import glob
import datetime, shutil
import time, inspect
import getpass

totalstarttime = datetime.datetime.now()

dateString = datetime.date.today().strftime("%Y%m%d_")+datetime.datetime.now().strftime("%H%M%S") #datetime.date.today().strftime("%Y%m%d")
debugfile = inspect.getfile(inspect.currentframe()).replace(".py","_"+dateString+"_Debug.txt")
stopfile = inspect.getfile(inspect.currentframe()).replace(".py",".stop")
newdebugfile = False

codeDir = os.path.dirname(inspect.getfile(inspect.currentframe())).replace("\","/")

def printit(inText):
    global newdebugfile

    print inText

    if os.path.exists(debugfile):
        if (newdebugfile == False):
            tmpfile = open(debugfile,"w")
            newdebugfile = True
        else:
            tmpfile = open(debugfile,"a")
    else:
        tmpfile = open(debugfile,"w")

    tmpfile.write(inText)
    tmpfile.write("n")
    tmpfile.close()
    newdebugfile = True

stopFileExists = False
printit("Code Directory: "+codeDir)
printit("Starting at: "+datetime.date.today().strftime("%Y-%m-%d_")+datetime.datetime.now().strftime("%H:%M:%S"))
printit("Stopfile : "+stopfile+"/n")
while (stopFileExists == False):
    for iFile in glob.glob(codeDir+"/test_*.py"):

        thisStartfile = iFile.replace(".py",".start")

        if not (os.path.exists(thisStartfile)):
            printit ("Launching: "+iFile)
            iTmpfile = open(thisStartfile,"w")
            iTmpfile.write("started")
            iTmpfile.close()
            os.system("Start "+iFile)

    if (os.path.exists(stopfile)):
        stopFileExists = True
    else:
        time.sleep(60)

    printit("nEnd of Loop: "+datetime.date.today().strftime("%Y-%m-%d_")+datetime.datetime.now().strftime("%H:%M:%S")+"n")    

printit("Done!")

44.852994-93.55073

Checking to see if a Field Index Exists Using Arcpy (ArGIS 10.0) redux

Tue, 05 Jul 2011 12:53:34 -0500

I’ve previously posted python code to check if a field index exists for both ArcGIs 9.3 and ArcGIS 10.0.

Recently I have been working on a process that was using this code but it was not working because it looks for an index with a specific name. It was not working in this case because the name of the indexes was getting incremented as they were being created. For example, I was building an index on the table C5ST, field RelateId ([C5IX].[Relateid]) named I_C5IX_RelateId. That worked fine until we switched our process so now we keep multiple versions of some tables, each with a date-based suffix.

We now have tables name C5St_20110625 and C5St_20110626–the Index-name scheme, however was still creating I_C5IX_RelateId and it worked great on the first one. But when it created the second one, even on a different table, it was automatically name I_C5IX_RelateId_2 even though the name I_C5IX_RelateId was used when trying to create the index.

Before generating relates, our code checks to see if the key fields are indexed, and if they are not, builds an index. Because of the naming situation, multiple, duplicate indexes were being created. Probably not too harmful but it is a little messy.

So I re-wrote the code so that you pass the function the table name and field name that you want to check and it checks to see if there is an index existing for that field and return a Boolean. The one little wrinkle I put in is to account for indexes that span multiple fields–the ' if (iIndex.fields[0].Name.upper() == fieldname.upper()):' statement is checking the index to see if it is on a single field or multiple fields.

def fieldHasIndex(tablename,fieldname):
if not arcpy.Exists(tablename):
return False

tabledescription = arcpy.Describe(tablename)

for iIndex in tabledescription.indexes:
if (len(iIndex.fields)==1):
if (iIndex.fields[0].Name.upper() == fieldname.upper()):
return True

return False

44.852994-93.55073

Feature classes and Tables with names starting with 'nd_'

Mon, 23 May 2011 15:55:24 -0500

Random luck me to discovering a bug related to feature classes whose names start with ‘nd_'. It appears that you are allowed to create feature classes starting with ‘nd_’ but ArcCatalog will not display them. Further research shows this behavior also occurs for table and for ArcSDE (PostGres) geodatabases, personal geodatabase, and file geodatabases–I am using ArcCatalog 10.0.

I first noticed something odd was occurring while importing a series of shapefiles into a geodatabases. After importing 15 shapefiles, I only had thirteen feature classes despite receiving no errors during the process. The two shapefiles that failed to import were named ND_oil_and_gas.shp and ND_Bendix_Study.shp. Subsequent attempts to import them individually returned an error ‘Invalid Target Name’.

I discovered in pgAdmin III (Postgres SDE Geodatabase) that the table existed and there was an entry in sde.sde_layers for the feature class but ArcCatalog refused to show it.

I used some un-supported methods to try to resolve the problem and despite some sweating, I failed to find a way to get ArcCatalog to display these feature classes. I did, however, at least found a way to delete them–arcpy can detect that the feature classes exists so it is able to delete them.

At least by deleting them, I can prevent leaving ‘invisible’ feature classes from hanging out in my geodatabase.

I suspect the problems stems from how ESRI has implemented the Network dataset table-naming structure –dirty areas are stored in tables named _nd__dirtyareas and nd__dirtyobjects. Possibly the developer working on the ArcCatalog GUI ended up suppressing showing feature classes and tables whose names start with ‘nd’.

And, just for posterity’s sake, here is a python code snippet listing the feature classes in a workspace:

import arcpy
 
arcpy.env.workspace = 'c:/temp/_nd/F.gdb'
 
print arcpy.env.workspace  
for fc in arcpy.ListFeatureClasses():  
   print fc
 
print 'Done!'

44.852994-93.55073

Renaming Raster Dataset and arcpy.Exists()

Tue, 03 May 2011 09:33:48 -0500

Discovered something today. I was working on an arcpy script that copies a raster dataset from a file geodatabase into a Postgres SDE geodatabase and then does some boring routine tasks–building stats, creating a mosaic dataset, adding the raster to the mosaic dataset and making a couple referenced mosaic datasets.

It sometimes has trouble with the initial step of uploading the raster because of the sheer size of if (1m elevation raster for counties) and it failed today on one. It failed today so I used the ArcCatalog GUI to copy the raster and renamed it.

I then proceeded to run launch my script. Before each step, I use arcpy.Exists extensively to check to see if various items exist before I attempt to create them. It was continuously reporting that my raster set did not exist even though I could see it in ArcCatalog.

Finally, I realized that I needed to close ArcCatalog before arcpy recognized the fact I had renamed something. To note, I was running arcpy from a separate PythonWin window, not from the ArcCatalog session I had renamed the raster dataset with.

Once I closed ArcCatalog, arcpy recognized the renaming and life was good.

I’m also suspicious now about a problem I often have running statistics on my rasters. The ArcTool reports no errors when I create them but for some reason the raster does not show that it has statistics afterwards. I normally have multiple ArcApplication sessions open and now suspect that perhaps this problem is due to sessions not letting go of the connection. Stay tuned for further developments on this.

44.852994-93.55073

Sorting a Coded-Value Domain Add-In (ArcGIS 10)

Tue, 03 May 2011 03:33:01 -0500

I am working on an data-entry application to edit feature classes that contain several coded-value-domains. The problem with some of the domains, however, is that some entries have been added after the initial creation. So the first 25 entries are in alphabetical order and there are some stragglers at the end that are in the order they were appended.

This can be confusing for users–they go to select ‘Milli Vanilli’ and look between ‘Madonna’ and ‘Motley Crue’ but can not find their favorite band there–they have to go to the end of the list to find their selection.

In the past, I have gone through the tedious process of exporting the domain to a table, sorting the table, removing the domain from the necessary field(s), deleting the domain, re-importing the table back in a new domain and finally re-applying the domain to the necessary field(s). Let’s just say I didn’t do this until someone asked a few times and I didn’t have anything more exciting–like a root canal–I could busy myself with.

But this new application contains more domains than any of other datasets so it was time to find a better solution. ESRI does have a Domain Sort Developer Sample. It, however, did not play nice with ArcGIS 10.

So I went ahead and update it from VB 6 to VB.Net/ArcObjects 10. I made an Add-In that can be installed by downloading the .esriaddin file and double-clicking on it. The source code is also available.

This will add an ArcCatalog Toolbar that can be added by going to Customize-Toolbars-Domain Sorter Toolbar.

This will add a toolbar with one button. The button enables whenever you select a geodatabase with at least one coded-value domain.

This brings up a Windows form that lets you sort any domain by either the code or description, ascending or descending. Once you hit ‘OK’ it re-sorts your domain.

The only problem I have had is that only the owner of a domain is allowed to edit it on an SDE geodatabase.

But other than that, the button allows you to easily keep your domains sorted.

http://edndoc.esri.com/arcobjects/9.2/CPP_VB6_VBA_VCPP_Doc/COM_Samples_Docs/Geodatabase/Schema_Creation_and_Management/Sort_a_domain/e826c5a8-9740-4f0b-86b6-d3b834735574.htm

44.852994-93.55073

Quick & Dirty arcpy: Batch Splitting Polylines to a Specific Length

Sun, 01 May 2011 05:01:50 -0500

For some odd reason, I wanted to split all the arcs in a polyline feature class to a specific length–if a specific feature was longer than the target length, it would become two or more separate polyline records.

Here is the bare-bones script that copies an existing feature class into a new feature class then processes each record, splitting it into multiple records if the polyline is longer than the user-specified tolerance. Some cautionary notes:

This is Quick & Dirty code–minimal error catching or documentation.
I basically tested this against one feature class (the one I wanted to split) once I got it to work, I quit.
There is some rounding error–features may be a tad bit off (a few ten-thousandths of a unit).
I did not test against multi-part features.
The tolerance is the native units of the data–if your data is in meters but you want to split the polylines every mile, enter 1,609.344.

I have included both a toolbox file (.tbx) and python script (.py). After loading the toolbox, you’ll have to change the Source of the script by right-clicking on it, selecting the Source tab, and then navigating to the .py file.

Here is the code for the Googlebots, but you are better off just downloading it.

import arcpy
import sys, math

def printit(inMessage):
&nbsp;&nbsp;&nbsp; print inMessage
&nbsp;&nbsp;&nbsp; arcpy.AddMessage(inMessage)

if len(sys.argv) > 1:
&nbsp;&nbsp;&nbsp; inFC = sys.argv[1]
&nbsp;&nbsp;&nbsp; outFC = sys.argv[2]
&nbsp;&nbsp;&nbsp; alongDistin = sys.argv[3]
&nbsp;&nbsp;&nbsp; alongDist = float(alongDistin)
else:
&nbsp;&nbsp;&nbsp; inFC = "C:/temp/asdfasdf.mdb/jkl"
&nbsp;&nbsp;&nbsp; OutDir = "C:/temp/asdfasdf.mdb"
&nbsp;&nbsp;&nbsp; outFCName = "jkl2d"
&nbsp;&nbsp;&nbsp; outFC = OutDir+"/"+outFCName
&nbsp;&nbsp;&nbsp; alongDist = 1000

if (arcpy.Exists(inFC)):
&nbsp;&nbsp;&nbsp; print(inFC+" does exist")
else:
&nbsp;&nbsp;&nbsp; print("Cancelling, "+inFC+" does not exist")
&nbsp;&nbsp;&nbsp; sys.exit(0)

def distPoint(p1, p2):
&nbsp;&nbsp;&nbsp; calc1 = p1.X - p2.X
&nbsp;&nbsp;&nbsp; calc2 = p1.Y - p2.Y

&nbsp;&nbsp;&nbsp; return math.sqrt((calc1**2)+(calc2**2))

def midpoint(prevpoint,nextpoint,targetDist,totalDist):
&nbsp;&nbsp;&nbsp; newX = prevpoint.X + ((nextpoint.X - prevpoint.X) * (targetDist/totalDist))
&nbsp;&nbsp;&nbsp; newY = prevpoint.Y + ((nextpoint.Y - prevpoint.Y) * (targetDist/totalDist))
&nbsp;&nbsp;&nbsp; return arcpy.Point(newX, newY)

def splitShape(feat,splitDist):
&nbsp;&nbsp;&nbsp; # Count the number of points in the current multipart feature
&nbsp;&nbsp;&nbsp; #
&nbsp;&nbsp;&nbsp; partcount = feat.partCount
&nbsp;&nbsp;&nbsp; partnum = 0
&nbsp;&nbsp;&nbsp; # Enter while loop for each part in the feature (if a singlepart feature
&nbsp;&nbsp;&nbsp; # this will occur only once)
&nbsp;&nbsp;&nbsp; #
&nbsp;&nbsp;&nbsp; lineArray = arcpy.Array()

&nbsp;&nbsp;&nbsp; while partnum < partcount:
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; # Print the part number
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; #
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; #print "Part " + str(partnum) + ":"
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; part = feat.getPart(partnum)
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; #print part.count

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; totalDist = 0

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; pnt = part.next()
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; pntcount = 0

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; prevpoint = None
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; shapelist = []

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; # Enter while loop for each vertex
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; #
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; while pnt:

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; if not (prevpoint is None):
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; thisDist = distPoint(prevpoint,pnt)
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; maxAdditionalDist = splitDist - totalDist

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; print thisDist, totalDist, maxAdditionalDist

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; if (totalDist+thisDist)> splitDist:
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; while(totalDist+thisDist) > splitDist:
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; maxAdditionalDist = splitDist - totalDist
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; #print thisDist, totalDist, maxAdditionalDist
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; newpoint = midpoint(prevpoint,pnt,maxAdditionalDist,thisDist)
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; lineArray.add(newpoint)
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; shapelist.append(lineArray)

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; lineArray = arcpy.Array()
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; lineArray.add(newpoint)
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; prevpoint = newpoint
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; thisDist = distPoint(prevpoint,pnt)
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; totalDist = 0

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; lineArray.add(pnt)
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; totalDist+=thisDist
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; else:
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; totalDist+=thisDist
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; lineArray.add(pnt)
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; #shapelist.append(lineArray)
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; else:
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; lineArray.add(pnt)
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; totalDist = 0

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; prevpoint = pnt&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &nbsp;
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; pntcount += 1

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; pnt = part.next()

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; # If pnt is null, either the part is finished or there is an
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; #&nbsp;&nbsp; interior ring
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; #
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; if not pnt:
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; pnt = part.next()
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; if pnt:
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; print "Interior Ring:"
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; partnum += 1

&nbsp;&nbsp;&nbsp; if (lineArray.count > 1):
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; shapelist.append(lineArray)

&nbsp;&nbsp;&nbsp; return shapelist

if arcpy.Exists(outFC):
&nbsp;&nbsp;&nbsp; arcpy.Delete_management(outFC)

arcpy.Copy_management(inFC,outFC)

#origDesc = arcpy.Describe(inFC)
#sR = origDesc.spatialReference

#revDesc = arcpy.Describe(outFC)
#revDesc.ShapeFieldName

deleterows = arcpy.UpdateCursor(outFC)
for iDRow in deleterows:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &nbsp;
&nbsp;&nbsp;&nbsp;&nbsp; deleterows.deleteRow(iDRow)

del iDRow
del deleterows

inputRows = arcpy.SearchCursor(inFC)
outputRows = arcpy.InsertCursor(outFC)
fields = arcpy.ListFields(inFC)

numRecords = int(arcpy.GetCount_management(inFC).getOutput(0))
OnePercentThreshold = numRecords // 100

printit(numRecords)

iCounter = 0
iCounter2 = 0

for iInRow in inputRows:
&nbsp;&nbsp;&nbsp; inGeom = iInRow.shape
&nbsp;&nbsp;&nbsp; iCounter+=1
&nbsp;&nbsp;&nbsp; iCounter2+=1&nbsp;&nbsp; &nbsp;
&nbsp;&nbsp;&nbsp; if (iCounter2 > (OnePercentThreshold+0)):
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; printit("Processing Record "+str(iCounter) + " of "+ str(numRecords))
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; iCounter2=0

&nbsp;&nbsp;&nbsp; if (inGeom.length > alongDist):
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; shapeList = splitShape(iInRow.shape,alongDist)

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; for itmp in shapeList:
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; newRow = outputRows.newRow()
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; for ifield in fields:
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; if (ifield.editable):
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; newRow.setValue(ifield.name,iInRow.getValue(ifield.name))
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; newRow.shape = itmp
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; outputRows.insertRow(newRow)
&nbsp;&nbsp;&nbsp; else:
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; outputRows.insertRow(iInRow)

del inputRows
del outputRows

printit("Done!")

44.852994-93.55073

Change Detector arcpy Script

Fri, 08 Apr 2011 10:48:59 -0500

During a process I was working on, I needed to compare a feature class before and after some edits. I did not quickly find anything in ArcToolbox but searching ArcResources led me to Change Detector script by Bruce Harold. After making a couple of tweaks–for some reason in one of my feature classes, the Shape field had an upper case ‘S’ and in the other it was a lower case ’s'. I also discovered that it needs to export to the same format (personal geodatabase, file geodatabase, shapefile) as the source data (or at least one that uses the same field name deliminator).

After minor adjustments, though, it worked like a charm. I’ll be submitting the changes I made to Bruce and let him incorporate the changes into the official code.

FOLLOW-UP: Mr. Harold quickly responded to my email & made the change (although I haven’t checked it). Way to go Bruce! Thanks for a handy script.

44.852994-93.55073

ArcIdeas: Scale-Dependant Settings

Fri, 25 Mar 2011 11:29:36 -0500

Someone mentioned an idea on ArcIdeas for making various display settings on a feature classes scale-dependent. Right now some of that can be accomplished by loading a feature classes multiple times, adjusting the settings, and setting the visible range. Working more and more in ArcGIS Server, I can see the value of increased scale-dependent settings.

I’m not sure how rapidly ESRI takes ‘Ideas’ into consideration but if you feel like it would benefit you, why not promote this idea: Scale Range, SQL Query and Symbology Rendering in ArcMap.

Have a good weekend, all!

Quick & Dirty arcpy: Field Listings

Thu, 17 Mar 2011 17:03:39 -0500

I have to often get a table structure for a feature class or table into either a spreadsheet or word processing document. There might be an easy way to do this in ArcGIS 10 but I haven’t found it. So, as is my nature, I decided to roll my own.

This is a bare-bones script that iterates through the fields, printing the field name, type, width, and precision. There are three optional features to it:

You can choose to have it list the domain, if there is one, on each field.
You can have it write to a text file (otherwise you can just copy & paste the results from the results window).
You can have it count the number of populated records. This can take a long time if working with a large dataset. Also note that my logic for determining what constitutes being populated may not be what you need but the structure is there. I also do not account for all field types, if the field is of a type I have not account for, the code will return -999.

To use the script from ArcToolbox, you need to pass it four parameters, their Names, type, whether they are input or output, and whether they are required or optional are:

featureclass, Table, Input, Required
includedomainstring, Boolean, Input, Required (controls whether or not domains are exported)
doCountsRespone, Boolean, Input, Required (controls whether or not you want to get the number of populated records. (Your definition of populated may vary from my code)
outputFile, File, Output, Optional (optional output file to write)

Here is the code, but you are better off just downloading it since I haven’t figured out a good way to have WordPress play nicely with python’s indenting.

# Name: ListFields-arcpy.py
#
# Purpose: Lists the fields, their type, width, and precision
# Can either have it export it to a CSV file or copy
# and paste from the results window.
#
# To use, create a tool from the script and add 3 parameters:
#  1) Table, Input, Required
#  2) Boolean, Input, Required (controls whether or not domains are exported)
#  3) Boolean, Input, Rekquired (controls whether or not you want to get the number of
#  Populated records.&nbsp; (Your defintion of populated may vary from my code)
#  4) File, Output, Optional (optional output file to write)
#
#

import arcpy,sys,os

def printit(inMessage):
 print inMessage
 arcpy.AddMessage(inMessage)

if len(sys.argv) > 4:
 featureclass = sys.argv[1]
 includedomainstring = sys.argv[2]
 doCountsRespone = sys.argv[3]
 outputFile = sys.argv[4]
else:
 featureclass = "C:/temp/before.shp"
 includedomainstring = "false"
 doCountsRespone = "true"
 outputFile = "C:/temp/before.csv"

if (outputFile == ""):
 doOutputFile = False
else:
 doOutputFile = True

if (str(doCountsRespone).lower() == "true"):
 doCounts = True
else:
 doCounts = False

if (str(includedomainstring).lower() == "true"):
 includedomain = True
else:
 includedomain = False

lfields=arcpy.ListFields(featureclass)

d = arcpy.Describe(featureclass)
printit("Dataset: "+d.baseName)
printit("Type: "+d.dataType)
printit("Path: "+d.catalogPath)
printit(" ")

tableheaders = 'name,type,width,precision'

if (doCounts == True):
 tableheaders+=",count"

if (includedomain == True):
 tableheaders+=",domain"

if (doOutputFile):
 tmpfile = open(outputFile,"w")
 tmpfile.write(tableheaders)
 tmpfile.write("n")

printit (tableheaders)
for lf in lfields:

 pThisline = lf.name+","+lf.type +","+str(lf.length)+","+str(lf.precision)

 if (doCounts == True):

 rowCount = 0

 #Note that I do not account for all field types
 #Also note that my definition of being populated may vary from yours.
 #I am using -999 as a flag to indicate a field type was not successfully
 #identified.
 if (lf.type == "Double") or (lf.type == "Single")&nbsp; or (lf.type == "Integer") or (lf.type == "SmallInteger"):
  queryString = '"'+lf.name + '" > 0'
  rows = arcpy.SearchCursor(featureclass, queryString, "", "", "")
 elif (lf.type == "String"):
  queryString = '"'+lf.name + '" <> ' + "''"
  rows = arcpy.SearchCursor(featureclass, queryString, "", "", "")
 else:
  rowCount = -999
  #rows = arcpy.SearchCursor(featureclass, "", "", "", "")

 if (rowCount == 0):
  for row in rows:
   rowCount+=1

 pThisline=pThisline+","+str(rowCount)

 if (includedomain == True):
  pThisline=pThisline+","+lf.domain

 printit (pThisline)

 if (doOutputFile):
  tmpfile.write(pThisline)
  tmpfile.write("n")

if (doOutputFile):
 tmpfile.close

44.852994-93.55073

Updating Access ODBC connections using VBA

Wed, 16 Mar 2011 20:06:59 -0500

I don’t do much in Access or VBA but sometimes you inherit things that force you, at least temporarily, to go retro.

We have an Access-based data extraction routine that uses several (21) pass-through queries that use an ODBC connection. The queries are pulling data from an Oracle database maintained by a different agency that enforces a time-out of passwords every three months.

So every three months I have to go through the ODBC connections for these queries and change the password. And before you chastise me for storing the password in the VBA code, I can only say ‘you’re right’ but that’s the way it is done–not ideal but gets the job done and we’re not dealing with overly sensitive data.

Well, my three-month deadline was approaching and I figured I would simplify the process which I found error-prone.

Some searching and object model diagramming scouring and I came up with this barebones solution, of course, your connection string may vary.

Public Sub UpdateConnectStrings()

Dim obj As AccessObject
Dim dbs As Object
Set dbs = Application.CurrentData

Dim i As Integer

Dim newConnectionString As String
newConnectionString = "ODBC;DRIVER={Microsoft ODBC for Oracle};SERVER=vader;UID=luke;PWD=iMufathr;"

Dim comparisionConnectionString As String
comparisionConnectionString = "ODBC;Driver"

For Each obj In dbs.AllQueries

If (obj.Type = acQuery) Then
  For i = 0 To CurrentDb.QueryDefs.Count - 1
    If (CurrentDb.QueryDefs(i).Name = obj.Name) Then
      If (Left(CurrentDb.QueryDefs(i).Connect, Len(comparisionConnectionString)) = comparisionConnectionString) Then
        CurrentDb.QueryDefs(i).Connect = newConnectionString
      End If
    End If
  Next i
End If

Next obj
End Sub

44.852994-93.55073

Using arcpy to List Domains Assigned to Featureclass Fields

Mon, 14 Mar 2011 18:04:36 -0500

I was making an edit (adding leading ‘0’s) to a coded-value domain in an SDE database and realized that my edits were changing the order of the rows of my domain. Rows were moved to the bottom of the list when they were edited.

So I went through the process of converting my domain back to a table, made my edits in Access and exported the rows to a .dbf in the order I wanted them.

I removed the domain from the field I knew it was assigned to but when I tried to delete the domain, I received an error (The domain is used as a default domain).

The Google Machine led me to an ArcForums post by with some python code for listing all the fields with a domain.

I tweaked the original a bit, first because it was only examining feature classes in a feature dataset, not stand-alone feature datasets. And secondly, I updated it to use arcpy. I posted both the 9.3 version and the 10.0 version, but here is a quick look. The key addition is the “listfc('')” line that is the first line of the def listds() module.

import arcpy

min_workspace = r"C:\Users\mranter\AppData\Roaming\ESRI\Desktop10.0\ArcCatalog\min.minstaff.sde"

infgdb=(min_workspace)
arcpy.env.workspace = infgdb

def listfc(inDataset):
   featureclasses = arcpy.ListFeatureClasses("","",inDataset)
   for f in featureclasses:
      print "feature class: ",f

lfields=arcpy.ListFields(f)

for lf in lfields:
   if lf.domain < "":
      print "      domain",f, lf.name, lf.domain

def listds():
   listfc("")

   datasets=arcpy.ListDatasets ("","")
   for d in datasets:
      print "  dataset: ",d

listfc(d)
listds()

44.852994-93.55073

Revoking Public Permissions in PostgreSQL ArcSDE

Fri, 11 Mar 2011 11:11:48 -0600

While banging my head on how to grant access to a referenced mosaic dataset , I did something out of frustration that I normally would not do–I granted ‘public’ access to some data.

Then, after figuring out the problem, I went to revoke public access using ArcCatalog and received this error message:

Error 999999: Error executing function.

The Object being referenced does not exist [ERROR: role ‘public’ does not exist::SQL state: 42704]
Failed to execute (ChangePrivileges).

I actually expected that would occur–ArcSDE isn’t aware of PostgreSQL’s public account.

The solution was to go into pgAdmin II and revoke the privileges there.

44.852994-93.55073

Referenced Mosaic Datasets Permissions on SDE

Wed, 09 Mar 2011 09:08:27 -0600

While I’m enjoying the functionality of the new Referenced Mosaic Dataset that have been introduced in ArcGIS10, something that I stumbled over recently was administering the privileges to a referenced mosaic dataset.

A referenced mosaic dataset is a raster datatype that uses a mosaic dataset as a base. For example, we loaded a series of DEMs into a mosaic dataset and then created a referenced mosaic dataset that does the meters to feet conversion for us. This saved us from having to process each DEM before loading it. We also created a referenced mosaic dataset hillshade from the original meter data.

The problem I came across, and this was more my oversight than anything, but I had a heck of a time trying to grant privileges to the referenced mosaic dataset. The solution was actually very simple–I needed to also grant access to the mosaic dataset that was being used as the base.

That makes some sense, you should not be able to grant access to a dataset (the referenced mosaic dataset) that would inadvertently be exposing data (the base mosaic dataset) that a user does not have access to. This, however, conflicts a bit in that the permissions on individual rasters that make up a mosaic raster dataset are ignored.

ArcMap Field Calculator: Calculating Geometry using arcpy

Tue, 08 Mar 2011 09:30:40 -0600

One of the things I had not gotten around to doing in ArcGIS 10 was figure out how to directly manipulate the geometry of a record using the Field Calculator. When I stumbled upon a bug in the way the Extract Values to Points tool handles Null geometries, I figured it was time to figure it out. Setting the X, Y to 0,0 was sufficient for my needs.

I set the Parser to Python and the formula was simple once I figured out the syntax:

pPoint = arcpy.Point(0,0)

Then, to extend my knowledge, I wanted to see how to calculate the geometry based off of two fields (X and Y). The only real trick is knowing the bracket field names using exclamation points instead of brackets:

arcpy.Point(!X!,!Y!)

So turns out everything it pretty easy and straight-forward. Whew!

Extract Values to Points (Spatial Analyst) Bug

Mon, 07 Mar 2011 08:37:33 -0600

One of the Spatial Analyst tools we often use in ArcGIS is the Extract Values to Points tool. This allows us to take a point file (well locations in our case) and attach a value (elevations) from a raster image (a DEM) to each point.

Today I was running it for the first time against an Image Service we recently published and I received a warning message,‘WARNING 000957: Skipping feature(s) because of NULL or EMPTY geometry’. But the script seemed to run and the final results said ‘Succeeded’ so I thought it was probably fine. But as I double-checked, I realized the results were wonky.

Turns out that I had two records with Null geometry in my point file of 397 records. These two records threw the above error but actually had a value in the [Rastervalu] field. Turns out all 397 records had values. These two records were consecutive–let’s say the 100th and 101st records in my shapefile. What happened is record 100 got the value for record 102, record 101 get the value for 103, record 102 (which has valid geometry) had the value for 104. This pattern, each record having the value for the record 2 place after it, continued until record 396 which had the value for record 397. Record 397 also had the value for 397. So the final three records all had the value for the final record.

What I would have expected would be for the two records with Null Geometry to have null values in the [Rastervalu] field and the rest of the records to have the correct values. Despite the warning, it is very misleading for all the records to end up with a value.

I have a simplified example below. I made a point shapefile with four records. The first, third, and fourth records have valid geometries; the second has Null geometry. The second record ends up with the value for the third record. The third record, has the value for the fourth. The fourth record being the last record, ends up with the last valid value, which was its own.

The results that I would have hoped for would be for the third record to have a Null value.

The way I envision what is occurring behind the scenes is this: the process makes a list (more of a stack in programming terms) of result values as it processed the points but just assumes that every record will return a value so it does not track which value goes with which shape.

When it reached the two null geometries, it threw an error but continued on. It did not add a value for these records to the stack of values–when it comes across records with valid geometry but do not intersect the raster it adds a psuedo-null value of (-9999) to the stack. After it processed all the records it had 395 values in the stack. It then went, one-by-one through the stack and populated the records in the output shapefile, the first record got the first value in the stack, the second record got the second value, the 100th record got the 100th value (which came from the location of the 102nd record) and so on. At the end, the final two records received the last valid value.

This final behavior–using the last valid value–corresponds a bit to a behavior we’ve seen with ArcObjects in general. When iterating through a table, if a field is Null for a specific row, the value from the last non-Null value for that field is often returned.

I’m in the process of submitting a bug to ESRI. I’m not sure if this existed prior to ArcGIS 10.0 (I’m guessing it did) or if it occurs in other processes (I’m guessing it does). I did find out that the ‘Extract Multi Values to Points’ works as expected. I’m guessing it is because unlike the ‘Extract Values to Points’ which creates a new shapefile, this tool appends fields to the existing shapefile and presumably processes records one-by-one without putting the results in a virtual stack. The ‘Extract Multi Values to Points’ tool also does not throw any warnings.

Instantiating Add-In Objects at Load Time

Wed, 02 Mar 2011 15:18:14 -0600

In migrating a toolbar consisting of a button and a couple of tools for use in ArcMap 10, I decided to take advantage of the ease of deployment enabled by add-ins which was introduced in 10.0. So far, I’m loving the functionality.

One thing, however, that I have to figure out is that the controls are not instantiated until they are clicked on. One of the results is that the controls, by default, are enabled. This is not the functionality I wanted.

I found the solution in ESRI’s topic, Advanced add-in concepts (ArcObjects .NET 10 SDK) in the Delayed loading section:

'By default, the assemblies associated with add-in buttons and tools are not loaded until the corresponding item on a toolbar or menu is clicked by the user. This behavior helps conserve application memory and other resources. Since the enabled state of an add-in button or tool is controlled by the OnUpdate method within code, the button or tool will initially appear enabled. If you need tighter control over the initial enabled state of a button or tool, you need to override the default behavior and force the item to load at startup by setting the onDemand Extensible Markup Language (XML) attribute to false.'

The verbiage, ‘setting the onDemand Extensible Markup Language (XML) attribute to false’ was a bit non-specific to me. I guessed right, however, that they were referring to the control definiton in the Command section of Config.esriaddinx. In the example below, I set the onDemand attribute to false by adding the “onDemand=‘false’” tag and the control did, in fact, get instantiated at load time, giving me the ability to disable it.

<Tool id="MGS_QDIAddin_QDIAddTool" class="QDIAddTool" message="QDI Add New Location." caption="QDI Add Tool" tip="QDI Add Tool." category="QDI Add-In Controls" image="ImagesQDIAddTool.png" onDemand="false">

Loading Tiled, Same-Name Data in Batch Mode.

Mon, 28 Feb 2011 12:30:43 -0600

I have been loading existing raster data into a geodatabase to be included in a new Mosaic Dataset–a very cool and useful addition to ArcGIS 10. The most time-consuming part of the process for the human (at least this human) has been getting the names of the rasters right.

Our existing data is organized by tiles with the directory name representing the tile name and then the data within each tile directory having the same name.

For example:
C:\GIS_dataAdamsparcels.shp
C:\GIS_dataBuchetteparcels.shp

This makes batch loading the data less efficient because I end up having to rename the data or else end up with a series of feature classes named parcels, parcels_2, parcels_n.

So I hacked out a quick script that takes an input raster and figures out the final name I want it to have based of the directory name.

First, I used the Copy Raster (In ArcToolbox: Data Management-Raster-Raster Dataset-Copy Raster) and copied on sample to my geodatabase.

Then, I went to the Results Tab (Select Geoprocessing from the Menubar, Geoprocessing-Results) and right-clicked on the Copy Raster result and selected ‘Copy as Python Snippet’.

I then created a new python script and pasted the one line.

I added some imports, accepted a parameter, some string manipulation, and some result outputs and I had a quick & easy script. In added the script in ArcToolbox and now I can right-click on it and run it in Batch mode. I do a quick search in Windows Explorer to get all the rasters I want to run it on and select & drag them to my ArcToolbox Batch Dialog.

Actual code can be downloaded HERE and you don’t need to worry about WordPress messing up the spacing.

import arcpy
import os, sys

inRaster = sys.argv[1] 
basedir = os.path.basename(os.path.dirname(inRaster)).lower()
outRaster = "Database Connections/mgs_lidar.lidar.sde/mgs_lidar.lidar."+basedir

def printit(inMessage):
    print inMessage
    arcpy.AddMessage(inMessage)
    
if not (arcpy.Exists(outRaster)):
    printit ("Importing: "+basedir)
    arcpy.CopyRaster_management(inRaster,outRaster,"#","#","#","NONE","NONE","#")
else:
    printit ("Skipping: "+basedir+" because it already exists!")

44.852994-93.55073

ESRI Font Samples.pdf (application/pdf Object)

Wed, 23 Feb 2011 08:12:59 -0600

Whenever I want to skim through ESRI’s fonts to find some symbology I need for a specific purpose, I go through some laborious process to skim the fonts searching for an adequate symbol.

However, in the ESRI Mapping Center blog post, user use2b311 post a link to a pre-made pdf showing ESRI Font Samples.pdf (application/pdf Object).

Back when ESRI use to actually print hard-copy manuals, there were similar diagrams available but either I haven’t searched enough of they don’t exist in the digital help systems so I was happy to stumble upon this.

A tangent discovery in checking this out was dropbox, a free (with 2 Gig limit) online file sharing/online backup site–something I’ve had a need for but haven’t gotten around to researching. I have not tried them but may, just to see how well it works.

ArcGIS Online Basemaps Now Free

Tue, 22 Feb 2011 12:57:58 -0600

I’ve seen several posts like this one that reiterate that ESRI has announced that:

‘As of the end of January, ArcGIS Online basemaps published and hosted by Esri are now freely available to all users regardless of commercial, noncommercial, internal, or external use. This means that you no longer have to pay a subscription fee for including ArcGIS Online basemaps in your commercial-use web applications. Basemaps included in this new business model are World Imagery Map, World Street Map, World Topographic Map, USA Topographic Maps, and DeLorme World Basemap. The only restriction is a high-volume transaction limit of 50,000,000 transactions in a 12-month period, which is equivalent to 400,000,000 tile requests per year, or over 1,000,000 per day – a volume that most likely very few users will reach. ‘Read the ArcGIS Online blog to get the latest news and updates.’'

I was unable to find a copy of the announcement on the ArcGIS Online blog or on ESRI’s news section although I did receive an email that Charles Convis, ESRI Conservation Program Coordinator, sent to the Society for Conservation GIS mailing list with the same information so I’m guessing it is true. Perhaps ESRI pulled the post hoping to release the information with more of a splash?

Regardless, I think that this only helps make quick, affordable, professional web application more possible.

While skimming the ArcGIS Online Blog, I found a post about embedding a hosted map into your blog. Like this one of Arlington, Texas, where Super Bowl XLV was played a couple weeks ago. And, in case you missed it, the now 13-Time World Champions, Green Bay Packers, beat the Pittsburgh Steelers 31-25, winning the team’s fourth Super Bowl. ESRI also published this neat application that spatially shows which team football fans were rooting for.

Unfortunately, the free WordPress blogging software I am using allows only limited set of embedded frames–arcgis online maps not being one of them. There are wordpress add-ins to allow this but my host, WordPress.com, does not include it so you’ll just see a link. Rather disappointing.

View Arlington Map

44.852994-93.55073

Checking to see if a Field Index Exists Using Arcpy (ArGIS 10.0)

Mon, 21 Feb 2011 09:55:35 -0600

Updating some python code from 9.3 that using geoprocessing to 10.0 using arcpy and the first real object I’ve had to change relates to detecting whether or not an index exists on a table.

I previously posted code using a 9.3 geoprocessing commands, the core of it being:

indexList = gp.listindexes(tablename)
for iIndex in indexList:
    if (iIndex.Name == indexname):
       return True
return False

With arcpy, ESRI has gone back to using the Describe methodology. Way back when GISers use to type most of their commands into their wood-burning computers, the command ‘Describe’ was used to retrieve information about a coverage, grid, or other data set in our fancy AMLs (or SMLs). The snippet below shows a function for checking to see if a table has an index with a specified named.

def indexExists(tablename,indexname):

 if not arcpy.Exists(tablename):
  return False

 tabledescription = arcpy.Describe(tablename)

 for iIndex in tabledescription.indexes:
  if (iIndex.Name == indexname):
   return True

 return False

44.852994-93.55073

Multiple geodatabases and ArcSDE services from one PostgreSQL server

Thu, 17 Feb 2011 18:22:07 -0600

To better organize our ArcSDE data, we wanted to create multiple geodatabases and multiple ArcSDE services using one PostgreSQL database cluster (a cluster containing 1 machine at this point). A side question is why can’t tables and raster be placed in Feature Datasets? This wouldn’t be an end-all solution for what we want to do but there are some messy consequences of this limitation.

ESRI has instructions on Setting up multiple geodatabases in one PostgreSQL database cluster on Windows which was helpful but we repeatedly got an ‘The ArcSDE Repository was unsuccessfully completed. Would you like to view this error?’ error during the Repository Setup step, step two of four, of the ArcSDE Post Installation process. The odd thing, however, was that the wise_err.log was blank so I had no clue on why it was failing. After much Googling and head-banging, I came across a post from Kim Doty on ESRI’s forums reporting the same problem but they were able to create a service by just continuing through the process . Figuring I had nothing to lose, I thought I might as well try to go through the entire ArcSDE Post Installation process and see what happened. Although I received blank error messages, I was able to successfully create the second ArcSDE instance.

I will touch on some of the highlights of the process. In my case, I was creating a database & service with these parameters:

Database name: mgsgdb_lidar
Service name: mgs1_sde
Tablespace folder: D:mgsdb1dmgsgdb_lidar

Following ESRI’s instructions, I first made custom copies of dbinit.sde, dbtune.sde, and giomgr.defs that contain the service name (mgs1_sde) within the file name. These copies were named dbinit_mgs1_sde.sde, dbtune_mgs1_sde.sde, and giomgr_mgs1_sde.defs.

When I have trouble with the ArcSDE Post Installation, I like to go through the four steps individually so I select a ‘Custom’ install.

The four steps making up the ArcSDE Post Installation process are shown on the next dialog.

I was able to complete the first step, Define SDE User Environment, without any problems.

Make sure to set your database name, default tablespace, and tablespace folders are distinct from your first installation.

During the second step of the ArcSDE Post Installation process, Repository Setup, make sure to use the custom files you made earlier when you come to the ArcSDE configuratino files dialog.

Later in the Repository Setup, make sure to use the correct database name:

This is the first error message I received in the process (still during Repository Setup):

Clicking ‘Yes’, however, shows an empty wise_err.log file:

After viewing this, I canceled out of the Repository Setup step.

Taking a look in pgAdmin III, however, I can see that both my database (mgsgdb_lidar) and tablespace (mgsgdb_lidar) were created:

I received the same error message during the third step, Authorize ArcSDE, as in the second step:

But running the fourth step, Create ArcSDE Service, resulted in this sweet message:

After editing my pg_hba.conf and opening the port in my firewall, the service was created, running, and visible. So far, it seems to be fully functional and without problems.

44.852994-93.55073

Walkthrough: Building custom UI elements using add-ins (ArcObjects .NET 10 SDK)

Tue, 15 Feb 2011 17:05:55 -0600

I was working my way through this ESRI Walkthrough: Building custom UI elements using add-ins (ArcObjects .NET 10 SDK). And came across a couple minor errors that I had to correct during the process.

First, while implementing the OnClick() code for ZoomToLayer.vb, Visual Studio gave me a ‘Name ‘ArcMap’ is not declared.' error.

In the walk-through, they mention that the ArcMap method of your class. For me, however, it appeared under the .My method. Not sure if this is something specific to my set-up or, as I’m guessing, something that got changed after the first documentation was created and the final libraries published.

The fix is just adding ‘My.’ to the namespace in this line:

ZoomToActiveLayerInTOC(TryCast(ArcMap.Application.Document, IMxDocument))

To get this:

ZoomToActiveLayerInTOC(TryCast(My.ArcMap.Application.Document, IMxDocument))

When I added the code for AddGraphics.vb, I got 8 errors. There was essentially two errors, repeated four times. I took a screen shot after fixing the first error pair:

The fixes in this case was also to use the complete name space path. Examples:

Change this:

(geometry.GeometryType) = esriGeometryType.esriGeometryPoint Then

To this:

If (geometry.GeometryType) = ESRI.ArcGIS.Geometry.esriGeometryType.esriGeometryPoint Then)

And change this:

simpleMarkerSymbol.Style = esriSimpleMarkerStyle.esriSMSCircle

To this:

simpleMarkerSymbol.Style = ESRI.ArcGIS.Display.esriSimpleMarkerStyle.esriSMSCircle

Overall, the walk-through is very well done, just a couple minor tweaks. I am now working my way through modifying an existing solution–one that included seven projects–to see if I can create an ArcGIS 10 Add-In.

44.852994-93.55073

Zipping a File Geodatabase using Python

Tue, 08 Feb 2011 13:44:34 -0600

Ever since the ever-popular post, Zipping a shapefile using python, came out, people have been asking (one person, yesterday) for a sample of how to zip a file geodatabase using python.

The key trick, as shown in line 17, is appending the basename of the file geodatabase (‘nfg.gdb/’ in my example) in front of each file as you write it to the zipfile.

UPDATE: WordPress messes with the spacing when I post code, making it difficult to post code that can just be copied & pasted and have work. So I have posted a the code HERE for downloading.

import os
import zipfile
import glob

infile = "c:/temp/nfg.gdb"
outfile = "c:/temp/nfg.zip"
def zipFileGeodatabase(inFileGeodatabase, newZipFN):
   if not (os.path.exists(inFileGeodatabase)):
      return False

   if (os.path.exists(newZipFN)):
      os.remove(newZipFN)

   zipobj = zipfile.ZipFile(newZipFN,'w')

   for infile in glob.glob(inFileGeodatabase+"/*"):
      zipobj.write(infile, os.path.basename(inFileGeodatabase)+"/"+os.path.basename(infile), zipfile.ZIP_DEFLATED)
      print ("Zipping: "+infile)

   zipobj.close()

   return True

zipFileGeodatabase(infile,outfile)

44.852994-93.55073

Using ArcGIS 9.3.1 Desktop to Extract Data From an ArcGIS 10 Server Geodata Service

Fri, 04 Feb 2011 18:26:53 -0600

In building our Enterprise GIS Database, we need to support users with different needs. Some of our users just need to see the data on a map while others may want to download a copy of the data so they can use it within their own desktop system.

After doing some exploring, one of the options that looks like it will feel the bulk of our internal needs is to create a Map Service/Geodata Service pair–by creating a Map Service, we can make an easy-to-use visual representation of our data. Combining that with a paired Geodata Service–a Geodata Service with the same name as the Map Service–we can accommodate those users who want to extract the data to their hard drive.

A user who has added paired Map & Geodata Services to an ArcMap dataframe can use the Distributed Geodatabase Toolbar to extract data to a local geodatabase. The Extract Data button is on the far right of the Distributed Geodatabase Toolbar.

Unfortunately, my first attempts at testing this did not work. It would look like it was processing, I would end up with a .mdb or .gdb file on my hard drive but I could not open it. I would get a ‘Failed to connect to database. This release of the GeoDatabase is eitehr invalid or out of date. [Unknown GeoDatabase release.] The Microsoft Jet database engine cannot find the input table or query ‘GDB_ReleaseInfo’. Make sure it exists and that its name is spelled correctly.' error.

When exporting to an .mdb file, I could open it in Access and see the data.

Luckily the error is fairly helpful–I was getting an ArcGIS 10 Geodatabase even though I am running ArcGIS 9.3.1 on my desktop. I was able to open the exported geodatabases using a copy of ArcGIS 10 on another machine. The reason for the inconsistency is pretty interesting. Our office is, and for the foreseeable future, running in a mixed-version environment. We have a mixture of 9.3.1 and 10 desktops, a 9.3.1 ArcSDE server and a 10 ArcGIS Server. The Extract Tool is a server-side function apparently that creates a version 10 Geodatabase.

The way to get around this is to extract the data to XML and then import the XML workspace into a geodatabase. Not sure if that will work for all schemas but it was successful for me.

44.852994-93.55073

Checking to see if a Field Index Exists Using Python (geoprocessing 9.3).

Thu, 27 Jan 2011 15:31:48 -0600

NOTE: I have a post here that shows how to check if a field exists using arcpy in ArcGIS 10.0.

In developing a python script to reload a geodatabase, I wanted to create any necessary indexes.

No problem creating the index, for example:

gp.AddIndex_management(tablename, field, IndexName, "NON_UNIQUE", "NON_ASCENDING")

But before creating the index, I wanted to verify that it did not exist. I tried the ever-popular, exists but could not get it to work–either it does not detect indexes or I just never got the fully-qualified name for the index right (ArcSDE using a postgres datastore).

gp.Exists(mgs_c5ix_fullname)

I finally found this ArcGIS Desktop Help 9.3 – ListIndexes method from ESRI. Unfortunately, it doesn’t work-it did not like the ‘while’ loop construction. I’m guessing it worked in 9.2 and despite ESRI’s own warning about differences in 9.2 & 9.3, they did not update the sample code.

A key is to make sure you create a 9.3-version geoprocessing object and the following code can be used. The caveat that I need to include is that the code only checks one table, if the index is on a different table, it will give you a false-negative.

gp = arcgisscripting.create(9.3)

def indexExists(tablename,indexname):
 if not gp.Exists(tablename):
  return False

 indexList = gp.listindexes(tablename)

 for iIndex in indexList:
  if (iIndex.Name == indexname):
   return True

 return False

To call it, just pass the table and indexname you are looking for.

indexExists(tablename,indexname)

ArcGIS 9.3.1 to ArcSDE 10 Connection Error

Thu, 13 Jan 2011 14:41:58 -0600

We finally installed an instance of ArcSDE 10 today. My first attempt at connecting in ArcCatalog 9.3.1 failed with the following error:

Failed to connect to the specified server.

This release of the GeoDatabase is either invalid or out of date. [Please run the

ArcSDE setup utility using the -o install option.]

DBMS table not found [sde.sde.GDB_Release]

Turns out the solution was simple, this article points out that Service pack 2 is required. After updating my ArcGIS, the problem was solved:

I was the proud, new recipient of a non-misleading error message. After some minor head-slapping, I realized that 9.3.1 and older versions of ArcGIS are not able to connect to ArcSDE 10. Doh! My bad–I should have read ESRI’s clear information on this topic. At least I never actually tried to install ArcSDE with the -o option as the first error message instructed me to do.

44.979965-93.263836

Displaying Node Dangles in Arcedit

Thu, 06 Jan 2011 16:50:42 -0600

Since the name of the blog is Node Dangles, I get several hits daily from searches on ‘Node Dangles’ and I have no information on node dangles. This post is the first in a series to change that.

First let us, by us, I mean ESRI, define what a node dangle is. Their online glossary actually defines a dangling arc, a dangling node is a node (endpoint of an arc) that does the non-connecting mentioned below:

'An arc having the same polygon on both its left and right sides and having at least one node that does not connect to any other arc. It often occurs where a polygon does not close properly, where arcs do not connect properly (an undershoot), or where an arc was digitized past its intersection with another arc (an overshoot). A dangling arc is not always an error; for example, it can represent a cul-de-sac in a street network.'

Matt’s definition is ‘a node that isn’t connected to any other node’. The Arcedit screen shot below shows a coverage of roads and its arcs (white lines), nodes (white squares) at the ends of the arcs, and node dangles (red squares) indicating what nodes are dangling. Too bad clearly flagging dangling chads isn’t so easy.

Displaying (and illustrating) how to display dangling nodes in Arcedit is simple, the command is

'drawenvironment node dangle'

or, as I've done above,

'drawe node dangle'.

To display the dangles in a different color than the default white, follow-up with the command 'nodecolor dangle 2'. The 2 is ESRI's value for red.

Multiple outputs for Python scripts

Fri, 17 Dec 2010 15:19:38 -0600

Related to my post on how I enable a script to accept parameters from different sources, I also often set up pythons scripts to output information a variety of ways. This is largely due to the fact that some are called by ArcToolbox scripts. Running in ESRI’s domain, these scripts need to send the output through the arcgisscripting object but if you are running the python outside the ArcGIS framework, you can just print.

If you assume one output method but then run your code in the opposite framework, you don’t get to see all the pretty little messages. What I do is create a simple little routine that broadcasts the message both ways. This is probably an obvious solution but took a few cases before I went ahead and started implementing it.

gp = arcgisscripting.create()

#This will print both to the geoprocessing window or Python output window
def gpprint(inmessage):
 gp.addmessage(inmessage)
 print inmessage

<Code to do stuff>

#Ok, I want to send a message:
gpprint('Hello, sailor!')

Zipping a shapefile via ArcToolbox

Wed, 15 Dec 2010 14:36:04 -0600

UPDATE:

After receiving a request to modify the code to ignore .lock files, I have an updated to this post.

I’ve received a request on how to use the Zip Shapefile code I posted last week from ArcGIS. Sorry, I did not set the code up to call directly from ArcGIS but only as an illustration of how it can be done.

I have, however, with some minor tweaking, made a version that can added to ArcToolbox. The steps to install are below, please note that at one point I had you download a *.zip file that had been renamed to *.jpg–this should now be corrected and the link should lead you directly to zipshapefile.zip. Because of this steps two and three are obsolete.

Download the code from here.
~~Rename the file from zipshapefile-zip.jpg back to zipshapefile.zip.~~
Unzip the file.
Move ZipShapefile.py to C:Program FilesArcGISArcToolBoxScriptsZipShapefile.py.
Optionally, move Zip Shapefile.tbx, perhaps C:Program FilesArcGISArcToolBoxToolboxes.
Add the toolbox to ArcToolbox. ESRI has instructions here on how to do this.

You should now have a new toolbox named ‘Zip Shapefile’ with a script named ‘Zip a Shapefile’ in it. Clicking on on the tool will bring up this dialog.

**************
In response to Chris:

I believe you need to copy the ZipShapefile.py file from the .zip that you downloaded to C:Program FilesArcGISArcToolBoxScripts, the error message is consistent with the tool not being about to find the python script there.

If you prefer to place the ZipShapefile.py in a different location, you will need to change the source on the tool. To do this, right click on the tool in ArcCatalog and change the path of the Script File as set in the Source tab (see below):

44.852994-93.55073

Launching a Python script with parameters–3 methods.

Mon, 13 Dec 2010 14:56:43 -0600

Since I use python for different tasks, I launch python scripts a variety of ways. Depending on what I am doing, a single script may need to accept parameters from either:

Passed in from an ArcGIS Toolbox Tool.
Re-occurring default value. Often used in scheduled processes, a nightly backup, for example.
A temporary set of values used in an interactive, debugging session.

What I often do is make the parameter interpretation flexible to meet my needs. The sample below shows how I do this. The logic first checks to see if the correct number of parameters were used to launch the script (i.e. if it is called by an Arc Toolbox tool), where the files for the default files exist or if the debug values are valid, including checking the current date against a hard-coded date variable. Juggling the conditional structure would allow you to prioritize the options differently.

I am also using Tkinter to display an interactive dialog if none of the three conditions are successfully met.

import os.path
import datetime
import shutil
import sys

theEmailText = "nStart: " + str(datetime.datetime.now())

#This example shows three different ways for a script to receive input paramters.
#
# First, if it was run with the necessary number of parameters, as if launched by
# an ArcToolbox tool, it uses those parameters.

if len(sys.argv) == 2:
    wellsShapeFile = sys.argv[0]
    unwellsShapeFile = sys.argv[1]
    theEmailText = theEmailText + "nUsing Parameters Passed In:nn Located Wells File: "+wellsShapeFile+"n Unlocated Wells File: "+unwellsShapeFile
else:
    dateString = datetime.date.today().strftime("%Y%m%d")
    wellsShapeFile = "C:/cwi5_bk/wells/temp/wells_.shp"
    unwellsShapeFile = "C:/cwi5_bk/wells/temp/unloc_wells.shp"

    #Second attempt is if there are default values that should be used.
    #I use this for a process that is run via scheduled Windows Task
    if (os.path.exists(wellsShapeFile)) and (os.path.exists(unwellsShapeFile)):
        theEmailText = theEmailText + "nUsing Automated Date-Based file names:nn Located Wells File: "+wellsShapeFile+"n Unlocated Wells File: "+unwellsShapeFile
    else:
        #The third method is used for debugging/running in the IDE.
        #I put a check condition so this is valid for one day only
        #And then hard-code the temporary paths.
        #
        #Note that you may want to modify the structure of the IF statement
        #used for methods 2 & 3 so that it checks for the manual-override (3rd)
        #method first
        if dateString == '20101213':
            wellsShapeFile = "C:/cwi5_bk/wells/temp/wells.shp"
            unwellsShapeFile = "C:/cwi5_bk/wells/temp/unloc.shp"
            theEmailText = theEmailText + "nUsing Manual Override file names:nn Located Wells File: "+wellsShapeFile+"n Unlocated Wells File: "+unwellsShapeFile
        else:
            theEmailText = theEmailText + "n Manual Override for file names does not meet data filter"

    if (not os.path.exists(wellsShapeFile)) or (not os.path.exists(unwellsShapeFile)):
        from Tkinter import *
        msgbox = Tk()
        msgbox.title('Error')
        Message(msgbox,text="Must either use ArcTool to launch or edit file parameters", bg='royalblue',fg='ivory', relief=GROOVE).pack(padx=10, pady=10)
        msgbox.mainloop()
        print theEmailText
        quit()

print theEmailText

44.852994-93.55073

Zipping a shapefile using python

Fri, 10 Dec 2010 18:20:13 -0600

UPDATE:

After receiving a request to modify the code to ignore .lock files, I have an updated to this post.

One of the tasks I’ve been automating is publishing a weekly data update to a website. The update consists of shapefile. The trouble with shapefiles is they consist of 3 or more files with the same basename but different extensions in the same directory.

Not an overly complicated situation but a common one that ArcGIS does not have a solution out-of-the-box. Below is a bare-bones code snippet to do it. It has both the input shapefile and output zip file hard-coded. The call to the subroutine that does the work is: zipShapefile(wellsShapeFile,wellsZipFile)

import zipfile
import sys
import os
import glob
wellsShapeFile = "C:/cwi5_bk/WELLS/wells.SHP"
wellsZipFile = "C:/cwi5_bk/WELLS/test5.zip"

def zipShapefile(inShapefile, newZipFN):
   print 'Starting to Zip '+inShapefile+' to '+newZipFN

if not (os.path.exists(inShapefile)):
   print inShapefile + ' Does Not Exist'
   return False

if (os.path.exists(newZipFN)):
   print 'Deleting '+newZipFN
   os.remove(newZipFN)

if (os.path.exists(newZipFN)):
   print 'Unable to Delete'+newZipFN
   return False

zipobj = zipfile.ZipFile(newZipFN,'w')

for infile in glob.glob( inShapefile.lower().replace(".shp",".*")):
   print infile
   zipobj.write(infile,os.path.basename(infile),zipfile.ZIP_DEFLATED)

zipobj.close()
return True

zipShapefile(wellsShapeFile,wellsZipFile)
print "done!"

44.979965-93.263836

X-section Xacto Updated

Fri, 10 Dec 2010 18:15:51 -0600

I just happened to stumble upon an update of a tool to Create Geologic Cross Sections, eXacto Section v. 2.0, that I mentioned in July. Jennifer Carrell at the Illinois Geological Survey wrote this tool. The latest update is from December 8, 2010 and can be downloaded at ESRI’s ArcGIS Resource Center. Our office has used previous versions and finds in very useful in creating cross-sections. I have not tried this latest update.

44.852994-93.55073

Example of how to add controls in data grid VB.NET

Thu, 09 Dec 2010 17:15:35 -0600

I have been working on some data entry forms that utilize a DataGrid. Using a PostGres Geodatabase that had domains set on several fields, I could not directly bind to the controls on my dialog. So I am going the round-about way of populating my own comboboxes with valid names and displaying within the DataGrid.

Having not done this previously, I found this example: Example of how to add controls in data grid VB.NET very useful and just wanted to point it out to anyone. George Shephard’s Windows Forms FAQ also had several useful tips.

Finally, I has a problem with the masked textboxs I added to the Datagrid, they required users to click once to get focus and a second to start editing. After much googling, I found used some information from MSDN that allowed me to find a work-around. In my Mousedown event, I included this snippet (QdiControl is the control for the specific cell that the HitTestInfo says the mousedown hit):

Dim WindowsControl As System.Windows.Forms.Control = CType(QdiControl, System.Windows.Forms.Control)
If (TypeOf WindowsControl Is MaskedTextBox) Then
 Dim pTextBox As MaskedTextBox = CType(WindowsControl, MaskedTextBox)
 Dim dgdtblStyle As DataGridColumnStyle = RelatedForm.DataGrid.TableStyles(0).GridColumnStyles(CurrentColumn)

 RelatedForm.DataGrid.BeginEdit(dgdtblStyle, CurrentRow)
 pTextBox.Select(pTextBox.Text.Length, 0)
 WindowsControl.Focus()

 End If

Peace.

ArcMap Copy and Paste Features

Wed, 13 Oct 2010 16:28:39 -0500

Sometimes it seems like I could write the data down by long hand faster than ArcMap can copy & paste features from one feature class to another. geeeeezzz.

44.852994-93.55073

Another TopoToRaster Error

Wed, 15 Sep 2010 13:36:34 -0500

Subtitled: Why error messages are good.

Came up with another error while running TopoToRaster but this time ArcGIS gave an error message that led to a solution. Turned out all my contour lines had an elevation of 16 which TopoToRaster did not like. I had intended to increase the elevation and inadvertently set them all to sixteen. I had saved the previous values before editing so it turned out to be a simple fix and I didn’t have to spend a day trying figure out what was wrong.

TopoToRaster Error

Mon, 13 Sep 2010 20:25:01 -0500

In running an automated process, I had a TopoToRaster repeatedly fail on me. The only input theme was a contour theme. The process ran fine when I used the envelope of the contour theme as the output extent but when I changed it to the envelope of a polygon theme, it would bomb. The polygon’s envelope was smaller than the contour theme.

ArcCatalog would bomb out without presenting any sort of useful message.

I determined eventually that If I set the left extent as 435210, the process would work if set the right extent to 446655 or greater but would bomb if I used 446654 or less. If the the left extent was 435209, the right extent had to be 446654 or greater.

The image below shows the contours (with vertexes) and the polygon theme I was using. The two circle graphics show the approximate extent I was clipping to.

I eventually wrote python script to do a series of tests with the intent of determining the pattern of what does and does not work. The unexpected benefit was I actually got some sort of error message back. Turns out it had to do with my lack of vertices–I densified my polylines and it ran fine. Something I should have thought of earlier but it was difficult lacking any feedback.

ArcBruTile

Thu, 09 Sep 2010 14:38:46 -0500

Stumbled across ArcBruTile, an ArcMap application that allows you to display map tile services–such as OpenStreetMap, Google Maps, Bing, Spatial Cloud, and Tile Map Service (TMS)–in your ArcMap documents. It works pretty well from my preliminary testing of it. I have found that at times that the graphical tiles are distorted but I am guessing it is because they are optimized for display in a specific projection, at specific scales and, in ArcMap, you can use any projection/scale combination you want.

This is a free, open-source project that is still in Beta but definitely an application I plan to keep an eye on. As with any open-source project, the cool thing is the code is available for you to look at, learn from, and hopefully contribute to.

Icon Transparency

Wed, 08 Sep 2010 15:47:36 -0500

Maybe I am just dense, but skimming ArcIdeas, I found out that the transparency for an icon is determined by the upper-left pixel. Don’t know if this is an ESRI convention or not but thought I would pass it one.

Workstation ArcInfo Interchange Import Error

Tue, 07 Sep 2010 19:08:39 -0500

I have been working on updating some data that was published using Arc/Info Workstation Export (.e00) files, converting them to shapefiles, or in the case of annotation, to a geodatabase feature class.

While converting some INFO tables, I have received this error:

It turns out that Workstation can create export files that contain values that did not fit within their field–I’m guessing that somehow you are actually able to enter invalid values into those fields but I did not confirm that.

The error, ‘Unable to encode the 38th item in the 65th record…’ was only partially helpful. The error was in the 65th record but I was not able figure out which field contained the error. The nature assumption that it was the 38th column was incorrect, at least if you are counting from left to right. Whatever logic that is used, is consistent, at least within a specific table, because I had numerous records with problems and when the error messages indicated it was the same item, it did turn out to be the same field.

What I ended up having to do was open the export file in my favorite text editor, locating the info for problematic record and go through the columns one-by-one, comparing the value in the text file with that in the table. In this case, 14.1 was suppose to be the value in the WP_TIO2 field.

One thing I noticed, and this is consistent with behavior I have observed in ArcObjects in other cases, is that the record with the invalid value in a field ends up with the same value in that field as the previous record. So in this case, the bad record has a value of 1.12 in the WP_TIO2 field because that is what previous record has. That leads me to believe that ArcObjects is re-using a variable as it does the import, just updating each value, field-by-field.

Just to make sure that I had diagnosed the problem right, I attempted to change the value to 14.1 in an ArcMap edit session and got this message:

A look at the field properties confirmed that the field would not accept 14.1 as a value. The precision (3) means the field can hold 3 digits with 2 (the Scale) of them being right of the decimal point–leaving only one digit available to the left of the decimal point.

Once I identified the problem, I used two methods to correct it. The majority of my cases were actually cases where a psuedo-null value was being used to indicate that a field was really Null. In some cases, either -999.9 or 100 were being used. In these cases, I just substituted a different psuedo-null value (-99.9) that conformed to the field limitations. The data I was working with was not tightly controlled in the past and there were other existing records that used these value for the same fields, so I was able to do this without much concern.

Some values, however, were not nulls and I felt needed to be retained. These, I had to add a new field that could contain the values, copy the values over and then edit the ones that had caused problems during the import. Once complete, I exported the data to a new table with the new table structure I created.

Trying to perform 'relate' in Model Builder

Wed, 11 Aug 2010 17:31:21 -0500

I was using Model Builder (ugh!) to select records in one table (CWI.C5ST) that relate to a subset of records ([BHGEOPHYS] = ‘Y’) in another table (CWI.C5IX). There is not an existing tool for doing this in ArcGIS. I did find a post by Layne Seely in ArcForums titled ‘trying to perform ‘relate’ in Model Builder.’ that led me to the Make Table View under Data Management Tools-Make Table View and even had the basic syntax I was looking for (if it hadn’t I probably would have guessed that it would not allow subqueries).

Since I always like to see exactly the syntax someone uses, I decided to show the syntax I used in case that helps anyone else.

It does require that the tables reside in the same workspace, which is a reasonable limitation that I can live with. Another drawback is that if you export the model to a python script, the command can get pretty long, especially if you have many fields.

Utah Geological Survey

Wed, 04 Aug 2010 16:50:04 -0500

The Utah Geological Survey has a good supply of Maps and data available online and some online maps. They also have an online Blog.

The GIS data is a fairly complete package–the sample I downloaded included .pdfs, shapefiles, layer files, images and ArcMap project (.MXD) files used to generate the .pdfs. File sizes are larger because of the completeness. Feature classes have a moderate level of attribution.

The highlight, however, are the sample Keyhole Markup (KMZ) files of the of the St. George 30′ x 60′ quadrangle that can be viewed in Google Earth. Too cool.

Accessing Network Drives via Scheduled Python Task

Mon, 26 Jul 2010 12:43:02 -0500

As previously mentioned, I have a scheduled nightly backup that is written in Python. Most of it has been working fine but I had not gotten it to copy files to network drives. I finally got around to correcting that part. My first attempt was just to map the network drives for the user account that is used to run the task. No good.

My second attempt was to call a .bat file from python before accessing the network drives and that worked.

The batch file has two lines (one for each network drive):

net use n: \domain1sharename mypassword /USER:domainusername
net use t: \domain2sharename mypassword /USER:domainusername

And my python script includes two lines to call it before I access the network locations:

import os
os.system('C:/cwi5_bk/mapDrive.bat')

Arizona Geological Survey

Fri, 23 Jul 2010 19:07:52 -0500

The Arizona Geological Survey has a great looking site with some good information. They have a publication list that can either be downloaded or ordered and are making pdf versions of their recent publications available online.

I was going to note that there was no download-able data available until Google told me that they in fact had a WMS/WFS available online. The WMS is a great looking map, I have not gotten the WFS to work, however–probably more due to my abilities than the service.

They are also temporarily hosting a surficial geology map of Clark County, Nevada.

Lee Allison, the State Geologist, also has a blog with frequent postings. One of which led me to the Nation Park Service’s Data Store which is worth a look.

Create Geologic Cross Sections–eXacto Section v. 2.0, ArcMap 9.3

Fri, 23 Jul 2010 13:59:44 -0500

The Create Geologic Cross Sections–eXacto Section v. 2.0, ArcMap 9.3 written by Jennifer Carrell of the Illinois State Geological Survey is a handy tool for creating cross sections.

It requires ArcGIS and a 3D Analyst Extension license.

You can create profiles against multiple DEMs at once, define the vertical exaggeration, and have it include contact points. It is well documented and comes with sample data to use with the tutorial.

I did not see it mentioned but it also include a personal geodatabase that include a grid that can be used along with the profiles you create.

If you make profiles, think this is definitely an option you should look at. The fact that it is a VBA application is a bit worrisome in that its lifespan is limited.

Michigan Office of Geological Survey

Thu, 22 Jul 2010 15:31:20 -0500

The Michigan Office of Geological Survey appears to have pdf versions of all the documents in their Digital Geological Library available for download. The transcripts of some early (beginning in 1871) field notes are a fun inclusion in the available archives.

Actual GIS data was a bit hard to find although I found both bedrock geology and quarternary geology available from the state Geographic Data Library in shapefile format. I also found oil and gas well data but did not download it.

The data carried minimal attribution, one interesting thing I found in the bedrock data was the red, green, blue values apparently used for each polygon.

One thing to be aware of is that the shapefiles do not come with a prj file which could be a problem. Michigan has their own defined coordinate system called Michigan GeoRef. This coordinate system defines the entire state into a single zone instead of multiple zones like the USGS' stateplane system does. This is nice to work with UNLESS it comes undocumented as I found out while doing some consulting work in Au Train, Michigan a decade ago–I was able to finally find something online about it around 2 am one morning but I think I still have scars from the head-pounding that occurred that night.

ESRI, however, makes it easy to deal with this data–they include the specifications for Michigan GeoRef in inventory of predefined coordinate systems that ships with ArcGIS. If you define the projection on each shapefile, it will then load automatically in real-world coordinates. To define the projection on a shapefile, right-click on it in ArcCatalog and select ‘Properties…'. Select ‘XY Coordinate System’ on the Shapefile Properties dialog. Hit the ‘Select’ button and navigate to Projected Coordinate Systems-State Sytems-NAD 1983 Michigan GeoRef (Meters).prj. Hit the ‘Add’ button and then the ‘Apply’ button. Repeat for any other shapefiles you download.

One item I found out about ArcGIS while doing this is that if you change the projection on a shapefile, it will recognize that change the next time you use that data. I originally set the units to be feet on the data so it did not load in the proper real-world location. I saved my ArcMap document, quit out, and corrected the problem. When I re-opened the document, Michigan had migrated back to where we are used to seeing it. This contrasts with how ArcView 3.x use to handle raster data, you had to actually remove the layer from your view and re-load it to recognize the change in spatial references.

Wisconsin Geological and Natural History Survey

Thu, 15 Jul 2010 13:18:28 -0500

The Wisconsin Geological and Natural History Survey has a well organized, appealing website. Their data is easy to find although there is not much GIS data available.

Although because I am more interested in sample data sets than anything, the GIS data they do have–statewide geologic map; pleistocene geology, precambrian geology, quaternary geology, (surficial?) geology and bedrock geology for various counties–may actually be a great reference. Other than the statewide geologic map, the data is available on a per-publication basis.

Some other data is available in the open-file reports.

They also have a well contructor’s reports CD available for purchase, albeit only scanned images of the wells and a very simple database.

Other .pdfs are also available of various state-wide maps and for various open-file reports.

I found some interesting things while reviewing the data I downloaded.

There is minimal attribution on the data–enough to symbolize it.
Dip & Strike triangles are captured as points with attribution of the horizontal azimuth and inclination.
One item I found odd at first but quickly grew to appreciate is that they provide a georeferenced .sid of each plate. This allows you to directly compare the digital data to the final cartographic product. Think this may be a keeper of an idea.

ArcGIS Ideas

Wed, 14 Jul 2010 14:20:47 -0500

Ok, maybe I have been hiding under a rock, but just saw this is a post from James Fee’s Spatially Adjusted blog. ESRI has a public suggestion box called ArcGIS Ideas where users can submit suggestions for ESRI. One thing I like is that it is open for other users to see and comment. Not sure how responsive ESRI will be but at least it gives us a place to vent.

Iowa Geological Survey

Wed, 14 Jul 2010 13:01:47 -0500

The Iowa Geological Surveyhas a lot of data available to download here. PDF versions of many of their publications can be found in their List of Publications.

Their GIS data is minimally attributed for the most part, their public wells data set did have a more robust attribution scheme. One cool feature I found in the wells data is that they provide a hyper-link for many features to an on-line site record. Found both shapefiles and file geodatabases in the samples I downloaded.

More interestingly, I opened up their Bedrock Geological Map and noticed something odd to me. See the bulls-eye just NW of center? That, even to a non-geologist like me, does not look too geologic.

Turns out it is related to a meteorite impact known as the Manson Crater, once thought to be The Dinosaur Killer until it was determined to be too old.

Entering a Null Value in VB.Net DateTimePicker

Tue, 13 Jul 2010 16:26:45 -0500

While working on a VB.Net form to query a database that includes an optional date range query, I decided I wanted to display a datetimepicker with a blank value by default and then, if the user sets a date show the date. By default, a datetimepicker can not have a Null date, so I went a’Googling and found this post, .NET Reference: Setting the DateTimePicker to a Blank Value,that had exactly the technique I wanted.

Basically, I set the datetimepicker’s CustomFormat to a space (' ‘) when I want it to be Null and then when the user makes a selection, set the custom format to the date format I wanted to display. This is the sample code I found:

Private Sub ToDateField_ValueChanged(ByVal sender As System.Object, ByVal e As System.EventArgs) _
   Handles ToDateField.ValueChanged
     Me.ToDateField.Format = Windows.Forms.DateTimePickerFormat.Custom
     Me.ToDateField.CustomFormat = "M/dd/yyyy"
End Sub

To let the user delete the date they entered, I add this subroutine that handles the KeyDown event for the datetimepicker. If the user hits delete or backspace, I set the format back to my null CustomFormat.

Private Sub ClearDates_KeyPress(ByVal sender As Object, ByVal e As System.Windows.Forms.KeyEventArgs) _
 Handles ToDateField.KeyDown

 If (e.KeyCode = Windows.Forms.Keys.Delete) Or (e.KeyCode = Windows.Forms.Keys.Back) Then
 Dim mDateTimePicker As Windows.Forms.DateTimePicker
          mDateTimePicker = TryCast(sender, Windows.Forms.DateTimePicker)
          If Not (mDateTimePicker Is Nothing) Then
               mDateTimePicker.CustomFormat = " "
          End If
End If
End Sub

When building my SQL statement, I check the .CustomFormat property to determine whether or not to include a condition for the data:

If Not (Me.ToDateField.CustomFormat = " ") Then
 'Code here to add to my SQL statement
 End If

South Dakota Geological Survey

Tue, 13 Jul 2010 12:33:07 -0500

The South Dakota Geological Survey website provides some good base data–imagery, DRGs, DEMs, and DLGs. There is a searchable database of core cuttings and the state data server has bedrock data for the eastern portion of the state and a statewide layer, presumably surficial geology, called geology. Quad-based geological maps are also available (PDF format) in the Publications and Maps section. The data I downloaded was in shapefile format with minimal attribution.

North Dakota Geological Survey

Mon, 12 Jul 2010 13:53:28 -0500

Took a quick look at the North Dakota Geological Survey’s website for GIS data.

The only data I found served directly from their website was surface geology map .pdfs for some 24k and 100k maps. Also available is a search for Oil & Gas wells although I did not see the data was downloadable. Some other publications have .pdfs that can be downloaded.

The North Dakota Hub Explorer serves as the state’s official data portal including for GIS data. The downloadable data includes bedrock geology, surface geology, and contours. Interestingly, the site allows you to choose from 7 different coordinate systems and 27 different file formats including several AutoCad, ESRI, MapInfo, and KML. The data I looked at had more attribution than the bare minimum shown on a hard-copy map. Bedrock data, for example, had additional lithology.

Even though I appeared to be getting a Request is complex error message, their system functioned very well, you step your way through a few selections and a few minutes later you receive an email providing a link to start a download of your compressed data.

ESRI releases Service ArcGIS 9.3.1 Pack 2

Mon, 12 Jul 2010 12:36:49 -0500

Right before the Users Conference, ESRI has released ArcGIS Desktop 9.3.1 Service Pack

Haven’t had a chance to see what bugs this addresses yet.

AASG

Fri, 09 Jul 2010 19:49:51 -0500

I was looking for links to all the State Geological Surveys and found the Association of American State Geologists (AASG) Homepage to have exactly what I needed.

I hope to build some sort of inventory of data samples available. If I do one a day, I can be done in 10 weeks, or about 3 months.

What I will find the most useful is data distribution methods and data structures.

The USGS has a sample web application and it includes statewide data for four states–Arizona, Idaho, Oregon and Washington, so those, along with Illinois and the states that border Minnesota will be among the first I take a look at.

Python emailer

Fri, 09 Jul 2010 17:20:38 -0500

So I have a nightly process that runs and I, being the lazy programmer I am, didn’t want to bother checking a log file each morning to see how it went. The natural answer is to have the results emailed to me because I do have to check my email.

Since the process is already handled mostly in python, thought I would implement the emailing via python. It was actually simple enough to do–just required figuring out the setting for my SMTP server (actually I didn’t figure it our for my work server so I’m using my GMail account).

I first wrote eMailer.py (see code below) and included it in my python path. Now, whenever I want to send code from a python application, it takes two easy lines:

import eMailer
eMailer.SendMess("someoneg@company.com","Subject","Text Body")

eMailer.py

import smtplib
from email.MIMEMultipart import MIMEMultipart
from email.MIMEText import MIMEText

fromaddr = 'node.dangles AT gmail.com'
password = 'myFancyPassword'
smtpaddr = 'smtp.gmail.com'
smtpport = 587

def SendMess(toadd, subject, body):

msg = MIMEMultipart()
msg['From'] = fromaddr
msg['To'] = toadd
msg['Subject'] = subject
msg.attach(MIMEText(body, 'plain'))

text = msg.as_string()

server = smtplib.SMTP(smtpaddr, smtpport)
server.ehlo()
server.starttls()
server.ehlo()
server.login(fromaddr, password)

server.sendmail(fromaddr, toadd, text)

Hogeweg: ArcGIS Editor for OpenStreetMap

Wed, 07 Jul 2010 19:10:35 -0500

As someone who lives in the boon-docks whose street (both my current and previous one) does not appear in the data for either NAVTEQ or TeleAtlas and someone who used both of those data sets extensively in his previous job, I’ve watched OpenStreetMap for awhile, although have not contributed.

Marten Hogeweg, ArcGIS Server Geoportal Extension Product Manager, ESRI announce in his blogs an ArcGIS Editor for OpenStreetMap. Sounds like a great idea, giving GIS professionals–some whom have the source materials the ability to easily contribute should only help the project.

Unfortunately, it is only available at 10.0 but that gives me another reason to install it on a virtual machine and start playing more.

NED Processing in Python and Geoprocessing

Wed, 07 Jul 2010 14:05:20 -0500

The USGS updated a significant portion of NED data for my state in June. I recently downloaded the updates, processed them–projecting and converting the elevations from meters to feet–using python and geoprocessing. My python skills are still pretty crude but I was able to get the job done.

One of the benefits of working for the public sector is that I can more freely publish code without worrying about ‘trade secrets’ or what have you so I thought I would put my code out for anyone to see, maybe someone will find it useful. It is split into two separate files, mostly because I used Model Builder to generate the main processing chunk of code (Project_Reclassify.py) and another script to control looping, etc. The code isn’t pretty and you’ll see the results of my development process with two subroutines in LaunchScript.py that could really be one.

LaunchScript.py

import Project_Reclassify

mConversion = "3.28084"
mProjection = "PROJCS['NAD_1983_UTM_Zone_15N',GEOGCS['GCS_North_American_1983',DATUM['D_North_American_1983',SPHEROID['GRS_1980',6378137.0,298.257222101]],PRIMEM['Greenwich',0.0],UNIT['Degree',0.0174532925199433]],PROJECTION['Transverse_Mercator'],PARAMETER['False_Easting',500000.0],PARAMETER['False_Northing',0.0],PARAMETER['Central_Meridian',-93.0],PARAMETER['Scale_Factor',0.9996],PARAMETER['Latitude_Of_Origin',0.0],UNIT['Meter',1.0]]"
mCellSize = "10"

#Project_Reclassify.Process("C:\mgs\Projects\NED_10m\n44w092\grdn44w092_13", mProjection, mConversion,"C:\mgs\Projects\NED_10m\n44w092\try30",mCellSize)

def CallIt (inDir,inGrid,outGrid):
fullinFN = inDir+'\'+inGrid
fulloutFN = 'C:\mgs\Projects\NED_10m\Output\'+outGrid
tmpGridFN = 'c:\temp\t'+outGrid
Project_Reclassify.Process(fullinFN, mProjection, mConversion,fulloutFN,mCellSize,tmpGridFN)

def CallIt2 (inGrd,inoutGrd):
outDir = 'C:\mgs\Projects\NED_10m\' + inGrd + '\' + inGrd

outGrd = 'grd'+inGrd+'_13'

CallIt(outDir,outGrd,inoutGrd)

#CallIt2("n44w092","LaCrosse_W")
CallIt2("n44w093","MasonCity_E")
CallIt2("n44w094","MasonCity_W")
CallIt2("n44w095","Fairmont_E")
CallIt2("n44w096","Fairmont_W")
CallIt2("n44w097","SiouxFalls_E")
#CallIt2("n45w092","EauClaire_W")
CallIt2("n45w093","StPaul_E")
CallIt2("n45w094","StPaul_W")
CallIt2("n45w095","NewUlm_E")
CallIt2("n45w096","NewUlm_W")
CallIt2("n45w097","Watertown_E")
#CallIt2("n46w093","Stillwater_E")
#CallIt2("n46w094","Stillwater_W")
#CallIt2("n46w095","StCloud_E")
CallIt2("n46w096","StCloud_W")
CallIt2("n46w097","Milbank_E")
#CallIt2("n47w092","Ashland_W")
#CallIt2("n47w093","Duluth_E")
CallIt2("n47w094","Duluth_W")
CallIt2("n47w095","Brainerd_E")
CallIt2("n47w096","Brainerd_W")
CallIt2("n47w097","Fargo_E")
#CallIt2("n48w090","Hancock_W")
#CallIt2("n48w091","TwoHarbors_E")
#CallIt2("n48w092","TwoHarbors_W")
#CallIt2("n48w093","Hibbing_E")
#CallIt2("n48w094","Hibbing_W")
CallIt2("n48w095","Bemidji_E")
CallIt2("n48w096","Bemidji_W")
CallIt2("n48w097","GrandForks_E")
#CallIt2("n48w098","GrandForks_W")
#CallIt2("n49w090","ThunderBay_W")
#CallIt2("n49w091","Quetico_E")
#CallIt2("n49w092","Quetico_W")
#CallIt2("n49w093","InternationFalls_E")
#CallIt2("n49w094","InternationFalls_W")
CallIt2("n49w095","Roseau_E")
CallIt2("n49w096","Roseau_W")
#CallIt2("n49w097","ThiefRiverFalls_E")
CallIt2("n49w098","ThiefRiverFalls_W")
#CallIt2("n50w095","Kenora_E")
#CallIt2("n50w096","Kenora_W")

Project.Reclassify.py

# Import system modules
import sys, string, os, arcgisscripting

# Create the Geoprocessor object
gp = arcgisscripting.create()

# Check out any necessary licenses
gp.CheckOutExtension("spatial")

# Load required toolboxes...
gp.AddToolbox("C:/Program Files/ArcGIS/ArcToolbox/Toolboxes/Spatial Analyst Tools.tbx")
gp.AddToolbox("C:/Program Files/ArcGIS/ArcToolbox/Toolboxes/Data Management Tools.tbx")

# Script arguments...
def Process( inInputRaster, inOutputCoord, inConversionFactor, inOutputRaster, inCellSize, intempRasterFN):
Input_Raster = inInputRaster #sys.argv[1]
Output_Coordinate_System = inOutputCoord #sys.argv[3]
Input_raster_or_constant_value_2 = inConversionFactor #sys.argv[4]
Output_CellSize = inCellSize

#if Input_raster_or_constant_value_2 == '#':
# Input_raster_or_constant_value_2 = "3.28084" # provide a default value if unspecified

Output_raster = inOutputRaster

# Local variables...
Temp_Raster = intempRasterFN

print Input_Raster
print Temp_Raster
print Output_Coordinate_System
print inCellSize

# Process: Project Raster...
gp.ProjectRaster_management(Input_Raster, Temp_Raster, Output_Coordinate_System, "CUBIC", Output_CellSize, "", "", "")

print Input_raster_or_constant_value_2
print Output_raster

# Process: Times...
gp.Times_sa(Temp_Raster, Input_raster_or_constant_value_2, Output_raster)

Sticky Move Tolerance

Thu, 01 Jul 2010 16:51:42 -0500

While I am pretty experienced in editing data in previous generations of ESRI’s software, I have not done a lot in ArcGIS. So in my new position where I am supporting a group of geologists who do a lot of data creation in ArcMap, I am learning some of the intricacies of the ArcGIS platform.

One of the potential gotchas that a coworker informed me about that I was vaguely aware of was the issue of sticky move tolerances. What can happen, when editing existing features, is that each time you select a feature it can easily get moved very slightly if the mouse moves while the feature is selected. While the movement itself may be well within any allowable accuracy requirements, it can make a huge mess if you are working with a shapefile polygon coverage for example.

ArcMap has a setting, Sticky Move Tolerance, that helps prevent these tiny, accidental moving of features. A feature will not be moved until the mouse moves at least as many screen pixels as this setting. The default setting is 0, disabling this feature.

This setting can be set changed under Options on the Editor Toolbar.

The Sticky move tolerance is on the General tab. In this example, I have my setting at 25 pixels which may be a bit high but I do not do a lot of data editing myself so it is workable. The downside of having it set this high is if I want to move a feature just a tiny bit, I have to first drag the mouse relatively far away and then bring it tighter into where the feature is.

Not sure how well this is documented although it can be found in the help system.

ArcMap-Excel Join Error

Fri, 28 May 2010 02:02:37 -0500

Came across a new ArcMap bug today. A staff member was trying to join an Excel 2007 file (.xlsx) to a point shapefile in ArcMap and it did not appear to be working. She only had Office 2003 on her machine so I tried it on my spanking new machine that has 2007 installed. Same result–the fields get appended but they are all blank.

I discovered, however, that when I had the table open, if I switched form showing ALL records to only the selected, Poof!!! The values filled in. From there I was able to successfully export the data to a new shapefile that always showed the values.

I have an inkling that the file name, which contained space and other special characters, may have been a factor.

August 11, 2010 Addendum:

I was just working with some more and, after joining a table to an Excel table, having all the fields look blank, I discovered that if I close and then re-open the table all looks good–all fields have values.

Another problem I had is that even after successfully joining a table to the Excel table and creating an XY Event Theme from it, I was unsuccessful in exporting it to either a shapefile or a personal geodatabase feature class. My personal opinion at this point would be to convert the Excel file to a different format before working with it in ArcGIS.

Geology Information Systems

Mon, 24 May 2010 02:52:24 -0500

In the last couple months, I’ve had a bit of an eye-opening about Geography. People actually trained in geography may implicitly understand this but I, with my Accounting degree, have spent the last 16 years ‘doing’ GIS without realizing the foundation of geography–geology. I picked up enough geography to understand some of the inter-relationships between people and the lands they live on.

But what I didn’t pick up on was how the lands we live on are formed. Maybe it is a moot point–how physical geography is arrived at doesn’t matter, just how we react to it. But, and now working for a Geological Survey, I better have this opinion, a better understanding of what our geology is and how it was formed really does form the foundation for the geography.

Two examples. First, in talking with my new supervisor, he was talking about how, because of the geology in the southeast part of the state, they have a lot of water quality issues–the water tends to have sulfur in it. With the recent interest in alternative energy sources, and in this case ethanol plants water quality is a concern when deciding where to build the plants because they require a considerable amount of water. For some reason there have been plants placed in this part of the state and they are having some problems finding sufficient water.

The second, and I’m not sure that this is actually fact or just a theory but it is an interesting idea, was something I read in Malcolm Gladwell’s book Outliers. In the book, it was suggested that there was an almost epidemic of feuds, a la Hatfield-McCoy, in part of the US largely due to the geology of a portion of the country. The areas that relied on a herding economy developed a clannish ‘code of honor’ that led to these long-term feuds. While I understood that physical geography effects human behavior I hadn’t taken the mental step to link geology to physical geography.

Pretty silly and probably obvious oversight.

And to think, I use to think the ‘G’ in GIS should really stand for Geometry. Humph!

Measuring distance from a point to a line segment in Python

Sun, 16 May 2010 17:14:23 -0500

I recently had the need to calculate the distance from a point (address point) to a polyline (street segment) and wanted to avoid using any additional libraries because it was being done for an external client. Ok, I actually used arcgisscripting for reading the data but that lacked, from what I could tell, the fine-detail granularity of measuring distance between individual geometries.

But since the only spatial operations I needed were to measure the distance between two points and the distance between a point and polyline, I decided to just do it via brute force. The datasets being processed were not huge so it made this feasible.

Measuring point-to-point distance was easy enough. And while I knew I could work through the details of measuring the distance between a point and polyline, I decided NOT to re-invent the wheel if I did not have to. Google did not come up with any direct answers but I did come across this post from Paul Bourke at the University of Western Australia about measuring the distance between a point and a line segment. A polyline is just a series of line segments so I used it as the basis of my measurement–I just loop through all the consecutive vertice pairs, using the minimum distance found.

Paul did not have a python implementation included, so I went ahead and created my own from his information.

def lineMagnitude (x1, y1, x2, y2):
    lineMagnitude = math.sqrt(math.pow((x2 - x1), 2)+ math.pow((y2 - y1), 2))
    return lineMagnitude

#Calc minimum distance from a point and a line segment (i.e. consecutive vertices in a polyline).
def DistancePointLine (px, py, x1, y1, x2, y2):
    #http://local.wasp.uwa.edu.au/~pbourke/geometry/pointline/source.vba
    LineMag = lineMagnitude(x1, y1, x2, y2)

    if LineMag < 0.00000001:
        DistancePointLine = 9999
        return DistancePointLine

    u1 = (((px - x1) * (x2 - x1)) + ((py - y1) * (y2 - y1)))
    u = u1 / (LineMag * LineMag)

    if (u < 0.00001) or (u > 1):
        #// closest point does not fall within the line segment, take the shorter distance
        #// to an endpoint
        ix = lineMagnitude(px, py, x1, y1)
        iy = lineMagnitude(px, py, x2, y2)
        if ix > iy:
            DistancePointLine = iy
        else:
            DistancePointLine = ix
    else:
        # Intersecting point is on the line, use the formula
        ix = x1 + u * (x2 - x1)
        iy = y1 + u * (y2 - y1)
        DistancePointLine = lineMagnitude(px, py, ix, iy)

    return DistancePointLine

My Favorite Avenue Snippet Ever

Sun, 16 May 2010 16:33:30 -0500

I worked in Avenue for many years but do not expect to do too much more. Before my Avenue skills erode to nothing, thought I would share my favorite Avenue snippet. Really, it involves two ‘function’ scripts that clients scripts can use.

What I got use to doing is instead of using a mess of global variables is using one global variable–a dictionary–that was used to store what essentially became global variables. The main advantage of doing this is that I started storing the parameters passed to each script as a global variable named after the script. This came in EXTREMELY useful for debugging purposes because when I was having trouble with a script I could write out my global dictionary to an ODB file at the beginning of that script, thereby storing the application’s state prior to the problem.

The first script, MyApplication_Routine_AddSelfDict, is used to add/modify a dictionary entry:

theScript = self.get(0)
theSelf = self.get(1)

if (_MyApplication_SelfDict.is(Dictionary).not) then
  _MyApplication_SelfDict = dictionary.make(25)
end

if (self <> nil) then
  _MyApplication_SelfDict.add(theScript,theSelf)
  _MyApplication_SelfDict.set(theScript,theSelf)
end

The second script, MyApplication_Routine_GetSelfDict, is used to retrieve a value from the global dictionary.

if (_MyApplication_SelfDict.is(Dictionary).not) then
  _MyApplication_SelfDict = dictionary.make(25)
end

return (_MyApplication_SelfDict.get(self))

Then I can have client scripts first add an entry (primary key = script name)to the global dictionary storing the parameters passed in (self). Then I set a local variable to the entry just added. Then, if I need to debug later, I can throw an exit (shown in red) into the script, run the process and after it exits, I can step through the script and pull the parameters passed in from the global dictionary.

If (self<>nil) then
  av.run("MyApplication_Routine_AddSelfDict",{"MyApplication_ClientScript",self})
End

theSelf = av.run("MyApplication_Routine_GetSelfDict","MyApplication_ClientScript")

'**** The following EXIT is added for debugging purposes.  
'This gets temporarily added so I can capture 'the application  state.
'After running the process and getting to this point, I remove the exit and can step 
'through this script.  The snippet above retrieves the parameters as passed into this script.

EXIT

And that’s it. Took me several years to refine this technique–for a long time I just stuck a check in at the beginning of each script to see if there were any parameters passed and if so, store them in a separate global. But what I like abut this is I can write the whole global dictionary out to an ODB, quit out of ArcView and then return to the project, read in the ODB and reset my global dictionary and pick up from where I left off.

The one addition that I added later was to remove entries from the global dictionary once a script ends. I didn’t show it here.

Having all this data in one global dictionary is probably a horrendous design approach to anyone with an Object-Orientated mindset but, as they say, it got the job done.

Endpoints

Sat, 15 May 2010 13:27:09 -0500

I spent almost nine years working for a great company, Applied Data Consultants!, through a variety of roles–spent approximately 3 weeks doing Parcel work as a GIS Tech before I started working in Avenue for my own sanity. After that, spent several years doing various programming tasks including building tools for internal production and application development. The last couple years, I served more of a Consultant role, both for internal and external clients.

Where one polyline ends, another starts in this case.

In a couple days, I start my role as Geological Information Specialist at a state agency in Minnesota. I previously worked for the Wisconsin DNR (where I originally ‘discovered’ GIS) and Pierce County, Wisconsin so I have some governmental experience but I am wondering how I will adapt to the organizational differences.